bdsBlockchain data parsing and persisting results
BDS
Introduction
JD Cloud Blockchain Data Service (BDS) is a realtime data aggregating, analyzing and visualization service for chain-like unstructured data from all kinds of 3rd party Blockchains
Splitter is the key module of Blockchain Data Service (BDS) and provides data analysis capability.
Splitter is responsible for consuming blockchain data from message queue (kafka) and inserting data into persistent data storage services (relational database, data warehouse, etc.) for further processing
Architecture
Environment Deployment
Install BDS
Environment initialization
Before compiling and running BDS, you must install go's compilation environment locally: go install
Install Splitter steps
- Set the path of project :
$GOPATH/src/github.com/jdcloud-bds/bds/ - Input
go build -v github.com/jdcloud-bds/bds/cmd/bds-splitter，compile to get executable file bds-splitter - Build new configuration file splitter.conf, see
/config/splitter_example.confconfiguration file template - Run program
./bds-splitter -c splitter.conf
Install confluent and kafka
Install kafka
See kafka
Modify config/server.properties
- message.max.bytes=1048576000
Install confluent
see confluent
Unzip the confluent package and run Confluent REST Proxy
Modify /etc/kafka-rest/kafka-rest.properties
- max.request.size = 1048576000
- buffer.memory = 1048576000
- send.buffer.bytes = 1048576000
Database
Database we now support SQL Server, PostgreSQL, you can choose one as a data storage method.
SQL Server
PostgreSQL
After you run the database, you need to manually create new database and use the database name initialization splitter.conf.
Install Grafana
See Grafana Official
Source code
Development Steps
- Define the data structure of Kafka messages.
- Define table structure.
- Analyze Kafka message and store data in database.