BitCoin ticker with Python, Kafka and Bokeh

Hi all,

Today, just a variation of the previous bitcoin ticker.

The concept is still simple but involves differents tools :

• using Python
• calling public rest service for Bitcoin rates
• sending data into kafka topic
• polling the topic to retrieve bitcoin rates
• updating in real time, n second interval, a bokeh page with a nice graph

Architecture

The “architecture” for this projet is fairly simple :

• a server in the cloud running python and kafka
• a workstation running python an bokeh
• public REST service delivering bitcoin rates

Process

You can easily find rest services providing data about Bitcoin. I chose to use this rest call : https://min-api.cryptocompare.com/data/price?fsym=BTC&tsyms=USD,EUR&e=Coinbase&extraParams=your_app_name

Pretty simple, this service returns some compact json data. In this use case, I will parse the json object and retrieve the USD rate only.

{"USD":8089.97,"EUR":6556.01}

The overall process is :

• the cloud server is running a python script polling REST service to grab bitcoin rates and stores the data in kafka topic called “btc_ticker”
• the workstation is running a second python script polling the Kafka server and updates the data in a bokeh powered graph
• you can choose the polling rates as well as the graph update rates

All the pieces in action

Server side

Start the python script doint the REST poller Kafka producer. Here is an accelerated gif showing the polling in action. I managed to capture a some variation on the BTC rate here.

The BTC rates are now flowing into a Kafka topic called btc_ticker. We can verify with Kafka Tool :

Workstation side

On the workstation side, things are just a bit more complex. Just a bit.

First start the bokeh server. Careful here, very complex operation as you can see below.

The we can start our second python script, the client side one. This script will connect to kafka server, instanciate a consumer, then within an infinite loop, will retrieve data and update a real time graph in Bokeh.

I’m using Visual Studio Code for Python coding along with Jupyter. Once the script is started, a web page is automatically opened and we can discover our Bokeh graph in real time.

The output, with bokeh, is a simple html page displaying the graph and updating in real time. For this example, I chose a 3 seconds refresh rate (pretty useless but this was fun to see the high Bitcoin volatility).

Look at this accelerated gif, created from a record on October 23 2108, displaying the rates for approx 40 minutes. I tried to capture data having obvious variations.

The scripts

This is the server side script. Please contact me for the bokeh client side script.

More to come

This is a quite simple example. Next, I will add some “intelligence” and data processing in Kafka, for instance adding a moving average accross different Bitcoin rates suppliers.