Python Kafka Client Setup on MacOSX

There are many Python libs for Kafka. For now, we are using Confluent Kafka's version. We tested it and it works well.

We picked this one because it is supported by Confluent and it is based on the Kafka C lib so it should be as fast if not faster than the pure Python versions.

Using Lab 10 from the Java Kafka Course. Start up ZooKeeper and the three servers (each in there own terminal).

ZooKeeper - Terminal 0

$ pwd
~/kafka-training/lab10/solution
bin/run-zookeeper.sh 

Kafka server 0 - Terminal 1

bin/start-1st-server.sh 

Kafka server 1 - Terminal 2

bin/start-2nd-server.sh 

Kafka server 2 - Terminal 3

bin/start-3rd-server.sh 

From the IDE or the command line run com.cloudurable.kafka.producer.StockPriceProducer.

Install the C lib, the python lib, and run then run a consumer script described below.

Install Kafka C lib and Kafka Python lib

brew install librdkafka #C Kafka Lib
pip install confluent-kafka #Confluent Python Lib

Create a simple script to consum a Stock Price.

Simple script to consume from Python stock price example (lab 10)

from confluent_kafka import Consumer, KafkaError

c = Consumer({'bootstrap.servers': 'localhost:9092,localhost:9093,localhost:9094',
            'group.id': 'myPyConsumer'
              })
c.subscribe(['stock-prices'])


running = True
while running:
    msg = c.poll()
    if not msg.error():
        print('Received message: %s' % msg.value().decode('utf-8'))
    elif msg.error().code() != KafkaError._PARTITION_EOF:
        print(msg.error())
        running = False
c.close()

Run the script.

Run test script

python consumer.py