The following guide helps you go through setting up a 3 node kafka cluster using the docker-compose.
Install Docker
Take a look at my post “Running docker apps in VM which needs proxy to connect to Internet“
Downloading the confluent images
docker pull confluentinc/cp-kafka
docker pull confluentinc/cp-zookeeper
First, install docker-compose:
Docker-compose frees us from manually running each command to set up the cluster. For example, we need not run a command to set up every zookeeper host or kafka-server.
Docker-compose (at the time of writing) is a simple binary file which can be downloaded using curl or wget to your computer or VM.
The docker-compose release page contains the instruction to download the docker-compose binary.
Example (at the time of writing)
curl -L https://github.com/docker/compose/releases/download/1.23.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
You need to grant the +x (execute) permission to run the binary using chmod +x
chmod +x /usr/local/bin/docker-compose
Download the cp-docker-images
Confluent has a some docker example images which we will be going through:
git clone https://github.com/confluentinc/cp-docker-images
cd cp-docker-images/examples/kafka-cluster
This folder contains the docker-compose.yml file which we are going to use.
Start the cluster
docker-compose start
You can check the status of the containers started using the following command:
docker-compose ps
Some errors and debugging:
Sometimes, you may encounter errors where the zookeeper starts but not the kafka servers. (The above command says that the kafka is Exited with some status code i.e. the state column will not show Up for kafka-cluster) This happened (to me) due to the Connection Refused error while connecting to the socket. The problem can be solved by the following method:
Start them individually, (if the zookeeper is also not Up), then execute the following commands:
docker-compose start zookeeper-1
docker-compose start zookeeper-2
docker-compose start zookeeper-3
If the zookeeper is Up, then just execute the following commands:
docker-compose start kafka-1
docker-compose start kafka-2
docker-compose start kafka-3
Then you are ready to go.
You can tail the logs of each service using the following command:
docker-compose logs kafka-1
docker-compose logs kafka-2
docker-compose logs zookeeper-1
and so on with the service name you want to check.
You can also use the -f option, for follow.
docker-compose logs -f kafka-1
It just works like tail -f but displays the log from the starting onwards.
Note: Remember, that you need not execute docker-machine create —driver virtualbox command if you are running the docker on a Linux machine.
No comments:
Post a Comment