Docker Kafka

How to Install Apache Kafka Using Docker?

Let us start the article by understanding in detail the entire process of installing Apache Kafka via the docker containers.

As a pre-requisite, you shall be needing two Kafka Docker images for running the Apache Kafka. Below are two sample images that you can utilize for the same.

1. Sample Image One
2. Sample Image Two

Now the above images need not be downloaded manually. You can use the ‘docker-compose.yml’ shall do it for you automatically.

The entire code is given below:

Above code, you can edit the ports if 2782 or 8954 is not available for your system. Then, you can open a Terminal and search for the folder where you have saved the ‘docker-compose.yml’ file.

For pulling the images and creating the docker containers, you can execute the below command:

‘docker-compose -f docker-compose.yml up -d’

When you use the ‘-d’ flag both the Kafka Zookeeper and Kafka Broker execute in the background. Once the command is successfully run after they are about to start you can access the Terminal. You will now see the configuration and download are successfully printed to the Terminal.

By executing the ‘docker ps’ command, you can verify whether both Zookeeper and Kafka are up and running.

Single Node Setup

With a single node Kafka broker setup, users can meet almost many of the local development needs, let's quickly dive into how to do so one by one.

docker-compose.yml Configuration

Let us start an Apache Kafka server. To do so, we need to get the ZooKeeper server started.

• Users can easily configure the dependency in the ‘docker-compose.yml’ file. This way it is always taken care of beforehand that the Zookeeper server is started way before the Kafka server and eventually stops after the Kafa has stopped.

• Without must ado, let us quickly get started with a simple ‘docker-compose.yml’ file having the two services, that are the ZooKepper and Kafka:

You can implement the below code to do so:

• As seen in the above setup, the ZooKeeper currently could be seen listening from the port=2782 for the Kafka service, The same is specified in the Docker container setup. But, for other clients running on the same host, it shall still be exposed over the 22782 port.

• The Kafka service is also exposed to the host systems over the 25432 port, but actually, if you look deeper it is advertised over the 8954 port within the Docker environment. This docker container environment is configured via the ‘KAFKA_ADVERTISED_LISTENERS’ property.

Start Kafka Server

Let us start the Kafka server now. ​​

• You can start spinning up the docker container using the below command to start the Kafka server:

‘$ docker-compose up -d’

• Once done, you now need to implement the ‘nc‘ command for validating that both servers are correctly listening over their respective ports:

• In addition, you can also validate and verify from the verbose logs while in the background the containers are starting and validating if the Kafka server is up and running via the following command:

$ docker-compose logs kafka | grep -i started

Hurray! The Kafka setup is up and running and you can now utilize it.

Connection Using Kafka Tool

Lastly, we shall now be setting up connections via the Kafka tool. For this user can implement the Kafka Tool GUI utility. With this, users can easily establish the connection with the Kafka server we just created and visualize the Kafka server setup when needed.

It's recommended that users should implement the Bootstrap servers property to establish the connection with the Kafka server that listens from the listening at port 25432 at the host.

While doing so, you should also find that the Topics and Consumers entries are empty. It is so as this is a brand new setup you are doing on your machine. When you create the topics, visualizing the data across partitions should come easily. You shall also be able to view the details for any active users connected with the Kafka server.

Kafka Cluster Setup

As organizations expand, the need for stable environments increases and hence users need resilient Kafka setup too. As studied so far, the docker-compose.yml configuration, Kafka server along with the Kafka tool we learned how we can set it up for a single cluster.

Now we shall be exploring how we could expand our knowledge into the multi-node Kafka cluster setup and redefine the docker-compose.yml file for the same.

docker-compose.yml Configuration

For a multinode Kafka setup, you need to have redundancy added for both the servers that is, Zookeeper as well as the Kafka servers.

Let us dive into adding the configuration into the docker-compose.yml file when our use case is dealing with more than one node each for Kafka and ZooKeeper servers.

It is recommended that the KAFKA_BROKER_ID along with the service names remain distinct across the services. As seen below, port 2782 is being utilized by zookeeper-first and zookeeper-second for listening, and they are the ones exposing port 2782 to the host through ports 22782 and 32782 respectively.

The host machines take only a distinct port for each service exposed. Similarly, Kafka-one and Kafka-two, are the ones exposing port 2782 to the host through ports 55432 and 45432 respectively.

How to run Kafka with Docker?

• To execute the Kafka with the docker container, users can make use of the Docker Compose tool. With the Docker compose tool, users can define and run multi-container Docker applications. The Docker Compose tool turns out to be a perfect tool when we need to start executing each of the Kafka services, like the broker, zookeeper, etc. in various Docker containers.
• For using the docker-compose tool, users need to explicitly specify the Kafka service that should be together to make up the Kafka deployment in the ‘docker-compose.yml’ file. This can then be executed together in an isolated environment.
• First, you could visit the link to the GitHub project GitHub project.
• Now you need to clone or you can even download the entire project. This project has various docker-compose files serving the purpose of many configurations. 1 git clone https://github.com/conduktor/kafka-stack-docker-compose.git
• Once you see 100% clone complete, your project is perfectly cloned. Resolving deltas: 100% (375/375), done.
• Now users need to navigate to the ‘kafka-stack-docker-compose’ from where users can start listing the various files.
• For listing the files, you can use the command $ ls -l
• And you shall see various files like

• Now, start looking for the file ‘kafka-single-zk-single.yml’
• This is the main file that we shall be utilizing for launching a Kafka cluster containing a single Zookeeper along with a single broker.
• Then, run the below command for launching the Kafka cluster having one Zookeeper and one Kafka broker. If you use the ‘-d’ flag the entire execution will run in the background. 1 docker-compose -f kafka-single-zk-single.yml up -d
• To validate if both (Kafka Zookeeper and Kafka Broker are executing), use the below command to know: $ docker-compose -f kafka-single-zk-single.yml ps
• Once you run the command you shall see the state as UP which tells that both the services Kafka Zookeeper and Kafka Broker are Up and running. Additionally, the Kafka shall be exposed at the ‘localhost:9092’ on your system/ computer.

Running Commands Against Our Kafka Running on Docker

In this section, we shall be running commands against the docket Kafka which is still in progress over the docker. To do so, we generally have two ways:

• Directly execute Kafka commands from within the Docker container via the ‘docker exec’.
• Install binaries and execute commands from the host OS.

Let us see each in action one by one:

Executing the Commands From Within the Kafka Docker Container

`1 docker exec -it kafka1 /bin/bash

Once this command is executed, from within the docker container, we can start executing some Kagka commands as shown below. Now we don't need to use ‘.sh’.

1 [testusers@kafka1 ~]$ kafka-topics --version 2 6.2.1-ccs (Commit:fa4bec046a2df3a6)

Executing Kafka Commands From Outside the Docker Container:

Before you start executing the command make sure you have the required Java and the Kafka Binaries installed on your system:

Once you have completed your installation of the binaries, you shall see the Kafka binary in your command line interface path. Now you can start executing your Kafka command, such as 1 Kafka-topics. sh

Stopping Kafka on Docker

• Once the installation is verified, the user can move ahead to stop the ongoing services by eventually stopping the respective docker containers:

1 $ docker-compose -f zk-single-kafka-single.yml stop

• For removing the resources such as the docker altogether with other ongoing resources, users can simply implement utilizing the ‘down’ operation instead of ‘stop’.