Kafka CLI

Once working with Kafka, I often need to quickly interact with Kafka cluster via command line. This post is my collection of frequent commands used in daily work with projects having Kafka integrated.

Listing topics

kafka-topics --list --zookeeper localhost:2181

• The --list switch tells the kafka-topics CLI to list all known topics.
• The --zookeeper localhost:2181 switch tells the kafka-topics CLI where the Zookeeper ensemble Kafka is using is located. Note that in the newest versions of Kafka the --zookeeper switch is deprecated in favor of a --bootstrap-server switch that points to Kafka. The --zookeeper switch still works, but will likely be dropped in the next major revision of Kafka.

Creating topics

kafka-topics --create --topic "my-first-topic" --partitions 1 --replication-factor 1 --zookeeper localhost:2181

• The switch --topic "my-first-topic" tells Kafka what to name the topic
• The switch --partitions 1 and the switch --replication-factor 1 are required configuration
• To check that our topic was successfully created, let’s repeat the command to list topics with a slight modification:

  kafka-topics --list --zookeeper localhost:2181 --topic "my-first-topic"

Producing data

kafka-console-producer --topic "my-first-topic" --broker-list PLAINTEXT://localhost:9092

• The switch --broker-list serves the same purpose as --zookeeper for the kafka-topics command, it simply tells the tool where to find Kafka.
• Try typing out a few messages and hitting enter.

  root@6b48dc2bd81c:/# kafka-console-producer --topic "my-first-topic" --broker-list PLAINTEXT://localhost:9092
  >hello
  >world!
  >my
  >first
  >kafka
  >events!
  >

Consuming data

kafka-console-consumer --topic "my-first-topic" --bootstrap-server PLAINTEXT://localhost:9092 --from-beginning

• The --from-beginning switch tells kafka-console-consumer to read data from the beginning of the topic, not just data from when we connect.
• By default Kafka doesn’t provide historical messages to new consumers
• You should now see output that includes all of the messages you’ve produced:

  root@6b48dc2bd81c:/# kafka-console-consumer --topic "my-first-topic" --bootstrap-server PLAINTEXT://localhost:9092 --from-beginning
  hello
  world!
  my
  first
  kafka
  events!
  hello again!

Deleting topics

kafka-topics --delete --topic "my-first-topic" --zookeeper localhost:2181

• This command is nearly identical to the –create command from earlier, except now we’re calling the command with the –delete switch instead.
• This command does not print any output if it’s successful. To check that your topic is actually deleted, list the topics one more time:

  kafka-topics --list --zookeeper localhost:2181

Describe topic

kafka-topics --describe --zookeeper localhost:2181 --topic topic_name

• This is helpful once there is a need of understanding how data is managed by Kafka.