Kafka CLI

Once working with Kafka, I often need to quickly interact with Kafka cluster via command line. This post is my collection of frequent commands used in daily work with projects having Kafka integrated.

Listing topics

1
kafka-topics --list --zookeeper localhost:2181
  • The --list switch tells the kafka-topics CLI to list all known topics.
  • The --zookeeper localhost:2181 switch tells the kafka-topics CLI where the Zookeeper ensemble Kafka is using is located. Note that in the newest versions of Kafka the --zookeeper switch is deprecated in favor of a --bootstrap-server switch that points to Kafka. The --zookeeper switch still works, but will likely be dropped in the next major revision of Kafka.

Creating topics

1
kafka-topics --create --topic "my-first-topic" --partitions 1 --replication-factor 1 --zookeeper localhost:2181
  • The switch --topic "my-first-topic" tells Kafka what to name the topic
  • The switch --partitions 1 and the switch --replication-factor 1 are required configuration
  • To check that our topic was successfully created, let’s repeat the command to list topics with a slight modification:
    1
    kafka-topics --list --zookeeper localhost:2181 --topic "my-first-topic"

Producing data

1
kafka-console-producer --topic "my-first-topic" --broker-list PLAINTEXT://localhost:9092
  • The switch --broker-list serves the same purpose as --zookeeper for the kafka-topics command, it simply tells the tool where to find Kafka.
  • Try typing out a few messages and hitting enter.
    1
    2
    3
    4
    5
    6
    7
    8
    root@6b48dc2bd81c:/# kafka-console-producer --topic "my-first-topic" --broker-list PLAINTEXT://localhost:9092
    >hello
    >world!
    >my
    >first
    >kafka
    >events!
    >

Consuming data

1
kafka-console-consumer --topic "my-first-topic" --bootstrap-server PLAINTEXT://localhost:9092 --from-beginning
  • The --from-beginning switch tells kafka-console-consumer to read data from the beginning of the topic, not just data from when we connect.
  • By default Kafka doesn’t provide historical messages to new consumers
  • You should now see output that includes all of the messages you’ve produced:
    1
    2
    3
    4
    5
    6
    7
    8
    root@6b48dc2bd81c:/# kafka-console-consumer --topic "my-first-topic" --bootstrap-server PLAINTEXT://localhost:9092 --from-beginning
    hello
    world!
    my
    first
    kafka
    events!
    hello again!

Deleting topics

1
kafka-topics --delete --topic "my-first-topic" --zookeeper localhost:2181
  • This command is nearly identical to the –create command from earlier, except now we’re calling the command with the –delete switch instead.
  • This command does not print any output if it’s successful. To check that your topic is actually deleted, list the topics one more time:
    1
    kafka-topics --list --zookeeper localhost:2181

Describe topic

1
kafka-topics --describe --zookeeper localhost:2181 --topic topic_name
  • This is helpful once there is a need of understanding how data is managed by Kafka.
The Kafka Connect API Spark fundamental

Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×