Apache Kafka Consumer
Kafka Consumers is used to reading data from a topic and remember a topic again is identified by its name. So the consumers are smart enough and they will know which broker to read from and which partitions to read from. And in case of broker failures, the consumers know how to recover and this is again a good property of Apache Kafka. Now data for the consumers is going to be read in order within each partition. Now please refer to the below image. So if we look at a Consumer consuming from Topic-A/Partition-0, then it will first read the message 0, then 1, then 2, then 3, all the way up to message 11. If another consumer is reading from two partitions for example Partition-1 and Partition-2, is going to read both partitions in order. It could be with them at the same time but from within a partition the data is going to be read in order but across partitions, we have no way of saying which one is going to be read first or second and this is why there is no ordering across partitions in Apache Kafka.
So our Kafka consumers are going to be reading our messages from Kafka which are made of bytes and so a Deserializer will be needed for the consumer to indicate how to transform these bytes back into some objects or data and they will be used on the key and the value of the message. So we have our key and our value and they’re both binary fields and bytes and so we will use a KeyDeserializer of type IntegerDeserializer to transform this into an int and get back the number 123 for Key Objects and then we’ll use a StringDeserializer to transform the bytes into a string and read the value of the object back into the string “hello world”. Please refer to the below image.
So as we can see here choosing the right Deserializer is very important because if you don’t choose the right one then you may not get the right data in the end. So some common Deserializer is given below
- String (Including JSON if your data is adjacent)I
- Integer, and Float for numbers
- Avro, and Protobuf for advanced kind of data
Apache Kafka Consumer Example
In this example, we will be discussing how we can Consume messages from Kafka Topics with Spring Boot. Talking briefly about Spring Boot, it is one of the most popular and most used frameworks of Java Programming Language. It is a microservice-based framework and building a production-ready application using Spring Boot, takes very less time. Spring Boot makes it easy to create stand-alone, production-grade Spring-based Applications that you can “just run“. So let’s start with the implementation.
Prerequisite: Make sure you have installed Apache Kafka in your local machine. Refer to this article How to Install and Run Apache Kafka on Windows?
Step 1: Go to this link and create a Spring Boot project. Add the “Spring for Apache Kafka” dependency to your Spring Boot project.
Step 2: Create a Configuration file named KafkaConfig. Below is the code for the KafkaConfig.java file.
Step 3: Create a Consumer file named KafkaConsumer
Step 4: Now we have to do the following things in order to consume messages from Kafka topics with Spring Boot
- Run the Apache Zookeeper server
- Run the Apache Kafka server
- Send the messages from Kafka Topics
Run your Apache Zookeeper server by using this command
Similarly, run your Apache Kafka server by using this command
Run the following command to send the messages from Kafka Topics
C:\kafka>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic NewTopic
Step 5: Now run your spring boot application. Make sure you have changed the port number in the application.properties file
Let’s run the Spring boot application inside the ApacheKafkaConsumerApplication file
Output: In the output, you can see when you are sending the message from Kafka Topics it is displayed on the console in real-time.
Please Login to comment...