Apache Kafka is a tool for transmitting and processing data streams in real time. In this article, I will explain in a simple way how Kafka works and what it can be used for. I will show how data streams flow through the system, the problems Kafka solves, and why it is so popular.

What is Apache Kafka?

Apache Kafka can be compared to a super-fast intermediary between the data sender (producer) and the data receiver (consumer).

It is a distributed system and can run on multiple servers (called brokers) simultaneously, which allows it to handle enormous amounts of data. Additionally, it is highly scalable and fault-tolerant, making it ideal for modern, demanding applications.

The Origins of Apache Kafka

Imagine the year 2010. In LinkedIn offices, engineers are struggling with a flood of data. Every click, like, and profile view generates a stream of information growing at a rapid pace. Traditional queuing systems, which once worked flawlessly, start to fail. They are too slow, inefficient, and lack the flexibility to handle this scale. Instead of smooth data flow, a digital traffic jam threatens to paralyze operations.

At this critical moment, three engineers—Jay Kreps, Neha Narkhede, and Jun Rao—stepped in. Instead of searching for another “quick fix,” they decided to build something from scratch: a system capable of processing massive amounts of data in real time without losing performance. They wanted a platform that could not only meet current needs but also grow freely with the company.

The concept that emerged was revolutionary. Instead of a queue that deletes messages after processing, they created a durable, distributed log that multiple systems could access repeatedly at any time. The system had to be:

• Incredibly fast – capable of processing millions of events per second.
• Scalable – able to grow indefinitely without losing performance.
• Resilient – storing data safely and fault-tolerantly.

The name of the system, Kafka, was chosen by Jay Kreps, who was fascinated by Franz Kafka’s work.

How Apache Kafka Works

Apache Kafka works like a mail system for data—but on a much larger scale and at lightning speed. Data in Kafka flows between three main components: producers, brokers, and consumers.

• Producer: An application or system that sends data (e.g., user clicks in an app, system logs, or online store transactions).
• Broker: A server that stores and manages this data. In practice, there is a whole cluster of brokers working together to handle millions of events per second.
• Consumer: An application that receives data and processes it—for example, storing it in a database, analyzing it, or triggering further business processes.

Data in Kafka is stored in special “channels” called topics. Each topic can be divided into partitions, which allows parallel processing of huge amounts of data and distributes the load across multiple servers.

Importantly, Kafka does not delete data immediately after it is read. Instead, it stores data for a specified period, allowing different systems to retrieve it independently at any time. This makes Kafka not only fast but also flexible and reliable, enabling the creation of applications that react to data in real time without risk of losing information.

Key Concepts in Apache Kafka

Before you start using Kafka, it’s worth understanding a few basic elements:

Topic:
A place where all messages are sent. Think of it as a “subject” or “channel” of data. For example, in an e-commerce application, one topic could store new orders, while another stores user logs.

Partition:
Each topic can be divided into partitions. Partitions allow data to be split into smaller chunks for parallel processing, enabling Kafka to handle massive data volumes without slowing down.

Offset:
A number assigned to each message in a partition. The offset tells Kafka which messages have already been read by a consumer. You can imagine it as numbered rows in a table—Kafka always knows where to start reading.

Producer:
An application or system that sends data to Kafka. For example, a mobile app sending user click data or a payment system sending transaction data.

Consumer:
An application that receives data from Kafka and processes it. One topic can have multiple consumers, each retrieving data independently.

Broker:
A server that stores data in Kafka. Usually, multiple brokers work together in a cluster for scalability and fault tolerance.

Cluster:
A group of brokers working together. Kafka can process millions of messages per second and remain reliable even if one broker fails.

Practical Applications of Apache Kafka

Kafka is used wherever fast and reliable data flow is required. Some examples include:

Recommendation systems in e-commerce
When browsing products on Amazon or Allegro, Kafka transmits click and purchase data in real time to the recommendation system, which immediately suggests relevant products.

Real-time log analysis
In large applications, such as streaming services or social media platforms, every error or event is sent to Kafka instantly. Monitoring systems can detect problems and alert administrators immediately.

Streaming platforms and social media:
Kafka serves as the backbone of feeds in apps like TikTok or YouTube. Data on new videos, comments, or likes is transmitted in real time to various systems and users.

Microservices and system integration:
In modern apps, data flows to multiple independent systems. Kafka allows a single data stream to be consumed by several applications simultaneously—analytics, notifications, and databases without losing information.

Advantages and Disadvantages of Kafka

Apache Kafka is highly valued for its speed and reliability. Data is stored in a distributed system and can be read by multiple consumers simultaneously, making Kafka ideal for applications processing millions of events in real time. It is highly scalable—new servers can be added as data grows—and fault-tolerant, meaning that even if a broker fails, the system continues to operate without data loss. Additionally, Kafka allows flexible data reading: consumers can fetch data multiple times at any moment without affecting others.

On the downside, Kafka has limitations. Building and configuring the system can be complex, especially for beginners. A poorly designed pipeline can lead to performance issues or broker overload. The system requires constant monitoring and disk space planning, as large data streams can quickly consume significant storage. Despite these challenges, Kafka’s benefits make it one of the most widely used tools for real-time data processing.

Getting Started with Apache Kafka

Installation and Running Kafka (Docker)

A ready-to-use Docker file for Apache Kafka can be found in this GitHub repository. You can download and run it locally to follow the examples in this article.

Clone the repository

git clone https://github.com/dawidfila/apache-kafka-docker

Enter the project directory

cd kafka-docker-tutorial

This directory contains a docker-compose.yml file that launches Kafka along with all necessary components.

Start Kafka in Docker

docker-compose up -d

The -d flag runs containers in the background. After a few seconds, Kafka will be ready.

Check if Kafka is running

docker ps

You should see the Kafka container running on port 9092.

Stop Kafka

When you are finished experimenting or want to shut down use:

docker-compose down

Basic Kafka Commands

Once Kafka is running, enter its container:

docker exec -it kafka bash

1. Creating a Topic

Topics are the basic units where messages go in Kafka. Create a topic named my-first-topic:

kafka-topics --create --topic my-first-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Here we define the number of partitions (3) and the replication factor (1).

Partitions – imagine a topic as a large cabinet for messages. Partitions are the shelves in that cabinet.
If you set up 3 partitions, the messages are distributed among them (e.g., evenly or based on a key). This allows Kafka to operate faster and in parallel – multiple consumers can read different partitions simultaneously.

Replication – these are backup copies of a topic. If you set the replication factor to 1, it means there are no backup copies (everything exists on a single broker). If that broker fails, you lose the data. In a production environment, the replication factor is usually set to 3 or more, so that data is duplicated across multiple servers.

2. Checking Available Topics

To ensure that our topic has been created, we can list all existing topics:

kafka-topics --list --bootstrap-server localhost:9092

3. Topic Details

If you want to see the exact configuration of your topic, use the following command:

kafka-topics --describe --topic my-first-topic --bootstrap-server localhost:9092

This will show you details about the partitions, replicas, and leaders.

4. Producer – Sending Messages

Now it’s time for something practical: let’s send the first messages to the topic. To do this, start the producer:

kafka-console-producer --topic my-first-topic --bootstrap-server localhost:9092

After running this command, you can type messages in the console and press Enter – each message is sent directly to Kafka.

5. Consumer – Receiving Messages

Of course, the sent data also needs to be received. In a new terminal, start the consumer:

kafka-console-consumer --topic my-first-topic --from-beginning --bootstrap-server localhost:9092

Now you will see all the messages appearing in the topic – both the historical ones and the new ones.

6. Consumer Groups

So far, in our examples, we have used a single consumer that received all the messages from the topic. In practice, we often want multiple applications or instances to work in parallel, sharing the messages among themselves – and this is where consumer groups come in.

A Consumer Group is simply a group of consumers that work together to read messages from a topic. Kafka automatically distributes the partitions among the consumers in the group, ensuring that:

• Each consumer in the group receives a different portion of the messages (messages from a single partition are sent to only one consumer in the group).
• The system can scale horizontally – adding new consumers to the group allows faster data processing, provided there are enough partitions.
• Consumers in different groups can independently read the same messages from the topic – each group acts as a separate “data receiver.”

Example of Using Consumer Groups in Practice

Let’s assume you have a topic called my-first-topic with 3 partitions and you want to see how consumer groups work.

1. Start Two Consumers in the Same Group
In the first terminal (inside the Kafka container), enter:

kafka-console-consumer --topic my-first-topic --group group1 --bootstrap-server localhost:9092

In the second terminal, start another consumer in the same group:

kafka-console-consumer --topic my-first-topic --group group1 --bootstrap-server localhost:9092

What happens?

• Kafka automatically distributes the partitions between the two consumers.
• Each consumer receives only a portion of the messages – there are no duplicates.
• This allows the system to process multiple messages in parallel.

2. Sending Messages via the Producer
In a new terminal, start the producer:

kafka-console-producer --topic my-first-topic --bootstrap-server localhost:9092

Now type messages and press Enter. You will see that Consumer A receives one portion, while Consumer B receives a different portion of the messages – exactly how Kafka distributes them among the group members.

3. Viewing the Consumer Group
You can check the list of groups and their details:

# List of all groups
kafka-consumer-groups --bootstrap-server localhost:9092 --list

# Details of the group "group1"
kafka-consumer-groups --bootstrap-server localhost:9092 --group group1 --describe

In the details, you will see which partitions are assigned to each consumer and the current state of their offsets.

Additional Useful Commands

When working with Apache Kafka, it’s useful to know a few additional commands that help manage topics, check offsets, and send messages with keys.

Deleting a Topic

If you need to delete an existing topic, use the following command:

kafka-topics --delete --topic topic-name --bootstrap-server localhost:9092

After executing this command, the topic will be removed from the Kafka cluster. Keep in mind that all messages in the topic will be lost, so use this command with caution.

Checking Offsets

Offsets show which messages have already been read by consumers in a given partition. To check the offsets for a topic, you can use the following tool:

kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic my-first-topic

The result will show the number of the last message in each partition – this is useful for monitoring consumers and processing delays.

Producer with a Key

Sometimes we want messages with a specific key to always go to the same partition. In such cases, you can start the producer with the option --property parse.key=true:

kafka-console-producer --topic my-first-topic --bootstrap-server localhost:9092 --property "parse.key=true" --property "key.separator=:"

• After starting, you enter messages in the format: key:message
• Kafka will use the key to deterministically assign the message to a specific partition.

How Kafka Assigns Messages to Partitions

It’s important to understand that Kafka does not send messages randomly to consumers in a group. The message key determines this. The mechanism works in a simple way:

1. Kafka takes the message key (e.g., order_id).
2. It calculates a hash from it → an integer.

It selects the partition using the formula: partition = hash(key) % number_of_partitions

• Each partition is assigned to a single consumer in the group.
• This ensures that all messages with the same key always go to the same partition → and therefore always to the same consumer.
• Messages with different keys can go to different partitions, and consequently, to different consumers in the group.

This allows effective control over data flow while enabling parallel processing of multiple messages within the same topic.

Mini-Project – E-Commerce Order Simulation in Apache Kafka

Now I’ll show you in practice how partitions, keys, consumer groups, and offsets work, using simple e-commerce data instead of artificial strings.

Project Goal

• Topic: orders – simulating an online store’s orders
• Partitions: 3 → to demonstrate consumer parallelism
• Data: random orders with order_id, amount, category
• 2 producers → send orders with the order_id as the key
• 2 consumers in one group → demonstrate how Kafka distributes partitions
• Observation: offsets and rebalancing when a consumer is stopped

Creating the Topic with 3 Partitions

kafka-topics --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

• 3 partitions → for testing parallelism
• RF=1 → local test, one broker is sufficient

Each partition allows different consumers to read data in parallel.

Producers – Order Simulation

Python Producer Example:

from kafka import KafkaProducer
import json
import time
import random

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    key_serializer=lambda k: str(k).encode('utf-8')
)

categories = ['books', 'electronics', 'clothes', 'food']

order_id = 1
while True:
    order = {
        'order_id': order_id,
        'amount': round(random.uniform(10, 500), 2),
        'category': random.choice(categories)
    }
    producer.send('orders', key=order_id, value=order)
    print(f"Sent order: {order}")
    order_id += 1
    time.sleep(0.5)

• key = order_id → deterministically sent to a single partition
• You can run 2 producers in separate terminals to simulate parallelism

Consumers in One Group

Consumer A:

kafka-console-consumer --topic orders --group group1 --bootstrap-server localhost:9092 --property print.key=true

Consumer B:

kafka-console-consumer --topic orders --group group1 --bootstrap-server localhost:9092 --property print.key=true

• Kafka automatically distributes the partitions among the consumers:
  • Consumer A → partitions 0 and 1
  • Consumer B → partition 2

This way, each consumer reads only a portion of the messages, and the system works in parallel.

Practical Experiments

Observing Message Distribution:

• Watch which consumer receives which orders
• Try changing order_id and see how the key hash determines the partition

Stopping a Consumer:

Press Ctrl+C in the terminal → Kafka triggers a rebalance → the remaining consumer takes over the stopped consumer’s partitions

Checking Offsets:

kafka-consumer-groups --bootstrap-server localhost:9092 --group group1 --describe

• Shows CURRENT-OFFSET and LAG for each partition

Adding a New Consumer

• Kafka triggers a rebalance → partitions are distributed among 3 consumers

Summary

Apache Kafka is a distributed system for real-time data streaming and processing, acting as a mediator between producers and consumers. Data is stored in topics divided into partitions, allowing parallel and reliable processing. Kafka is fast, scalable, fault-tolerant, and supports multiple reads of the same data. It is used in e-commerce, log analysis, streaming platforms, and microservices integration.