Apache Kafka

What is Apache Kafka?

Apache Kafka is a distributed event-streaming platform — at its core, a durable, ordered, replayable log of events that producers append to and many consumers read from independently. Unlike a traditional message queue that hands a message to one consumer and deletes it, Kafka retains the stream (for a configured time or size), so multiple independent consumers can each read it at their own pace and even re-read from any point. This makes it the backbone of high-throughput data pipelines, event-driven architectures, event sourcing, log aggregation, and real-time analytics at companies operating at massive scale.

The log model — why it’s different

The mental model that unlocks Kafka is “distributed commit log,” not “queue.” Events are written to topics, which are split into partitions for parallelism and scale; within a partition, events are strictly ordered and each has an offset. Consumers track their own offset, so they control where they are in the stream — this is what enables replay (reprocess history by resetting the offset), multiple independent consumer groups reading the same data for different purposes, and decoupling producers from consumers entirely. Kafka replicates partitions across brokers for durability and availability, so a broker failure doesn’t lose data. This design delivers enormous throughput and durable, replayable streams — but it also means Kafka is operationally heavier than a simple queue and is overkill when you just need to distribute tasks to workers.

Kafka vs a message queue — the decision that matters

The perennial choice is Kafka vs RabbitMQ/SQS-style message queues, and they solve different problems. A message queue is a smart broker for task distribution: it pushes a message to one consumer, tracks per-message acknowledgement, and typically removes the message once processed — ideal for work queues, job processing, and complex routing at moderate volume. Kafka is a durable event log for high-throughput streaming where many consumers read independently and history matters — event pipelines, streaming analytics, event sourcing. Choose Kafka when you need replayability, multiple independent consumers, very high throughput, and an ordered event history; choose a queue when you need a work queue with flexible routing and per-message ack at manageable scale. The classic mismatches: using Kafka as a simple task queue (drowning in its operational complexity for no benefit), or forcing a queue to be a high-throughput replayable stream. Managed options (Confluent Cloud, AWS MSK) remove much of the operational burden that makes self-hosted Kafka demanding.

What people get wrong

• Using Kafka as a task queue — its log model shines for replayable, multi-consumer streaming; for simple work distribution a message queue is far simpler to run.
• Ignoring partition/key design: ordering is only guaranteed within a partition, and a bad partition key destroys parallelism or ordering guarantees you assumed you had.
• Underestimating operational weight — self-hosted Kafka (brokers, replication, rebalancing, retention tuning) is demanding; use a managed service unless you have the expertise.

Primary source: Apache Kafka documentation