What Is Apache Kafka?
Kafka shows up in almost every "scale to millions of events" system design discussion -- here's what it actually is.
What Is Kafka?
Apache Kafka is a distributed event-streaming platform. At its core, producers write records to named topics, Kafka durably stores them in the order they arrived, and consumers read those records independently, at their own pace. Unlike a traditional queue where a message disappears once read, Kafka retains records for a configurable period, so multiple independent consumers can each read the full stream.
How Kafka Differs From a Traditional Message Queue
- Records persist for a retention window, not just until one consumer reads them
- Multiple consumer groups can independently re-read the same stream
- Topics are partitioned, which is what lets Kafka scale to very high throughput
- Ordering is guaranteed only within a single partition, not across an entire topic
- It's built for streams of events, not just point-to-point task queues
Common Use Cases
Kafka is typically used to decouple services in an event-driven architecture (an order-service publishes an "order placed" event, and inventory, billing, and notification services each consume it independently), to move large volumes of log or metrics data, and to feed real-time analytics pipelines.