Day 97: Kafka conceptual: partitions, offsets, consumer groups
Kafka: a distributed, ordered commit log
Kafka's core abstraction is different from RabbitMQ's: a topic is split into partitions, each an ordered, append-only log. A message's position in a partition is its offset. Unlike a traditional queue, consuming a message doesn't remove it — multiple consumers (or the same consumer, replaying) can read the same partition independently, each tracking their own offset.
A consumer group lets multiple consumer instances split a topic's partitions between them for parallel processing — each partition is only consumed by one member of the group at a time, but a topic with many partitions can be processed by many consumers in parallel.
ISR (In-Sync Replicas) are the set of replicas fully caught up with a partition's leader — Kafka only considers a write "committed" once it's replicated to enough ISR members, directly applying the quorum/replication concepts from Phase 7.
Key terms
- Partition
- An ordered, append-only log that a Kafka topic is split into.
- Offset
- A message's position within a partition.
- Consumer group
- A set of consumers splitting a topic's partitions for parallel processing.
Why can two completely different consumer groups read the same Kafka topic independently, each seeing all messages?
