Apache Kafka Topics, Partitions and Offsets: How the Log Works

Six events from a network of parcel-locker kiosks are written to a topic that has three partitions. Before any explanation, look at where each one ended up, using a tiny Python model of the storage layer.

import zlib

class Topic:
    def __init__(self, name, partitions):
        self.name = name
        self.parts = [[] for _ in range(partitions)]

    def append(self, key, value):
        # Real Kafka uses murmur2 on the key bytes; crc32 stands in here.
        p = zlib.crc32(key.encode()) % len(self.parts)
        self.parts[p].append((key, value))
        return p, len(self.parts[p]) - 1  # (partition, offset)

    def read(self, partition, offset, max_records=10):
        return self.parts[partition][offset:offset + max_records]

orders = Topic("parcel-events", partitions=3)
events = [
    ("locker-7", "PARCEL_DROPPED"),
    ("locker-2", "PARCEL_DROPPED"),
    ("locker-7", "CODE_SENT"),
    ("locker-9", "PARCEL_DROPPED"),
    ("locker-7", "PARCEL_COLLECTED"),
    ("locker-2", "DOOR_JAMMED"),
]
for key, value in events:
    p, off = orders.append(key, value)
    print(f"{key:9} {value:17} -> partition {p}, offset {off}")

print()
for p in range(3):
    print(f"partition {p}: {[v for _, v in orders.parts[p]]}")

print()
# Two independent readers keep their own position; the log is not consumed.
analytics_pos = 0
billing_pos = 0
batch = orders.read(0, analytics_pos)
analytics_pos += len(batch)
print("analytics read", len(batch), "records from p0, next offset", analytics_pos)
batch = orders.read(0, billing_pos, max_records=1)
billing_pos += len(batch)
print("billing read", len(batch), "record from p0, next offset", billing_pos)
print("p0 still holds", len(orders.parts[0]), "records")

locker-7  PARCEL_DROPPED    -> partition 0, offset 0
locker-2  PARCEL_DROPPED    -> partition 0, offset 1
locker-7  CODE_SENT         -> partition 0, offset 2
locker-9  PARCEL_DROPPED    -> partition 1, offset 0
locker-7  PARCEL_COLLECTED  -> partition 0, offset 3
locker-2  DOOR_JAMMED       -> partition 0, offset 4

partition 0: ['PARCEL_DROPPED', 'PARCEL_DROPPED', 'CODE_SENT', 'PARCEL_COLLECTED', 'DOOR_JAMMED']
partition 1: ['PARCEL_DROPPED']
partition 2: []

analytics read 5 records from p0, next offset 5
billing read 1 record from p0, next offset 1
p0 still holds 5 records

Three things in that output are the whole foundation of Kafka. Every record for locker-7 went to partition 0, in the order it was written. Offsets restart from 0 in each partition, so offset 0 exists in partition 0 and again in partition 1. And reading did not remove anything: two readers kept their own positions and the partition still holds all five records afterwards.

A log, not a queue

A traditional queue hands a message to one worker and deletes it once acknowledged. Kafka does something different. A partition is an append-only file (physically a series of segment files on the broker's disk). Producers only ever add to the end. Each record gets the next sequential number in that partition, its offset. Consumers do not take records out; they remember a position and read forward from it.

That one design decision explains most of Kafka's behaviour. Because reading is non-destructive, many independent applications can read the same data at their own pace: the analytics job, the billing job and a fraud checker above are three readers of one log. Because a reader's position is just a number, a crashed reader can resume exactly where it stopped, and a new reader can start from the beginning to rebuild its state. Records disappear only when the retention policy removes old segments, which you will configure in the last lesson.

The vocabulary, with what is unique where

The practical consequence: a record's true address is the triple (topic, partition, offset). Saying "offset 41" without a partition is like saying "seat 41" without a row. This matters later when you need to deduplicate or report a poison message.

Keys decide partitions, and partitions decide ordering

Kafka guarantees order within a partition and nothing across partitions. So the question "which records must stay in order?" becomes "which records must share a key?". If every event about one locker carries the locker id as its key, they all hash to the same partition and are read back in write order. Events about different lockers may interleave arbitrarily, and that is usually fine.

Notice that the demo is lopsided: partition 0 got five of six records and partition 2 got none. With few distinct keys, hashing does not balance anything. Skew like this also appears in production when one key (a very busy tenant, a default value like unknown) dominates, producing a hot partition that one consumer must handle alone. It is the same hot-shard problem described in Databases in System Design.

Replication in one paragraph

Each partition has a replication factor: a leader replica that handles all reads and writes, plus follower replicas on other brokers that copy it. The followers that are caught up form the in-sync replica set (ISR). If the leader's broker dies, one in-sync follower is promoted. A replication factor of 3 is the common production choice because it survives the loss of a broker while leaving two copies; a factor of 1 (fine on a laptop) means a dead disk is lost data. Recent Kafka versions coordinate the cluster with their own KRaft quorum instead of ZooKeeper.

Doing it on a real broker

Kafka is not installed in this course environment, so treat the following as the commands you would run against a local broker listening on port 9092. The scripts ship in the bin folder of the Kafka download; the .sh suffix is part of the name on Linux and macOS, and Windows uses the .bat equivalents under bin\windows.

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic parcel-events --partitions 3 --replication-factor 1
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic parcel-events

Created topic parcel-events.
Topic: parcel-events    PartitionCount: 3    ReplicationFactor: 1
    Partition: 0    Leader: 1    Replicas: 1    Isr: 1
    Partition: 1    Leader: 1    Replicas: 1    Isr: 1
    Partition: 2    Leader: 1    Replicas: 1    Isr: 1

The describe output above is abbreviated and illustrative: real output also prints a topic id and a configuration column, and the exact layout varies between versions. What matters is reading it: each partition has a leader broker id, the list of replicas, and the ISR. If Isr is shorter than Replicas, a follower is lagging or down.

Common mistakes

• Treating offsets as global ids. Using the offset alone as a unique key in a database causes silent overwrites, because partition 0 and partition 1 both have an offset 0. Use (topic, partition, offset).
• Expecting global ordering. Developers see ordered output on a one-partition dev topic, then ship to a twelve-partition topic and watch events for the same entity arrive out of order. The symptom is a state machine receiving ENDED before STARTED. Fix it by keying on the entity id.
• Sending everything with a null key and then needing per-entity order. Records are spread across partitions with no relation to the entity, so no consumer can rely on order.
• Creating thousands of partitions "for future scale". Each partition costs file handles, replication traffic and recovery time on the brokers, and more partitions do not help if you have only a few consumers.
• Assuming reads delete data. A new engineer "cleans up" a topic by consuming it, then is surprised it is still full. Only retention (or compaction) removes records.

How this shows up in real projects and interviews

In interviews, the opening Kafka question is almost always some form of "how do you guarantee ordering for one user?" and the expected answer is the key-to-partition mapping above, plus the admission that ordering is per partition. In real projects, the first design decision on any new topic is the key. Get it right and consumers can be stateless per record; get it wrong and you end up building reordering logic downstream. Partition count is the second decision, and it is usually set by the maximum parallelism you expect, which the next lesson on consumer groups explains.