Kafka Retention, Ordering and When Not to Use Kafka

Why a ride appears to end before it starts

A ride-hailing trip goes through REQUESTED, ACCEPTED, STARTED and ENDED. The trip service publishes each state change to a topic with two partitions. The billing consumer sometimes sees ENDED first and charges nothing. The cause fits in a dozen lines of code: compare keying the records by trip id with sending them without a key.

import zlib
from itertools import cycle

steps = [("ride-41", "REQUESTED"), ("ride-41", "ACCEPTED"),
         ("ride-41", "STARTED"), ("ride-41", "ENDED")]

def produce(keyed):
    parts = [[], []]
    rr = cycle([0, 1])
    for key, state in steps:
        p = zlib.crc32(key.encode()) % 2 if keyed else next(rr)
        parts[p].append(state)
    return parts

def consume_interleaved(parts):
    # two partitions are read at different speeds; partition 1 is drained first
    return parts[1] + parts[0]

for keyed in (True, False):
    parts = produce(keyed)
    print("keyed" if keyed else "no key", parts, "->", consume_interleaved(parts))

keyed [['REQUESTED', 'ACCEPTED', 'STARTED', 'ENDED'], []] -> ['REQUESTED', 'ACCEPTED', 'STARTED', 'ENDED']
no key [['REQUESTED', 'STARTED'], ['ACCEPTED', 'ENDED']] -> ['ACCEPTED', 'ENDED', 'REQUESTED', 'STARTED']

The unkeyed run is a stand-in: current Kafka producers batch unkeyed records to a partition rather than strictly alternating, but the principle holds, since without a key there is no relationship between a record and its partition. Once records of one trip are spread across partitions, a consumer reading partitions at different speeds sees them in a scrambled order. Keyed by trip id, all four land in one partition and arrive in the order written.

Ordering rules you can rely on

• Order is guaranteed per partition: records are read in the order they were appended.
• To keep related events in order, give them the same key. Everything about one trip, account or device shares a key.
• Per-key order from a single producer also needs either idempotence enabled or max.in.flight.requests.per.connection=1; otherwise a retried batch can land behind a later one on older setups.
• Within a consumer group, one consumer owns a partition, so per-key processing stays sequential. If you hand records to a thread pool inside the consumer, you give that guarantee up unless you shard the pool by key.
• There is no ordering between topics. If a trip's state changes and its payments are in two topics, do not assume cross-topic order; carry a version or timestamp in the payload.

Retention: how long Kafka keeps data

A topic's default cleanup policy is delete: old data is removed by retention. Two limits apply and whichever is reached first wins. retention.ms is the age limit (the broker default is seven days) and retention.bytes is a size limit per partition (unlimited by default). Deletion works on whole segment files, not on individual records, and the segment currently being written is never deleted. So a record can outlive retention.ms until its segment rolls over, governed by segment.ms and segment.bytes. Do not rely on retention as a precise timer.

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name ride-events --add-config retention.ms=86400000,segment.bytes=268435456

Completed updating config for topic ride-events.

That sets one day of retention and 256 MiB segments (the output line is typical but its wording can vary by version). Consider the consumer side too: a consumer group that is offline for longer than the retention period resumes to find its offset no longer exists. What happens next is decided by auto.offset.reset: earliest silently reprocesses everything still held, latest silently skips the gap, and none throws an error. Pick deliberately.

Log compaction: keep the latest value per key

Set cleanup.policy=compact and Kafka keeps, at minimum, the latest record for every key instead of deleting by age. It suits topics that represent current state, such as a product-price catalog, where history is noise. A record with a key and a null value is a tombstone: it marks the key for deletion, and the tombstone itself is removed after delete.retention.ms. Compaction runs in the background, so readers may still see older values for a while, and the active segment is not compacted. Offsets of surviving records never change, which is why gaps appear.

# (offset, key, value); value None is a tombstone (delete marker)
log = [
    (0, "sku-1", "price=100"),
    (1, "sku-2", "price=250"),
    (2, "sku-1", "price=95"),
    (3, "sku-3", "price=40"),
    (4, "sku-2", None),
    (5, "sku-1", "price=90"),
]

def compact(records):
    latest = {}
    for off, key, val in records:
        latest[key] = off
    # keep only the newest record per key; offsets of survivors never change
    kept = [r for r in records if latest[r[1]] == r[0]]
    return [r for r in kept if r[2] is not None]   # real Kafka drops tombstones later, after delete.retention.ms

def retention(records, keep_last):
    return records[-keep_last:]

print("compacted :", compact(log))
print("time/size retention keeping last 3:", retention(log, 3))

compacted : [(3, 'sku-3', 'price=40'), (5, 'sku-1', 'price=90')]
time/size retention keeping last 3: [(3, 'sku-3', 'price=40'), (4, 'sku-2', None), (5, 'sku-1', 'price=90')]

Designing with Kafka, and when not to

Kafka is excellent when you need a durable, replayable, ordered-per-key stream with many independent readers and high write volume: event feeds between services, change-data capture, clickstreams, audit trails and stream processing. It asks something in return: you operate a cluster (or pay for a managed one), you reason about partitions and consumer groups, and you give up per-message features that simpler queues provide.

A useful test before adopting it: write down who reads this data besides the first consumer, whether they need to replay it, and what ordering they need. If the honest answers are "nobody", "no" and "none", you probably want something simpler. For where messaging sits among caching and storage choices, see Redis for a lighter-weight option and System Design for the surrounding trade-offs.

Common mistakes

• Treating Kafka as a database of record without a plan. With default retention, data silently ages out after a week. Symptom: a replay for a new service finds only recent events.
• Compacting a topic that needs history. After compaction, the intermediate states are gone for good. Symptom: an audit request for "what was the price last Tuesday" cannot be answered.
• Choosing a key with low cardinality or heavy skew. One tenant or a default key dominates a partition. Symptom: a single consumer is pinned while the others idle and lag grows on one partition only.
• Putting a thread pool in the consumer without key-sharding. Per-key order silently breaks under load while passing every test at low volume.
• Using Kafka for request/response. Developers build reply topics and correlation ids, then fight timeouts and ordering instead of making an HTTP call.

How this shows up in real projects and interviews

In a design interview, naming Kafka is the easy part. Interviewers look for the follow-ups: what is the key, how many partitions, what happens when a consumer is down for a day, what is the retention, and how would you replay after a bug. Strong candidates also say when they would not use it. See the system design interview guide for how to structure those answers. On the job, the most valuable habit is writing a one-page topic spec: purpose, key, partition count, retention or compaction choice, expected consumers and their lag alerts, so the next engineer does not have to reverse-engineer it.