Kafka Delivery Semantics: Retries, Idempotence and Exactly-Once

A smart-meter gateway publishes a reading every minute. One evening the network drops the broker's acknowledgement for reading number 1, even though the broker stored it. The producer, doing exactly what it should, retries. Afterwards the log holds that reading twice, and a billing job sums it twice. Nobody wrote a bug; the combination of retries and an unreliable network produced the duplicate. This lesson is about reasoning through that failure rather than hoping it does not happen.

Three guarantees and what they cost

Delivery guarantees are properties of the whole pipeline: producer to broker, broker to consumer, and the consumer's own side effects. Each stage can fail between "did the work" and "told someone", and where you place that gap decides the guarantee.

At-least-once is the practical default for business data because losing a payment is worse than seeing it twice. But "twice" has to be survivable, which is where idempotence comes in.

Watching a retry create a duplicate

The simulation below has a broker that either blindly appends or remembers the highest sequence number it has seen per producer. The producer loses the acknowledgement for reading 1 on its first attempt and retries. This is the same idea the idempotent producer uses: each record batch carries a producer id and a sequence number, so the broker can recognise a resend.

class Broker:
    """One partition. Ack for a chosen send is 'lost' to mimic a network blip."""
    def __init__(self, dedupe):
        self.log = []
        self.dedupe = dedupe
        self.last_seq = {}          # producer_id -> highest sequence stored

    def append(self, producer_id, seq, value):
        if self.dedupe and seq <= self.last_seq.get(producer_id, -1):
            return "dup-ignored"    # already stored, still acknowledge
        self.log.append(value)
        self.last_seq[producer_id] = seq
        return "stored"

def run(dedupe):
    broker = Broker(dedupe)
    lose_ack_for = {1}              # the ack for reading #1 never reaches the producer
    readings = ["kwh=10", "kwh=11", "kwh=12"]
    for seq, value in enumerate(readings):
        attempts = 0
        while True:
            attempts += 1
            broker.append("meter-gw-1", seq, value)
            ack_lost = seq in lose_ack_for and attempts == 1
            if not ack_lost:
                break               # producer saw the ack
    return broker.log

print("retries, no idempotence:", run(dedupe=False))
print("retries, with sequence numbers:", run(dedupe=True))

retries, no idempotence: ['kwh=10', 'kwh=11', 'kwh=11', 'kwh=12']
retries, with sequence numbers: ['kwh=10', 'kwh=11', 'kwh=12']

Real Kafka does this per partition and per producer session, and it also protects ordering when several requests are in flight and one fails. In client versions from 3.0 onward, enable.idempotence is on by default with acks=all, but you should still set it explicitly in code you intend to keep, because older clients or other-language libraries may default differently. Check your client's documentation.

enable.idempotence=true
acks=all
retries=2147483647
delivery.timeout.ms=120000
max.in.flight.requests.per.connection=5

Here delivery.timeout.ms is the real retry budget: the producer keeps retrying until that total time runs out and then reports failure to your callback. Do not try to tune retries by guessing a small retries count; set the timeout you are willing to wait.

Exactly-once: what is real

Kafka supports transactions. A producer with a transactional.id can write to several partitions and also commit consumer offsets as one atomic unit, and consumers configured with isolation.level=read_committed skip records from aborted transactions. This gives exactly-once for the consume-transform-produce pattern entirely inside Kafka: read from topic A, write to topic B, advance the offset on A, all or nothing. It does not reach out into the world. If your consumer sends an email or charges a card, a crash after the side effect and before the commit still repeats it. For those, exactly-once is something you build from at-least-once plus idempotent handling.

Making the consumer idempotent

The simplest durable approach: give each record an identity and make applying it twice harmless. A record's (topic, partition, offset) is a natural identity for anything consumed from Kafka, as long as it is stored in the same transaction as the effect. Below, a SQLite table uses that triple as its primary key and INSERT OR IGNORE to skip repeats.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""
    CREATE TABLE meter_readings (
        topic TEXT, part INTEGER, off INTEGER,
        meter TEXT, kwh REAL,
        PRIMARY KEY (topic, part, off)
    )
""")

def handle(topic, part, off, meter, kwh):
    # The record's coordinates are its identity: a redelivery hits the same key.
    cur = db.execute(
        "INSERT OR IGNORE INTO meter_readings VALUES (?, ?, ?, ?, ?)",
        (topic, part, off, meter, kwh),
    )
    db.commit()
    return "applied" if cur.rowcount == 1 else "duplicate, skipped"

delivered = [
    ("meter-readings", 2, 40, "M-17", 10.0),
    ("meter-readings", 2, 41, "M-17", 11.0),
    ("meter-readings", 2, 41, "M-17", 11.0),   # redelivered after a crash before commit
    ("meter-readings", 2, 42, "M-17", 12.0),
]
for rec in delivered:
    print(rec[1:3], handle(*rec))

total = db.execute("SELECT SUM(kwh), COUNT(*) FROM meter_readings").fetchone()
print("rows:", total[1], "sum kWh:", total[0])

(2, 40) applied
(2, 41) applied
(2, 41) duplicate, skipped
(2, 42) applied
rows: 3 sum kWh: 33.0

Where the producer can attach a business id (an order number, a reading timestamp plus meter id), prefer that over offsets: it also survives a re-publish of the same event into a new topic. Either way the principle is the same, and it is the same principle behind idempotency keys on payment APIs.

Common mistakes

• Believing `acks=all` alone prevents duplicates. It prevents loss. Retries after a lost ack still duplicate unless idempotence is enabled. Symptom: totals slightly too high after network incidents.
• Auto-commit combined with slow processing. The timer commits an offset for records still being handled; a crash then loses them. Symptom: records "skipped" after a restart, with no error anywhere.
• Using `acks=all` with `min.insync.replicas=1` and calling it durable. If only the leader is in sync, all means one copy. Set min.insync.replicas=2 for a replication factor of 3 and expect NotEnoughReplicas errors when two brokers are down; that is the point.
• Claiming exactly-once for side effects. Transactions cover Kafka-to-Kafka work. A webhook call inside the loop is still at-least-once.
• Retrying forever on a poison record. One malformed message blocks its partition. Route repeated failures to a dead-letter topic with the original coordinates attached.

How this shows up in real projects and interviews

"How do you get exactly-once delivery?" is a trap question: the honest answer is that delivery over a network is at-least-once, and exactly-once is an effect you achieve with idempotent producers, transactions inside Kafka, and idempotent consumers outside it. In production, the best-run teams assume every handler will be called twice someday and test for it by replaying a day of traffic into a staging copy. For a broader view of designing around retries, see System Design.