Apache Kafka Tutorial
Understand Kafka's log model, consumer groups, delivery guarantees and when to reach for it
Intermediate4 lessonsabout 23 min of reading
This course is for backend developers who have heard "just put it on Kafka" in a design review and want to know what that actually commits them to. You should be comfortable reading Python or Java and know what a message queue is for. You do not need a running cluster: Kafka is not installed for any of the lessons, so the commands are explained carefully and the core ideas are demonstrated with small Python programs you can run anywhere.
By the end you will be able to explain Kafka as a partitioned, replicated, append-only log; choose keys and partition counts on purpose; reason about consumer groups, offsets and lag; pick between at-most-once, at-least-once and exactly-once handling; and decide when a plain queue, a database table or a simple HTTP call is the better answer.
The lessons build on each other. First the storage model (topics, partitions, offsets), then the two sides that talk to it (producers, consumers and groups), then what happens when things fail (retries, duplicates, idempotence), and finally the long-term questions: retention, ordering, and design trade-offs. For the wider context of where messaging fits, see System Design and Microservices Architecture.
Apache Kafka lessons
- 1Apache Kafka Topics, Partitions and Offsets: How the Log WorksLearn how Apache Kafka stores data: topics, partitions, offsets and keys, with a runnable Python model of the log and the CLI commands to inspect it.6 min read
- 2Kafka Producers, Consumers and Consumer Groups ExplainedSee how Kafka producers batch and send records, how consumers commit offsets, and how consumer groups split partitions, rebalance and show lag.6 min read
- 3Kafka Delivery Semantics: Retries, Idempotence and Exactly-OnceUnderstand Kafka at-most-once, at-least-once and exactly-once delivery, how retries create duplicates, and how idempotent producers and consumers fix it.5 min read
- 4Kafka Retention, Ordering and When Not to Use KafkaLearn Kafka retention and log compaction, how to preserve ordering with keys, and a checklist for deciding when Kafka is the wrong tool for the job.6 min read
Preparing for an interview? Practise System Design interview questions
Learning Apache Kafka to land a job? Browse open roles or check which skills are trending.