Design YouTube Top K: Most Viewed Videos Over Sliding Windows

There are three requirements. Every view must contribute to the rankings; we cannot sample or drop events. A client can request the top K videos, where K is capped at 1,000, and receive each video's view count alongside its ID. Finally, rankings exist for four windows: the most recent hour, the most recent day, the most recent thirty days, and all time.

Breaking rankings out by country or category is a natural follow-up, but it does not change the architecture. Each extra dimension simply becomes part of the key, so a list for India in the music category is just another window keyed differently. Naming this early shows the interviewer you have thought about extension without letting it widen the core scope.

Freshness is bounded at about one minute: a video going viral should appear in the hourly list quickly, but nobody needs second-by-second precision. Reads must be fast, with a p99 under 100 ms, because the list sits on high-traffic pages. That requirement alone rules out any counting at query time.

We target an exact top list. A ranking where a video sits at number three when it truly belongs at number nine is visible and embarrassing, though asking whether approximate is acceptable is a sharp clarifying question. Lastly, ingest is enormous, billions of events per day, and dropping them is not allowed, so the pipeline must scale horizontally and survive crashes without losing or double-counting views.

Assume 10 billion views per day. Dividing by the seconds in a day gives roughly 116,000 events per second on average, and peaks around 350,000 per second are plausible. Assume about 4 billion videos. A video ID plus a counter is roughly 16 bytes, so holding one counter per video needs about 64 GB, which fits across a modest cluster but not comfortably on a single machine.

Contrast that with the output. Each window's answer is at most 1,000 entries, a few kilobytes. When writes are this heavy and answers this small, the right move is to do the counting continuously as events stream in, and to compute the ranked lists ahead of time so reads become a lookup of a tiny precomputed result.

The domain needs only three entities: a Video, a View event and a Time window such as 1h or 30d. Treating the window as a first-class concept makes it easy to add new ones later without changing the read path.

Readers call GET /top?window=1h&k=100 and receive an ordered list of video IDs with view counts. Views arrive through an internal POST /views carrying video ID, user ID and timestamp, and it returns 202 because the event is accepted into a stream rather than counted synchronously. In a real deployment the player emits these events directly to the ingestion tier, so this endpoint is more of a contract than a public API.

Players emit a view event per playback into Kafka, using the video ID as the partition key. That choice is load-bearing: every view of a given video lands in the same partition, so exactly one downstream task owns its count. A Flink job consumes the stream and accumulates counts per video per minute in memory, flushing compact batches rather than issuing a write for every view.

Those minute-level counts are persisted to a counts store, which acts as the source of truth for rebuilding any window after a failure. In parallel, a top-K builder keeps one leaderboard per window and periodically (every few seconds) publishes the current lists into a Redis results cache. The read API is deliberately dumb: it fetches a precomputed list from Redis and returns it. Nothing is counted or sorted on the request path.

A database row per video, incremented on each view, is exact and easy to explain. But it means over a hundred thousand row updates per second, and a single viral video concentrates a large share of them on one hot row, creating lock contention exactly when traffic peaks.

A Count-Min Sketch uses a small grid of counters indexed by several hash functions. Each event increments one cell per hash row, and a video's estimated count is the smallest value among its cells. Memory is fixed and tiny, but collisions can inflate counts (never deflate them), and since the sketch stores no IDs it must be paired with a heap that tracks current leaders.

The chosen design keeps exact counters inside the stream processor, sharded by video ID. Since each counter lives in exactly one partition, capacity grows by adding partitions. If memory ever becomes the constraint, a hybrid works well: run a sketch as a cheap first-pass filter and keep exact counts only for videos whose estimate is high enough to possibly reach the top list.

Partitioning by video ID routes all of one video's views to a single counting task. That task sums events in memory and, at each minute boundary, emits a tuple of video, minute and count, shrinking a firehose of raw events into far fewer records. Each partition also maintains a min-heap of its local top K.

Merging is where partitioning pays off. Because no video's count is split across partitions, the global top K must be contained within the union of the local top-K lists. Combining N small heaps therefore gives an exact global answer without scanning every counter. Viral videos are handled by pre-combining at the ingest tier, which sends one summed record per video per second instead of millions of individual events. For crash safety, Flink snapshots its state together with the Kafka offsets it has consumed, so a restart resumes from a consistent point and no view is applied twice.

Every window is assembled from per-minute buckets. For the hourly ranking, each new minute is added and the minute that just dropped off the back is subtracted. Only videos that appear in those two buckets need updating, which keeps the work proportional to activity rather than to the total catalogue.

Long windows would be wasteful at minute granularity: thirty days is over 43,000 minutes. Instead, minutes roll up into hourly buckets and hours into daily ones, so a thirty-day window is maintained from about thirty day buckets. All time is simply a running total that only ever increases, and a nightly batch job checks it against raw events to correct any drift. Finally, the builder publishes the top 1,000 for each window into Redis every few seconds, which is why reads stay well inside the latency target.

Exact counting costs memory, while a sketch saves memory at the price of occasional over-counts; the hybrid filter is the escape hatch if the budget tightens. Publishing more often improves freshness but burns compute, and a few seconds is a comfortable middle ground given the one-minute freshness target.

Coarse buckets make long windows cheap, but those windows advance in bigger jumps, so the thirty-day list might shift once a day rather than smoothly. Counting every view is simple, whereas filtering bot traffic adds latency and complexity; for a public ranking, though, trust is worth the delay. The pattern that ties it together: key everything by video, aggregate before storing, and have every window's answer ready before anyone asks.