Design Facebook Post Search: Keyword Search Over Trillions of Posts

Two write actions feed the system: publishing a post and liking one. On the read side, a user enters one or more keywords and receives only posts that contain every term, which is AND semantics rather than any-match. Results can be ordered newest first or most liked first, and a cursor lets the client keep scrolling.

Privacy rules, per-user ranking and spelling correction are excluded on purpose. Each would add a separate subsystem, such as an access-control filter or a learned ranker, and none changes the fundamental index structure. Keeping them out lets the discussion stay on indexing, partitioning and freshness.

Latency target is under 500 ms per query, and that has to hold for extremely common words, which are exactly the terms with the longest index entries. Freshness is the second pillar: a newly published post should appear in results within about a minute, because people search for things that are happening right now.

The system must absorb billions of writes daily and remain available through machine failures. Since a like total that is a few seconds old harms nobody, we explicitly favour availability over strict consistency. That single decision unlocks batching and caching techniques that would be off the table if counts had to be exact at every instant.

Assume one billion new posts per day, which is about 12,000 writes per second. Likes are ten times as frequent, around ten billion a day or roughly 115,000 per second. Searches come to about 500 million per day, near 6,000 per second. Retained for ten years, posts accumulate to roughly 3.6 trillion documents in the index.

Two conclusions fall out. First, an index covering trillions of documents must be spread across many machines, so partitioning is mandatory. Second, likes outnumber searches by about twenty to one, so the dominant cost is not answering queries but keeping the index current. Every design choice that follows tries to make updates cheap.

User, Post and Like are the only entities needed. A post stores its author, text and creation time; a like links a user to a post. The like count per post is derived and lives both in the source database and, approximately, in the index.

POST /posts accepts content and returns a post_id. POST /posts/{post_id}/like records a reaction. Search uses GET /search with a query string, a sort mode of recency or likes, and an opaque cursor; it returns a page of posts plus next_cursor. Cursor pagination is preferable to page numbers because new posts keep arriving, and offsets would shift results between requests.

Traffic enters through an API gateway that sends reads and writes down separate paths. On the read path, the search service first checks a Redis cache for that exact query. On a miss, it fans the query out to every index shard, gathers their partial results and merges them into one ordered page.

On the write path, the post service persists posts and likes in the primary database, which is the source of truth, and emits an event to Kafka for each change. Indexer workers consume those events, tokenise post text, and apply additions and like-count updates to the in-memory inverted index.

Treating the index as derived data is a deliberate safety valve. If a shard is corrupted or lost, it can be reconstructed by replaying Kafka or by rescanning the posts database, so the index never needs the same durability guarantees as the primary store.

Indexing a post starts with normalisation: lowercase the text, break it into words and strip punctuation, so "Cricket!" and "cricket" map to the same term. For each term the index stores a posting list, the set of post IDs that contain it, much like the index pages at the end of a textbook.

Each posting list is maintained twice, once ordered by creation time and once by like count, so either sort mode can read entries in final order without sorting at query time. For a two-word query, the search reads both lists and intersects them; only IDs present in both survive. For example, if one word maps to posts 9, 7 and 3 and the other to 9, 4 and 3, the answer is 9 and 3.

Because the lists are pre-sorted, the intersection can walk from the top and stop the moment one page of matches is assembled. Even a term that appears in billions of posts costs only a short scan, which is how common-word queries stay inside the latency budget.

Partitioning by term places each word's entire posting list on one shard, so a query touches only a handful of machines. The catch is skew: everyday words produce gigantic lists and attract most of the traffic, creating hot shards that are hard to rebalance.

Partitioning by post ID is the chosen approach. Each shard holds a complete mini-index over its own subset of posts, load spreads evenly, and a new post or like touches exactly one shard. The price is that every query must visit every shard. In this scatter-gather pattern every shard reports its 25 best local candidates, and the coordinator combines those lists into the final 25.

Tiering keeps the hardware bill reasonable. Recent posts, which satisfy most searches, live in memory; older posts sit on SSD and are consulted only when the recent tier cannot fill a page. Because like ordering inside the index may lag slightly, the final candidates are re-ranked using current like totals just before the response is returned.

Kafka between the write services and the indexers smooths bursts. When a major event triggers a flood of posts, the backlog grows in the log rather than overloading index shards, and indexers drain it at a steady pace. Like updates get extra treatment: a post going viral may receive thousands of likes per second, so indexers accumulate per-post deltas for a few seconds and apply one combined update.

New documents become visible through a near-real-time refresh that runs roughly once a second, so freshly published posts are searchable almost immediately. Popular queries are cached in Redis for 30 to 60 seconds, which means a trending topic searched by millions hits the shards only once per expiry. Each shard also runs several replicas, spreading read traffic and keeping search available when individual machines fail.

Choosing post-based sharding gives balanced load and single-shard writes, but every query pays a fan-out to all shards, which raises tail latency and makes replica health important. Batching likes slashes index write volume, yet like-sorted results can trail reality by a few seconds.

Caching hot queries yields large savings during spikes, with the downside that a cached page may be up to a minute old. Finally, a managed engine such as Elasticsearch would be the fastest way to ship, while a purpose-built index offers tighter control over memory layout, tiering and update batching once the corpus reaches trillions of documents.