Design a Web Crawler: Polite, Deduplicated Crawling at Scale

The feature list is short on purpose. An operator supplies seed links, the system downloads the HTML behind each one, extracts outgoing links so the crawl keeps expanding, and persists both the raw markup and the cleaned text. Each of these maps to a distinct stage later: seeding feeds the frontier, fetching is the network-heavy stage, link extraction is CPU work, and storage is the hand-off point to consumers.

Drawing the boundary at storage matters. Indexing, ranking and query serving are entire systems on their own, and pulling them in would leave no time for the hard parts. Saying explicitly that the crawler's job ends once a page is safely written keeps the conversation focused on discovery, fetching and reliability.

Politeness is listed first because breaking it ends the project: a crawler that floods a small blog gets its IP ranges blocked and earns complaints. Honouring robots.txt and keeping to roughly one request per second per host shapes the entire frontier design. Scale is the opposite force, since ten billion pages in about five days requires massive parallelism across many different sites.

Fault tolerance and efficiency round out the list. With hundreds of machines running for days, crashes are routine rather than exceptional, so a dead worker must never take queued URLs with it. Efficiency means not paying twice, in bandwidth or in storage, for content we already hold. Both requirements push us toward durable queues and aggressive deduplication.

Assume ten billion pages over five days. Five days is roughly 432,000 seconds, so the system needs to sustain about 23,000 fetches every second, around the clock. If a typical HTML document weighs around 100 KB, the raw archive lands near one petabyte, which points straight at cheap object storage rather than any database.

Bandwidth follows from the same numbers: 23,000 pages per second at 100 KB each is close to 2.3 GB per second, or roughly 18 Gbps of inbound traffic. Because fetching is mostly waiting on remote servers, an async worker can keep hundreds of connections open; assume each handles about 300 pages per second. That gives around 80 fetch workers, and we provision extra headroom for slow sites, retries and machine failures.

Three entities carry the state. A URL record tracks status and when it was last crawled. A Domain record stores the parsed robots rules plus the earliest moment we may contact that host again, which is the hook for politeness. A Page record points at the stored content along with its fetch time and content hash, the hash being what powers duplicate detection later.

The external surface is deliberately thin. POST /crawls starts a job from a list of seeds and an optional page cap, GET /crawls/{crawl_id} reports counts of fetched, queued and failed URLs, and GET /pages?url= lets downstream teams locate the stored copy of a page. Everything else is an internal pipeline driven by queues, so spending interview time on REST details would be wasted.

The architecture is a cycle. Seeds enter the URL frontier, which holds pending work grouped by domain. A fetcher pulls the next eligible URL, resolves the host and checks robots rules through a shared Redis cache, downloads the page, and writes the raw HTML to object storage. It then emits a lightweight event so the parsing stage knows new content is waiting.

Parsers read the stored page, extract text and outgoing links, and hand those links to the dedup service. Dedup consults the URL metadata store, records anything new, and pushes only never-seen URLs back into the frontier, closing the loop. Splitting fetching from parsing matters because one is network-bound and the other CPU-bound, so each tier can scale on its own curve.

Crucially, no worker holds state that cannot be rebuilt. The frontier and the stores own everything durable, so adding fetchers during a big crawl or replacing a crashed parser is just a matter of starting another process. That property is what makes the fault tolerance deep dive straightforward later.

Not every page deserves the same urgency. A breaking-news homepage changes by the minute, while an archived manual may not change for years, so the frontier first assigns a priority based on signals like link popularity and observed change rate. High-value URLs get fetched sooner and revisited more often.

Politeness comes from grouping. If the frontier is partitioned by hostname, for instance as a Kafka topic keyed by domain, every URL for a given site lands on the same partition and therefore with the same worker. Two machines can never race against one host. Before each request that worker checks its cached copy of robots.txt and drops any disallowed path.

Inside a worker, a min-heap orders the domains it owns by their next permitted fetch time. The worker pops whichever domain is ready, fetches one URL, and pushes the domain back with a new time of now plus its crawl delay. When a server answers with 429 or 503, the delay for that host doubles, so a struggling site gets breathing room automatically instead of being retried at full speed.

Duplicates arrive at two levels, and the link rate is brutal: each page carries dozens of links, so the dedup tier must answer well over a million lookups per second. The first defence is canonicalisation. Lowercasing the hostname, stripping #fragment anchors and removing tracking parameters like utm_source collapses many spellings of one page into a single key before any lookup happens.

For the membership test itself, a Bloom filter is the natural fit. Sized for ten billion entries at a 1 percent false positive rate, it needs only about 12 GB of memory, small enough to keep in RAM. The failure mode is gentle: an occasional false positive means one genuinely new page is skipped, which is an acceptable loss for a crawl of this size.

URL checks cannot catch mirrors, syndicated articles or session-tagged copies that live at different addresses. Hashing the response body catches exact matches, so identical content is stored once. Near-duplicates, such as the same story wrapped in different ads or timestamps, need SimHash, whose fingerprints differ by only a few bits for similar documents. Finally, guard against traps like infinite calendar links by capping URL length, path depth and pages per domain.

Where the frontier lives decides what a crash costs. Keeping queues in process memory is the fastest option, but a restart wipes out every pending URL and RAM limits how big the backlog can grow. A relational table survives crashes and is easy to inspect, yet at a million new URLs per second the polling queries and row-level locking turn into the bottleneck.

A Kafka topic partitioned by domain is the chosen design. The log is replicated and replayable, and the partition-to-consumer mapping gives one owner per host, so politeness comes along for free. Workers commit their offset only after the page is safely in storage. If a worker dies mid-batch, its partitions are reassigned and the new owner simply refetches a few pages; nothing disappears.

Transient failures go to a chain of retry topics with growing waits, for example one minute, then ten, then an hour, before landing in a dead-letter topic for inspection. Every HTTP request also carries a strict timeout. Without it, a single server that accepts connections and then stalls could tie up a worker indefinitely and quietly drain throughput.

Each major choice gives something up. The Bloom filter trades a small, random loss of coverage for a memory footprint that fits on one machine instead of a cluster. Strict per-host limits mean very large sites take days to finish, but the alternative is getting blocked and finishing never. Using Kafka as the frontier buys durability and horizontal scale, at the cost of building priorities and delays from extra topics rather than getting them natively.

Accepting at-least-once fetching ties the design together. Exact-once delivery across fetch, store and offset commit would require heavy coordination, whereas occasionally refetching a page is cheap because the content hash catches the repeat before it is stored again. The guiding idea is to stay courteous to every host, filter repetition at every layer, and make sure any step can safely run twice.