Design a Metrics Monitoring System: Millions of Points Per Second

Ingestion comes first: services emit data points made of a metric name, a set of tags like host or service, a numeric value and a timestamp. Users then need to query those series over time, filter and group by tags, and compute functions such as averages, sums or the 99th percentile for dashboards.

Alert rules turn queries into automation, for example firing when p99 latency stays above 500 ms for five minutes. When a rule fires, notifications go to the on-call engineer through a paging service, a chat channel or email. Logs and distributed tracing are separate observability pillars with very different data shapes, so they stay out of scope.

Write throughput dominates: millions of points per second, continuously, with reads being a tiny fraction of traffic. Despite that, dashboards for the last few hours should still render in about a second, which means recent data must sit in fast storage.

Alerts must fire within roughly a minute and, critically, must keep firing during the very outages they exist to detect. That pushes the alerting path to be independent from dashboards and long-term storage. Retention should cover a year or more at low cost, which implies aggressive compression and summarising old data rather than keeping every raw point forever.

Assume 100,000 hosts, each exposing about 500 metrics, which gives 50 million active time series. If every series reports once every 10 seconds, the system receives about 5 million points per second. Over a day's 86,400 seconds that adds up to roughly 432 billion points.

Stored naively at 16 bytes per point, an 8-byte timestamp plus an 8-byte value, that is about 7 TB per day. Specialised time-series compression gets close to 1.4 bytes per point, shrinking the daily footprint to around 600 GB. That twelvefold difference is the main reason to choose a storage engine designed specifically for time series.

A series is identified by its metric name plus tag set, a data point is a timestamp and value attached to a series, and an alert rule combines a query with a condition and a notification target.

Agents send batches via POST /metrics and receive 202 Accepted immediately; processing happens asynchronously, so agents never block on storage. Dashboards call GET /query, passing an expression, start and end times, and a step that controls point spacing, and receive points per matching series. POST /alerts creates a rule with a query, a comparison operator, a threshold, a for duration and a notification target.

Agents on each host gather metrics locally and push compressed batches every 10 seconds to an ingest gateway, which authenticates the sender and validates the payload. The gateway writes into Kafka, partitioned by series, so traffic bursts are buffered and downstream failures can be recovered by replaying the log.

Ingester processes consume Kafka and append points into a time-series database that holds the most recent hours in memory and on SSD. Older data is compacted into blocks, downsampled and shipped to object storage like S3. A query service serves dashboards by reading hot data from the TSDB and cold data from object storage, then stitching the two together.

Alerting runs as its own component. An evaluator executes each rule against the hot store about every 30 seconds and hands firing alerts to a notifier that pages the right people. Because ingestion, queries and alerting are separate, a heavy dashboard query cannot delay a page, and each path scales independently.

Five million points per second would be impossible if each point were its own HTTP call, so agents accumulate a batch of hundreds of points, compress it and send one request every 10 seconds. Kafka partitions are keyed by a hash of the metric name and tags, which pins each series to a single ingester. Keeping a series on one machine lets that machine compress consecutive points effectively.

Cardinality is the classic way to break such a system. Adding a tag like user_id can turn one metric into millions of distinct series overnight, exhausting memory. Enforcing a per-tenant limit on active series, and rejecting obviously unbounded tags, contains the damage.

Ingesters keep recent points in memory, protect them with a write-ahead log so a crash loses nothing, and flush compressed two-hour blocks to disk. If they fall behind, Kafka simply retains the backlog; rising consumer lag triggers an alert instead of points being silently dropped.

A relational database with one row per point is familiar, but at 5 million inserts per second the tables and indexes balloon and write amplification becomes crippling. A wide-column store such as Cassandra handles the write rate, yet offers little time-series-specific compression and is weak at the aggregations dashboards depend on.

A purpose-built TSDB is the chosen option. Since points arrive at near-regular intervals, timestamps are encoded as the difference between successive gaps, often zero, and each value is XORed with the previous one, which yields mostly zero bits for slowly changing metrics. Together these tricks cut storage by roughly twelve times while still allowing fast range scans.

Downsampling handles long retention. Raw points live for 15 days, one-minute rollups for 90 days, and hourly rollups for two years in object storage. Each rollup bucket stores min, max, sum and count, so averages remain exact and a brief spike is still visible in the max even after the raw data is gone.

Consider a rule such as p99 latency greater than 500 ms for five minutes. With thousands of rules across many teams, evaluation is sharded: each evaluator instance owns the subset of rules whose ID hashes to it, so adding instances spreads the load evenly.

Every 30 seconds, an evaluator runs each of its rules against the recent window in the hot store only, so slow reads from object storage can never hold up a page. When the condition first becomes true, the alert enters a pending state and fires only if it remains true for the full duration. That filter removes most noise from momentary blips.

Firing alerts flow to a notifier that groups related alerts, collapses duplicates and honours silences configured during maintenance, so the on-call engineer gets one meaningful page rather than hundreds. The final safeguard is an external heartbeat: a check outside the system expects regular signals from alerting and pages someone if they stop.

Push-based collection works through firewalls and suits short-lived jobs that may exit before being scraped, while pull-based collection makes it obvious when a target has died because the scrape fails. Downsampling makes years of history affordable, but fine-grained detail disappears once raw data expires.

Placing Kafka between agents and storage adds durability and replay at the cost of a few seconds of extra latency. Alerting by running queries is simple and expressive, but the worst-case alert delay equals ingestion delay plus the evaluation interval. The overall philosophy is to squeeze storage hard, summarise aging data, and give alerting its own independent path.