Design a Job Scheduler: Running Billions of Timed Tasks Reliably

Users must be able to register a job that fires once at a specific timestamp or repeatedly according to a cron expression. Supporting both covers the common patterns, from a single delayed notification to a recurring billing run. When the time comes, the platform invokes the named task with the stored payload, so the scheduler stays generic and the business logic lives in task handlers.

Jobs also need management: cancelling, pausing or editing a job before its next run prevents stale work from firing. Finally, every run should be visible with a status such as queued, running, succeeded or failed, because operators debugging a missed invoice need a history to inspect. Dependency graphs between jobs, the kind of DAG workflows Airflow handles, are a separate orchestration problem and stay out of scope.

Timeliness is the first quality target: a job should begin within about two seconds of its due moment. Durability is just as important. Once the API acknowledges a job, a crash anywhere in the system must not make it disappear, so we commit to at-least-once execution.

Scalability matters because thousands of jobs can become due in the same second, with sharp spikes at the top of each hour. That pushes toward horizontally scalable storage and workers rather than one powerful machine. High availability rounds things out. A single scheduler process whose failure freezes every customer's jobs is the exact weakness we are designing away, so scheduling itself has to be spread across several nodes that can take over from each other.

Assume roughly 1 billion job runs per day, a blend of one-off and recurring jobs. That averages around 12,000 runs per second, but averages mislead here: recurring jobs cluster at round times, so assume peaks above 100,000 per second at the top of the hour or at midnight.

Execution history adds up as well. At about 200 bytes per run record, that is roughly 200 GB a day, or around 6 TB if we keep 30 days. The combination of a huge table and bursty access rules out the naive approach of querying all jobs for anything past due. We need an access path that touches only the slice of data relevant to the next few moments.

Three entities shape the model. A job defines what to execute: the task and its payload. A schedule defines when, either a single run_at or a cron expression. An execution represents one concrete run with its own status and attempt count. Splitting execution from job is what lets a recurring job keep a clean history and lets retries appear as separate records.

Creating work uses POST /jobs with the task, payload, either run_at or cron, and an optional retry policy; the response is a job ID. GET /jobs/{id}/executions lists runs with status, scheduled time and attempt number, which is the audit trail operators need. DELETE /jobs/{id} cancels a job and returns 204. Pause and update can follow the same pattern as additional endpoints.

Callers, usually other backend services, talk to a job API that validates requests and writes the job plus its next execution into a jobs database. That execution is indexed by due time, which is the property everything else depends on.

A scheduler service continuously looks ahead at executions due in the next short window, never the entire table. When an execution's moment arrives, the scheduler places it on a queue such as Kafka or SQS. The queue acts as a shock absorber: at midnight, the scheduler can enqueue a flood of work quickly while the worker pool drains it at its own pace.

Workers pull messages, call the right handler for each task and report the outcome back to the database. For recurring jobs, the worker also inserts the following execution. Several scheduler instances run in parallel, and a coordinator such as etcd assigns each one a distinct set of shards. Ownership is exclusive, so no due job is enqueued twice by competing schedulers, and if one instance dies its shards are reassigned.

Polling the jobs table every second for overdue rows is the obvious first answer, and it collapses once the table reaches billions of rows because each poll becomes an expensive scan. A Redis sorted set scored by run time gives fast range reads, yet the whole schedule must fit in memory and durability is weaker than a real database, which conflicts with the promise never to lose a job.

The preferred design is a time-bucketed execution table. The partition key is the minute a job is due, combined with a shard number so a crowded minute is spread across several partitions; rows within a partition sort by run time and execution ID. Fetching what is due means reading one small partition. Wide-column stores like Cassandra or DynamoDB suit this layout naturally.

Each scheduler loads its shards of the upcoming minute slightly ahead of time, holds those executions in memory and enqueues each at its precise second. A job created only 30 seconds before it is due could slip past that preload, so the API sends anything due within the next minute directly to the queue with a delivery delay.

Workers fail, so delivery must survive them. When a worker receives a message, the queue does not delete it; it hides it for a visibility timeout of perhaps 30 seconds. The worker then claims the execution by switching its status from QUEUED to RUNNING with a compare-and-set, so if two workers somehow hold the same message, only one succeeds in claiming it.

Jobs that take longer than the timeout send periodic heartbeats that extend both the lease and the message visibility. If a worker crashes, its heartbeats stop, the lease runs out and the message reappears for someone else to process. On success, the worker records SUCCEEDED first and only afterward acknowledges the message, so a crash between the two leads to a repeat rather than a loss.

That ordering means a task can occasionally execute twice, for example when a slow worker finishes just after its lease expired. Tasks therefore treat the execution ID as an idempotency key: a payment handler, for instance, records the execution ID with the charge and skips it if it has already been applied.

Not every failure deserves another attempt. Timeouts and 503 responses usually reflect temporary trouble and are worth retrying, while a malformed payload will fail forever and should be marked failed at once. Retries use exponential backoff, roughly 1, 2, 4 and 8 seconds, plus random jitter so that a batch of jobs that failed together does not hammer the dependency again in lockstep.

A neat simplification is that a retry is just another execution placed at the current time plus the backoff in the same time-bucketed index. No separate retry subsystem is needed. After a cap, say five attempts, the job moves to a dead-letter queue and its owner is alerted to investigate. Per-tenant concurrency limits complete the picture, preventing one customer's runaway job from overwhelming a shared downstream service with thousands of simultaneous calls.

Exactly-once execution across machines is not practically achievable, so the design pairs at-least-once delivery with idempotent tasks, which is the accepted industry answer. Bucket size trades precision for load: smaller buckets give tighter timing but mean more frequent queries against the database.

Placing a queue between schedulers and workers absorbs spikes and decouples the two tiers, at the price of another system to deploy and monitor. Storage choice is a familiar tension too: a SQL database makes transactional status updates easy, while a wide-column store scales write-heavy execution data with less effort. The overall formula is to index work by time, protect every run with a lease, and design each task to be safe to run again.