Design a Notification System: Push, Email and SMS Without Duplicates

The platform's first job is a single entry point: any internal team can request a push, an email or an SMS through the same call, without knowing which vendor sits behind it. Second, content comes from templates that are filled with per-user values and localised, so the checkout team never hard-codes copy in Hindi or Spanish inside its own service.

Third, users stay in control. They choose which channels they want, opt out of promotions, and define quiet hours. Fourth, the system supports both deferred delivery and large audience campaigns. We explicitly do not build carrier infrastructure; push goes through APNs and FCM, email through a service such as SES, and SMS through a third-party vendor. That boundary matters because it means our reliability depends on systems we cannot fix.

Reliability comes first: once the API has accepted a request, the message must either be delivered or end in a visible failure state, never vanish. That pushes us toward at-least-once processing, which in turn creates the duplicate problem solved later. Latency is selective. A login code has to land in seconds even while a huge promotion is in flight, whereas a newsletter can tolerate minutes.

Users must not be spammed, so the same message should never show up twice and promotional volume is capped per person. Finally, load arrives in spikes rather than a smooth curve. A marketing team pressing send on a big campaign can generate millions of messages in a few minutes, so every stage must scale horizontally and absorb bursts with buffering instead of falling over.

Assume about 1 billion notifications per day, split roughly 70% push, 25% email and 5% SMS. SMS is the smallest share but by far the costliest per message, which is why later sections care so much about not sending it twice. Spread evenly, a billion a day is around 12,000 sends per second.

Averages hide the real problem. Reaching 50 million people within ten minutes lifts throughput to around 80,000 per second, several times the mean. If each message leaves about 500 bytes of log data, the log grows by around 500 GB per day, or roughly 15 TB for thirty days of retention. These figures argue for durable queues in front of every stage, separated by channel, so bursts are smoothed and one provider's trouble stays contained.

Four entities carry the model: a Notification (the request), a Template, a User preference record, and a Delivery, which represents one attempt on one channel. Splitting Delivery from Notification is what lets us report that push succeeded while SMS is still retrying.

Callers use POST /notifications with a user ID, template ID, data for the placeholders and an idem_key. The server answers 202 Accepted with a notification ID immediately instead of waiting for providers, which keeps upstream services fast. GET /notifications/{id} returns status per channel for debugging and support. Users manage their settings through PUT /users/{id}/preferences, which stores channel choices, opt-outs and quiet hours.

A request enters through the notification API, which validates input and rejects repeats using the caller's idempotency key. It writes a record to the notification log and publishes an event to Kafka, then returns. Everything after that point is asynchronous, so a slow vendor can never block an order or payment flow.

A router consumes the event, loads the user's preferences and the template (both cached in Redis in front of a database), picks the eligible channels and renders the final text. It then places one message on each relevant channel queue. Push, email and SMS have independent queues and worker pools, so an email outage does not hold up push. Workers call the external provider and report the outcome back to the log, giving callers and support staff a complete delivery trail.

Routing is where policy lives. Time-sensitive traffic such as OTPs and security warnings rides a dedicated high-priority lane with its own workers, while promotions sit in a lower lane that can lag without harm. Before any rendering happens, the router checks preferences from Redis, because there is no point formatting a message the user has opted out of, and in many regions sending marketing without consent is a legal issue.

Quiet hours and frequency caps come next. A non-urgent push generated at 1 a.m. in the user's timezone is parked until morning, and promotions are limited to a small number per day. Templates are versioned, so a copy change ships as a new version without breaking services pinned to the old one. Campaigns never hit the router as one giant request; a separate fan-out job slices the audience into modest batches and feeds them in steadily.

Every vendor publishes limits, so each one gets a token bucket stored in Redis and shared across all workers. A worker sends only when it can take a token, which keeps the fleet under the cap regardless of how many workers are running. When a call fails, the error is classified. Timeouts, HTTP 429 and 5xx responses are transient and worth retrying; an invalid phone number or bounced address is permanent and should stop immediately.

Retries use exponential backoff with jitter and are scheduled through a delay queue, so no worker thread sits idle waiting. If a provider keeps failing, a circuit breaker trips and traffic shifts to a backup vendor, which is especially valuable for SMS codes. Messages that exhaust their attempts land in a dead-letter queue for inspection, and when a push service reports a device token as unregistered, that token is removed so we stop wasting calls on it.

Skipping deduplication is simplest, but with at-least-once processing it guarantees that some users eventually receive a second code. Checking the idempotency key only at the API blocks duplicate requests from callers, yet it does nothing for the worker that sends successfully, crashes before acknowledging, and is retried by the queue.

The chosen approach checks at both points. Right before calling a provider, a worker attempts SET NX in Redis on a key made of the notification ID and channel, with a 24-hour TTL. If the key already exists, the send is skipped. The remaining risk is a worker dying in the instant between the provider call and the key write, so push messages also carry a collapse key; if a duplicate slips through, the device simply replaces the older copy with the newer one.

A queue per channel isolates failures at the cost of more infrastructure to operate, and that cost is worth paying because cross-channel stalls are the most visible outage. At-least-once processing with deduplication means nothing silently disappears, but every hop needs an idempotency key and the discipline to check it.

Quiet hours make users happier and reduce opt-outs, while intentionally delaying non-urgent messages. Integrating several vendors per channel gives resilience and pricing leverage but multiplies the integration and monitoring work. The guiding rule: isolate channels, honour what the user asked for, and design retries so that repeating an operation can never deliver the same message twice.