Design a Payment System: Charge Exactly Once and Keep Books Balanced

Four capabilities cover the merchant's needs. Creating a payment charges a tokenised card for a given amount. A status lookup reports whether the payment is still processing, has succeeded or has failed. Refunds return all or part of the original amount. Webhooks push a notification to the merchant's server whenever a payment moves to a new state, so they do not have to poll.

Fraud scoring, payouts to merchant bank accounts and foreign exchange are deliberately excluded. Each is a large system with its own rules, and they all sit downstream of the core question this design must answer: how to move money once, correctly, and prove it afterward.

Correctness outranks everything. A customer must never be charged twice and a successful payment must never vanish. A few extra seconds of latency are acceptable; a wrong charge is not. Close behind is auditability: every movement of funds needs a permanent record, and the totals must always reconcile.

Security is non-negotiable because card data is among the most regulated information there is, which is why the design follows PCI DSS and keeps raw card numbers out of most services entirely. Availability matters too, targeting around 99.99 percent, because every minute of downtime translates directly into lost sales for every merchant on the platform.

Assume 50 million payments per day. That averages to roughly 600 per second, with peaks near 5,000 during big sale events. Each payment produces about four ledger entries covering the charge and fees, so the ledger grows by around 200 million rows daily.

At roughly 1 KB per payment, storage reaches about 50 GB a day and around 18 TB a year, and financial regulations require keeping it for many years. None of these figures stretch a well-sharded relational database. That is the point of the estimate: throughput is manageable, so engineering effort belongs in correctness, durability and auditability rather than exotic scaling tricks.

The model has four entities: Merchant, Payment, Ledger entry and Refund. A ledger entry records an amount moving into or out of a named account, which is what later lets us prove the books balance.

POST /payments takes an amount, a currency and a card_token, never a raw card number, plus an Idempotency-Key header that makes retries safe. GET /payments/{payment_id} returns current status. POST /payments/{payment_id}/refunds accepts the amount to return. In the opposite direction, the platform calls the merchant's registered webhook URL with events such as payment.succeeded, and keeps retrying until it receives a 2xx response.

Requests from a merchant's backend first pass through the payments API, where credentials and input are verified. Next, the payment service looks up the idempotency key and inserts the payment into Postgres with status CREATED before any money moves. Only after that row is durable does it call the PSP gateway, the component that reaches card networks and issuing banks while managing timeouts and retries.

Every state transition is written in the same database transaction as a row in an outbox table. A relay process reads the outbox and publishes those events to Kafka, which guarantees an event is emitted if and only if the state change committed. Two consumers react: the ledger service records double-entry postings, and the webhook service delivers signed notifications to merchants.

The underlying rule is to write down what you are about to do, then do it. If any process dies midway, the database shows exactly where the payment stopped, and the remaining steps can be resumed or replayed without guessing.

Networks drop responses, so merchants will retry. The defence begins with the merchant attaching a unique idempotency key, such as a UUID tied to the order, to each create request. The service inserts that key into a table with a unique constraint on merchant and key; a successful insert means this is a genuinely new payment.

If the insert collides, the request is a repeat, and the service returns the response it stored for the original call instead of charging again. A key reused with a different body signals a client bug and is rejected. If the original request is still in flight, a concurrent retry receives a 409 Conflict and is asked to try again shortly, which prevents two workers from racing on the same charge.

The same idea applies one hop further. When the payment service calls the processor, it passes its own payment ID as the processor's idempotency key. That way even the platform's internal retries after a timeout cannot produce a second charge at the bank.

Double-entry bookkeeping records every movement as a pair: a debit to one account and a matching credit to another. The platform keeps accounts for the processor receivable, each merchant's balance, its own fee revenue and refunds. Nothing is ever just added to a balance column; balances are derived by summing entries.

Take a 1,000 rupee purchase with a 2 percent platform fee. The ledger debits the processor receivable by 1,000, credits the merchant's account by 980 and credits fee revenue by 20. Debits and credits match, so that transaction nets to zero. A later refund would add new entries that reverse the flows rather than editing these.

The ledger is append-only. Corrections are made with compensating entries, which keeps a complete history for auditors. A continuous check verifies that the sum across all accounts is still zero; if it ever drifts, an alert fires immediately, because an imbalance means money is unaccounted for somewhere.

The scariest moment is a timeout from the bank. Retrying immediately risks a second charge if the processor lacks idempotency protection. Marking the payment failed is worse: the customer may have been charged, leaving money missing from the books and an angry cardholder.

The chosen approach adds an explicit UNKNOWN state. The payment moves through CREATED and PROCESSING to SUCCEEDED or FAILED, but when the outcome is unclear it sits in UNKNOWN while a background job queries the processor by payment ID until it gets a definitive answer. The cost is that some customers wait minutes for confirmation, which is far better than a wrong one.

A daily reconciliation job then compares the internal ledger against the processor's settlement file line by line, and any mismatch lands in a queue for human review. Merchant notifications carry a signature, may arrive more than once, and keep retrying with exponential backoff over several days, so receivers deduplicate on the event ID.

Feeding the ledger asynchronously through the outbox keeps the payment path fast and the ledger replayable, but ledger balances trail live payments by a few seconds; reconciliation catches any gaps. The UNKNOWN state means some outcomes are delayed, a deliberate choice of correctness over speed.

A relational database gives transactions, unique constraints and foreign keys, all of which payments rely on, and write volume is handled by sharding on merchant ID. At-least-once webhooks guarantee merchants hear about every event, with the expectation that they deduplicate on their side. The thread running through all of it is to make each operation repeatable, persist before acting and verify against an external source daily.