Design ChatGPT: Streaming LLM Chat on Scarce, Costly GPUs

Four behaviours define the product. A user submits a prompt and receives an answer. That answer is streamed, so text shows up while it is being produced rather than after a long silence. Conversations persist: old chats appear in a sidebar, and each follow-up question is answered with the earlier turns in mind, which is what makes "make it shorter" a valid request. Finally, the user stays in control and can halt a reply that is going the wrong way or ask for a fresh attempt.

Model training, file attachments, voice mode and tool calling are deliberately left out. Each of them is a large design on its own, and including them would hide the core problem: serving inference to millions of people with a limited fleet of GPUs.

The most visible quality target is time to first token, which we set below one second. People will happily read a long answer, but a blank response area feels broken after a moment or two. Streaming exists mainly to hit this number.

Behind the scenes, GPU utilisation is the cost lever. An accelerator that sits idle between requests is burning money, so the serving layer must keep each one packed with work. Fairness matters for the same reason: if one account submits enormous prompts in a loop, quotas and priorities must stop it from pushing everyone else into a slow queue. Durability is the last requirement. Losing a conversation that a user saved is a trust problem, so writes to history must survive crashes and disconnects.

Assume roughly 200 million daily users who each send about 5 messages. That gives 1 billion messages per day, or about 12 thousand per second on average and something near 35 thousand at the evening peak. If a typical answer runs to around 500 tokens, the fleet must produce about 5.8 million tokens every second, which translates into thousands of GPUs running continuously.

Storage barely registers next to that. At around 2 KB for a message plus its reply, history grows by about 2 TB a day, a volume any partitioned database handles comfortably. The arithmetic points to one conclusion: the expensive, constrained piece is inference capacity, so the architecture must queue requests, batch them on the hardware, and limit how much any one user can consume.

The data model has four entities: a User owns many Conversations, a Conversation holds an ordered list of Messages, and each Conversation is tied to a Model. Keeping the model on the conversation lets different chats use different model sizes without changing the message format.

Three endpoints cover the core flow. POST /conversations creates an empty chat for a chosen model and returns its ID. POST /conversations/{id}/messages submits a prompt, and instead of a single JSON body it returns a server-sent event stream that delivers tokens as they are produced. GET /conversations/{id} loads the title and full message list when someone reopens an old chat. The unusual part is the streaming response; everything else is ordinary request and reply.

A prompt enters through the API gateway, which authenticates the caller and routes the call. Before any real work starts, the gateway asks a Redis-backed rate limiter whether this user still has quota. That check is cheap, and it guards the most expensive resource in the building, so it belongs at the front door.

The chat service then does the bookkeeping. It persists the new user message, reads earlier turns from the conversation database, and assembles the full prompt the model will see. It also sends the input through a moderation check, because rejecting a harmful request costs milliseconds while generating a reply for it would cost seconds of GPU time.

The assembled prompt goes to the inference router, which holds a queue and dispatches each request to a GPU worker that has spare room. The worker produces tokens one by one; they travel back through the chat service and out to the client over SSE. When generation finishes, the chat service writes the completed reply to the database so the conversation stays whole.

Returning the answer in one piece is the simplest contract, but a long reply can take ten seconds or more, which ruins the first-token goal and risks gateway timeouts. WebSockets give low-latency, bidirectional messaging, yet each connection is stateful and pins a user to one server, which complicates load balancing and deploys.

Server-sent events fit the actual traffic shape. The client opens a normal HTTP request, and the server leaves the response open, writing each token as an event. Proxies and load balancers already understand this, and the only direction we need is server to client. That makes SSE the pick.

Two details complete the design. Since SSE carries no messages upward, the stop button sends a separate cancel request, which tells the worker to abandon the generation and release its batch slot immediately. And the server assembles and stores the full reply on its own, so if a laptop lid closes mid-answer, the text is waiting when the user returns.

Serving the model is where cost is won or lost, and it works as a pipeline. Admission comes first: each request is checked against the user's limits, and anyone over quota gets a 429 straight away rather than waiting in line for capacity they cannot use. Accepted requests join a queue for their target model, where paid plans can be given higher priority. The router only hands work to a worker that reports free room, so requests wait briefly instead of timing out on an overloaded machine.

On the worker, continuous batching keeps the hardware busy. A GPU computes the next token for many sequences almost as cheaply as for one, so at each decoding step the worker admits newly arrived requests into the running batch and drops those that finished. Nobody waits for the slowest sequence in a fixed batch to complete.

Memory, not raw compute, usually sets the ceiling. Each active sequence keeps a key-value cache of its attention state in GPU memory, and a long prompt makes that cache large, so the number of requests that fit at once depends on their combined length. When queues grow, the autoscaler adds workers; if the fleet is still saturated, the system sheds load, for instance by serving free-tier users from a smaller, cheaper model.

Chat history is stored partitioned by conversation ID and ordered by timestamp within each partition. Opening a chat then becomes a single sequential read rather than a scatter across nodes. A wide-column store in the style of Cassandra or DynamoDB matches this access pattern and scales horizontally as the daily 2 TB accumulates.

The model can only attend to a fixed number of tokens, so the chat service must choose what goes into each prompt. It always includes the system prompt, then adds messages from newest to oldest until the token budget is nearly full. For conversations that run much longer, older turns are compressed into a short summary so that facts stated early, such as a user's name or project details, survive.

Because each new turn starts with the same system prompt and history as the previous one, the GPU can reuse the cached computation for that prefix instead of rebuilding it, which shortens time to first token and saves compute. Work that does not affect the reply, like generating a chat title or indexing messages for search, runs in background jobs so it never delays generation.

Batch size is the central dial. Larger batches spread each GPU across more users and cut cost per token, but every individual stream slows slightly. SSE keeps streaming simple and proxy-friendly, at the price of a separate endpoint for cancellation. Sending full history gives the model complete context but burns tokens on every turn, while summaries save tokens and can drop a detail the user later asks about.

Strict quotas protect capacity for the whole user base, yet at peak some people will hit a limit message. A good answer names these costs openly and explains why each choice still fits the goals: start streaming quickly, keep accelerators saturated, and defend the fleet with queues and quotas.