Large Language Models: Tokens and Context Windows Explained

A support chat on a greenhouse-controller helpdesk runs fine for twenty messages and then the assistant starts ignoring its own instructions. Nobody changed the prompt. The conversation simply grew until something had to give, and what gave was the oldest text, which happened to include the rules. To reason about that kind of failure you need two ideas: tokens and the context window.

Models read tokens, not words

A language model never sees your text as letters or words. A tokenizer first splits it into tokens, which are pieces from a fixed vocabulary, and each piece is mapped to an integer. Common words are usually one token. Rare or long words are built from several pieces, which is how a vocabulary of a limited size can still represent any text: anything can be spelled out of smaller pieces, down to single characters or bytes.

Real tokenizers learn their pieces from statistics over large amounts of text, using methods such as byte-pair encoding. The toy below skips the learning and uses a hand-written vocabulary with greedy longest-match splitting, which is enough to show the behaviour. Real token counts and splits will differ, and they differ between providers, so never treat this as a counting tool.

VOCAB = {
    "un", "break", "able", "lock", "re", "load", "ing", "s",
    "the", "kiosk", "is", "a", " ",
}
# single characters act as a fallback so nothing is ever "unknown"
VOCAB |= set("abcdefghijklmnopqrstuvwxyz")

def tokenize_word(word):
    """Greedy longest-match split of one word into pieces from VOCAB."""
    pieces = []
    i = 0
    while i < len(word):
        for j in range(len(word), i, -1):      # try the longest slice first
            if word[i:j] in VOCAB:
                pieces.append(word[i:j])
                i = j
                break
        else:
            pieces.append("<unk>")             # character not in vocab at all
            i += 1
    return pieces

def tokenize(text):
    out = []
    for word in text.lower().split():
        out.extend(tokenize_word(word))
    return out

for sample in ["unbreakable", "reloading", "kiosk", "zzyzx", "the kiosk is unlockable"]:
    toks = tokenize(sample)
    print(f"{sample!r:28} -> {len(toks):2} tokens {toks}")

'unbreakable'                ->  3 tokens ['un', 'break', 'able']
'reloading'                  ->  3 tokens ['re', 'load', 'ing']
'kiosk'                      ->  1 tokens ['kiosk']
'zzyzx'                      ->  5 tokens ['z', 'z', 'y', 'z', 'x']
'the kiosk is unlockable'    ->  6 tokens ['the', 'kiosk', 'is', 'un', 'lock', 'able']

Three things to notice. "unlockable" was not in the vocabulary, yet it is representable as un + lock + able, so the model can reason about words it never saw whole. "zzyzx" fell back to one token per letter, which is why gibberish, long identifiers and unusual names cost more tokens than their length suggests. And the model sees integers for these pieces, not spelling, which is one reason models can stumble on tasks like counting the letters in a word.

What a context window actually is

The context window is the maximum number of tokens a model can consider in one call. It is a single budget shared by everything: your instructions, the conversation so far, any documents you paste in, the new question and the answer the model is about to write. The exact limit depends on the model and provider, so read it from the documentation instead of assuming.

Two consequences surprise people. First, the model has no memory between calls. A chat feels continuous only because your application resends the earlier turns every time, and every resent turn costs tokens and time. Second, when the history no longer fits, something must be cut, and if you cut carelessly you cut the instructions.

Budgeting and trimming in code

The function below keeps the system message always and then adds the newest turns until the budget is used up. It uses a deliberately crude length-based estimate; in a real application you would count with the tokenizer your provider documents.

def trim_history(messages, budget, count):
    """Keep the system message plus as many recent turns as fit in `budget`."""
    system, turns = messages[0], messages[1:]
    kept, used = [], count(system)
    for msg in reversed(turns):                 # newest first
        cost = count(msg)
        if used + cost > budget:
            break
        kept.append(msg)
        used += cost
    return [system] + list(reversed(kept)), used

rough = lambda text: max(1, len(text) // 4)     # crude: about 4 characters per token

chat = [
    "SYSTEM: You are a helpdesk assistant for a greenhouse irrigation controller.",
    "USER: Zone 3 valve will not close after the morning cycle.",
    "BOT: Please check whether the manual override lever is lifted.",
    "USER: It is down, and the status light is still amber.",
    "BOT: Power-cycle the controller, then run a 10 second test on zone 3.",
    "USER: The test ran. Does the amber light mean the schedule is lost?",
]
kept, used = trim_history(chat, budget=60, count=rough)
print("tokens used:", used, "of 60")
print("kept", len(kept), "of", len(chat), "messages:")
for m in kept:
    print("-", m)

tokens used: 52 of 60
kept 3 of 6 messages:
- SYSTEM: You are a helpdesk assistant for a greenhouse irrigation controller.
- BOT: Power-cycle the controller, then run a 10 second test on zone 3.
- USER: The test ran. Does the amber light mean the schedule is lost?

Dropping whole turns from the oldest end is the simplest policy. Better ones summarise the dropped part into a short note, or keep a few pinned facts (the customer's controller model, the zone they are asking about) outside the trimmed history.

Common mistakes and the symptom each one causes

• Counting words instead of tokens. Symptom: a request that "should" fit is rejected with a context-length error, or a long document is silently cut. Fix: count with the provider's tokenizer.
• Forgetting to reserve room for the answer. Symptom: replies that stop mid-sentence, often with a finish reason of "length" in the API response. Fix: budget input and output together.
• Trimming the system prompt along with old turns. Symptom: the assistant drifts off-role or ignores rules after a long chat. Fix: pin instructions and trim only the history.
• Assuming the model remembers an earlier call. Symptom: it asks again for details you gave in a different request. Fix: send what it needs every time.
• Estimating from characters across languages. Symptom: a rule of thumb that works for English underestimates other scripts, code and numbers. Fix: measure samples of your real traffic.

How this shows up in real projects and interviews

Every chat product eventually implements something like trim_history, usually with summarisation of older turns. Interviewers like the questions "why do models struggle to count letters or do exact arithmetic?" (they see tokens, not characters) and "how would you answer questions over a 500-page document?" (you would not paste it all in: split it into chunks and retrieve the relevant ones, which is the subject of retrieval-augmented generation). A solid answer states that the window is a shared budget, that cost and latency grow with tokens, and that you measure instead of trusting rules of thumb. Next, the lesson on how generation works shows what happens to those tokens inside the model.