Why RAG? Retrieval-Augmented Generation vs Fine-Tuning Explained

A university lab wants an assistant that answers safety questions from its own handbook: can acetone be stored next to this cabinet, who signs off a waste pickup, what is the booking rule for the fume hood. A general-purpose model has never read that handbook. Ask it anyway and it will produce something fluent and generic, possibly wrong for this lab. There are three ways to bring the handbook to the model, and choosing between them is the first real design decision.

Three ways to give a model your knowledge

Prompt stuffing puts the documents straight into the prompt. Fine-tuning continues training the model on your data so some of it ends up in the weights. Retrieval-augmented generation keeps the documents outside the model, searches them for each question and puts only the relevant passages into the prompt. The model then answers from text that is in front of it, which is both more reliable than recall from weights and easy to update.

The pipeline in one picture

INDEX TIME (offline; repeat when documents change)
  documents -> clean -> split into chunks
            -> embed each chunk -> store vectors + metadata

QUERY TIME (every question)
  question -> embed -> find nearest chunks -> (re-rank)
           -> build prompt with chunks -> model
           -> answer + citations

Everything interesting happens in the lessons that follow. For now, notice that the language model is only the last step. In a RAG system that disappoints, the fault usually lies earlier: the right passage never reached the prompt.

Do you need retrieval at all?

Before building anything, check whether the whole knowledge base fits in the prompt. If it does, and the per-call cost is acceptable, pasting it in is the simplest and often the most accurate approach, since the model sees everything and nothing can be missed by a search step. The script below does the arithmetic for the lab handbook, using placeholder numbers that you should replace with your own measurements.

# Does the knowledge base fit in the prompt, or do we need retrieval?
# All numbers are illustrative placeholders. Use your own token counts and your model's real window.
TOKENS_PER_WORD = 1.4            # rough English average; measure with your tokenizer
CONTEXT_BUDGET = 8_000           # tokens you are willing to spend on documents per call

handbook = {
    "chemical-storage.md":   1_900,    # words per document
    "fume-hood-rules.md":      800,
    "incident-reporting.md":   650,
    "waste-disposal.md":     2_300,
    "equipment-booking.md":    700,
}

total_tokens = int(sum(handbook.values()) * TOKENS_PER_WORD)
print(f"whole handbook: about {total_tokens} tokens (budget {CONTEXT_BUDGET})")

if total_tokens <= CONTEXT_BUDGET:
    print("-> small enough to paste in every call; try that before building retrieval")
else:
    top_k, chunk_tokens = 4, 300
    print(f"-> too big; retrieve {top_k} chunks of ~{chunk_tokens} tokens = {top_k * chunk_tokens} tokens per call")
    print(f"   that is {top_k * chunk_tokens / total_tokens:.0%} of the full handbook per question")

whole handbook: about 8890 tokens (budget 8000)
-> too big; retrieve 4 chunks of ~300 tokens = 1200 tokens per call
   that is 13% of the full handbook per question

Here the handbook is just over budget, so retrieval earns its place, and each question would use around an eighth of the tokens. If the handbook were half the size, pasting it all in would be the better first version. Recheck whenever the corpus or your budget changes.

When fine-tuning wins, and when to combine

• Behaviour and style: a consistent tone, a rigid output format or domain jargon handled the way your team writes it. Fine-tuning shapes how the model answers, not which facts it can cite.
• Short prompts at scale: if a long fixed instruction block is costing a lot on every call, tuning can bake it in. Compare against prompt caching, if your provider offers it, before committing.
• Facts that must be current or traceable: use retrieval. Teaching changing facts through training makes stale answers hard to find and fix.
• Both: it is common to fine-tune for format and use retrieval for facts. Do the cheap thing first, measure, and add complexity only when a test set shows it pays off.

Questions RAG handles poorly

RAG finds a handful of relevant passages. Questions that need the whole corpus break that assumption. "How many procedures mention fume hoods?" is a counting query over everything; "summarise the differences between this year's and last year's handbook" needs two complete documents side by side. For those, use a database query or structured extraction, run summarisation across the full text in stages, or let a model call tools that can search and count, as described in AI agents.

Common mistakes and the symptom each one causes

• Fine-tuning to "teach" facts. Symptom: answers that sound right, mix up details and cannot be traced to a source. Fix: put facts in retrieval, use tuning for behaviour.
• Building retrieval for a corpus that fits in the prompt. Symptom: a complex pipeline that misses passages a simple paste would have caught. Fix: do the size check first.
• Expecting global answers from top-k chunks. Symptom: confident counts and summaries based on three paragraphs. Fix: route aggregate questions to different tools.
• No evaluation before launch. Symptom: nobody can say whether a change made things better. Fix: build the test set described in the last lesson of this course.
• Ignoring permissions. Symptom: a user sees an answer drawn from a document they should not read. Fix: filter chunks by access rights before they reach the prompt, not after.

How this shows up in real projects and interviews

"Chat with our documents" is one of the most requested language-model features in companies, and RAG is the default design. In interviews you will be asked to compare RAG and fine-tuning, and the answer interviewers hope for is a decision rule, not a slogan: facts that change or need citations go through retrieval, behaviour and format go through tuning, and small corpora may need neither. Mentioning access control, evaluation and the cost of keeping an index fresh signals that you have thought beyond the demo. The next lesson opens the pipeline at its first step: splitting documents and turning them into vectors.