RAG Retrieval, Ranking and Prompt Assembly with Citations

A ferry operator indexes two versions of its timetable: last year's and this year's. A passenger asks when the island ferry leaves. Both chunks match the question almost equally well. Without a rule, the model gets two different departure times and picks one, and nothing in the system can say which choice was right. Retrieval is more than finding similar text. It is deciding which text deserves a place in a limited prompt.

Retrieve: top-k, thresholds and filters

Three controls shape what comes back. Top-k returns the k best chunks; too small misses supporting context, too large floods the prompt with noise. A minimum score refuses weak matches, so that an unanswerable question retrieves nothing instead of the least-bad chunk. Metadata filters narrow the search before scoring: only documents the user may read, only the current version, only the relevant product. Filtering by permission here, before text reaches the prompt, is the safe way to do access control.

A common pattern is to over-retrieve cheaply (say the top 20 by hybrid search), then re-rank to keep the best handful. The first stage is tuned for recall, the second for precision.

Merging a keyword ranking with a vector ranking raises a practical question: their raw scores are on different scales. Reciprocal rank fusion sidesteps this by using only each chunk's position in each list. Its constant (60 here) is a commonly used default that you can tune.

from collections import defaultdict

def reciprocal_rank_fusion(rankings, k=60):
    """Merge several ranked lists of chunk ids into one. Only ranks matter, not raw scores."""
    fused = defaultdict(float)
    for ranking in rankings:
        for position, chunk_id in enumerate(ranking, start=1):
            fused[chunk_id] += 1 / (k + position)
    return sorted(fused, key=fused.get, reverse=True)

# Query: "error E-4021 on door sensor"
keyword_ranking = ["manual-p87", "faq-12", "release-notes", "manual-p12"]   # exact match on the code E-4021
vector_ranking  = ["faq-12", "manual-p12", "manual-p87", "safety-3"]        # similar meaning: sensor faults

for chunk_id in reciprocal_rank_fusion([keyword_ranking, vector_ranking]):
    print(chunk_id)

faq-12
manual-p87
manual-p12
release-notes
safety-3

The two chunks that appear in both lists (faq-12 and manual-p87) float to the top, and the chunk found only by exact code match still outranks the one found only by vector similarity. The ranked lists are made-up illustrations, not output from a real index.

Assemble: a prompt that can be audited

The prompt is where retrieval turns into an answer. Give each chunk a number and its source and date so the model can cite it and you can check it. State the rules plainly: answer only from the sources, cite them, say "I don't know" if the answer is absent, and prefer the newest source when they conflict. Keep a token budget for the sources and stop adding chunks when it is spent. If nothing passes the threshold, do not call the model with an empty context at all; return a fixed "no answer found" response or hand over to a person.

import re

STOP = {"the", "a", "an", "is", "are", "to", "of", "for", "on", "in", "do", "does", "i", "my", "can", "what", "when"}

def words(text):
    return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}

# Each chunk keeps its source and last-updated date so we can cite and rank on them.
CHUNKS = [
    {"id": "tt-2023", "updated": "2023-05-01", "text": "Ferry timetable: the island ferry departs from pier 3 at 08:00 and 17:00."},
    {"id": "tt-2025", "updated": "2025-03-10", "text": "Ferry timetable: the island ferry departs from pier 4 at 08:30 and 17:30."},
    {"id": "bag",     "updated": "2024-09-02", "text": "Baggage rules: each passenger may carry two bags of up to 20 kg each."},
    {"id": "pets",    "updated": "2024-09-02", "text": "Pets travel free on the upper deck if kept on a lead."},
]

def retrieve(question, chunks, k=3, min_overlap=2):
    q = words(question)
    scored = []
    for c in chunks:
        overlap = len(q & words(c["text"]))
        if overlap >= min_overlap:                  # refuse weak matches outright
            scored.append((overlap, c["updated"], c))
    # Rank by overlap first; break ties by newest date.
    scored.sort(key=lambda t: (t[0], t[1]), reverse=True)
    return [c for _, _, c in scored[:k]]

def build_prompt(question, hits, max_chars=600):
    if not hits:
        return None                                  # nothing relevant: do not call the model with an empty context
    blocks, used = [], 0
    for n, c in enumerate(hits, start=1):
        block = f"[{n}] (source: {c['id']}, updated {c['updated']})\n{c['text']}"
        if used + len(block) > max_chars:
            break
        blocks.append(block)
        used += len(block)
    return (
        "Answer using ONLY the numbered sources below. Cite them like [1].\n"
        "If the sources do not contain the answer, reply exactly: I don't know.\n"
        "If sources conflict, prefer the most recently updated one and say so.\n\n"
        "SOURCES\n" + "\n\n".join(blocks) + f"\n\nQUESTION: {question}"
    )

for q in ["What time does the island ferry depart from the pier?", "Can I bring a parrot on board?"]:
    hits = retrieve(q, CHUNKS)
    prompt = build_prompt(q, hits)
    print("=" * 50)
    print(prompt if prompt else f"No relevant context for: {q!r} -> skip the model, say so")

==================================================
Answer using ONLY the numbered sources below. Cite them like [1].
If the sources do not contain the answer, reply exactly: I don't know.
If sources conflict, prefer the most recently updated one and say so.

SOURCES
[1] (source: tt-2025, updated 2025-03-10)
Ferry timetable: the island ferry departs from pier 4 at 08:30 and 17:30.

[2] (source: tt-2023, updated 2023-05-01)
Ferry timetable: the island ferry departs from pier 3 at 08:00 and 17:00.

QUESTION: What time does the island ferry depart from the pier?
==================================================
No relevant context for: 'Can I bring a parrot on board?' -> skip the model, say so

In the first run both timetable chunks match equally on overlap, so the tie-break puts the 2025 chunk first and the prompt tells the model what to do about the conflict. In the second run, the parrot question retrieves nothing because "parrot" shares no word with "pets", and the pipeline refuses rather than guessing. That is the right behaviour for the system and a reminder that this crude word-overlap scorer has the same synonym blindness as the toy in the previous lesson: a real pipeline would use embeddings or hybrid search so that the pets chunk could be found.

Conflicts, freshness and chunk text

• Prefer deletion to disclaimers. If an old document is superseded, remove or deactivate it in the index instead of hoping the model picks the newer one.
• Carry dates and versions in metadata and show them in the prompt, so conflicts are visible.
• Include titles in chunk text. A chunk that begins with "Baggage rules:" is easier to match and easier for the model to interpret.
• Keep citations checkable. Return the chunk IDs and let your application link them to the source page, instead of trusting source names the model typed.
• Treat retrieved text as data. A document can contain instructions, and following them would be a form of prompt injection. Delimit sources and keep the instruction hierarchy clear, as in the prompt injection lesson.

Common mistakes and the symptom each one causes

• No minimum score. Symptom: the system always answers, using irrelevant chunks and producing confident nonsense for out-of-scope questions. Fix: set a threshold and test unanswerable questions.
• Large k to be safe. Symptom: slower, costlier calls and answers distracted by marginal passages. Fix: smaller k plus a re-ranker.
• Indexing old and new versions together. Symptom: answers that quote last year's rule. Fix: version metadata, filter to current, remove superseded documents.
• Vector-only search for identifiers. Symptom: queries containing an error code or product SKU return thematically similar but wrong chunks. Fix: add keyword search.
• No source numbering. Symptom: you cannot tell which passage an answer came from, so reviewers cannot verify it. Fix: number sources and ask for citations.
• Filtering permissions after generation. Symptom: a user's answer contains information from a restricted document. Fix: filter before retrieval or before prompt assembly.

How this shows up in real projects and interviews

Most of the work in a production RAG system after the first demo is in this stage: adding hybrid search for identifiers, a re-ranker for precision, filters for versions and permissions, and thresholds so out-of-scope questions fail politely. In interviews, a good answer to "how would you improve a RAG system that returns wrong answers" starts by looking at what was retrieved for failing questions before touching the model or the prompt. That habit, inspect the retrieved chunks first, is the thread of the final lesson, where we measure retrieval and generation separately.