Evaluating RAG: Retrieval Metrics, Faithfulness and Failure Modes

A support team launches a warranty assistant, and after a week someone reports that it told a customer a cracked jug was not covered when it was. Is the model at fault, the prompt, the index or the data? Without measurements, every one of those is a guess, and teams burn days rewriting prompts for a problem that lives in the retriever. The core discipline of RAG evaluation is simple: test retrieval and generation separately.

Two halves, two sets of metrics

If the right chunk is not retrieved, no prompt can save the answer. If it is retrieved and the answer is still wrong, retrieval is fine and the problem is in generation. So you score each half on its own.

Recall@k is the fraction of questions whose correct chunk appears in the top k results. MRR averages 1 divided by the rank of the first correct chunk, so a first-place hit scores 1, second place 0.5 and a miss 0. The script below computes both for a small, honest test set over the warranty chunks, using the same kind of crude word-overlap retriever as before so that you can see exactly why it misses.

import re

DOCS = {
    "w1": "Warranty covers the motor of the blender for 24 months from purchase.",
    "w2": "The glass jug is not covered by the warranty if it is cracked or chipped.",
    "w3": "To claim, send the receipt and a photo of the serial number to the repairs desk.",
    "w4": "Return the blender within 30 days for a full refund if unused.",
}

def toks(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def retrieve(question, k=2):
    q = toks(question)
    ranked = sorted(DOCS, key=lambda d: len(q & toks(DOCS[d])), reverse=True)
    return ranked[:k]

# Labelled test set: a question plus the chunk id(s) that actually contain the answer.
TESTS = [
    ("How long is the motor warranty?",              {"w1"}),
    ("Is a cracked jug covered by warranty?",        {"w2"}),
    ("What do I send to make a claim?",              {"w3"}),
    ("Can I get my money back on an unused blender?", {"w4"}),
    ("What if the glass breaks?",                    {"w2"}),
    ("How do I get my money back?",                  {"w4"}),
]

def evaluate(k):
    hits, rr_sum = 0, 0.0
    for question, gold in TESTS:
        ranked = retrieve(question, k=len(DOCS))     # full ranking for MRR
        top_k = set(ranked[:k])
        hits += bool(top_k & gold)
        for pos, doc_id in enumerate(ranked, start=1):
            if doc_id in gold:
                rr_sum += 1 / pos
                break
    n = len(TESTS)
    return hits / n, rr_sum / n

for k in (1, 2):
    recall, mrr = evaluate(k)
    print(f"k={k}: recall@k={recall:.2f}  MRR={mrr:.2f}")

print()
for question, gold in TESTS:
    got = retrieve(question, k=1)[0]
    print(("ok  " if got in gold else "MISS"), question, "->", got)

k=1: recall@k=0.83  MRR=0.88
k=2: recall@k=0.83  MRR=0.88

ok   How long is the motor warranty? -> w1
ok   Is a cracked jug covered by warranty? -> w2
ok   What do I send to make a claim? -> w3
ok   Can I get my money back on an unused blender? -> w4
ok   What if the glass breaks? -> w2
MISS How do I get my money back? -> w1

Five of six questions find their chunk at rank one. The miss, "How do I get my money back?", shares no words with the refund chunk ("refund", "unused", "full"), so it ranks last and even k=2 does not rescue it. That points to the fix: a retriever that understands meaning, or query rewriting that expands "money back" to "refund". Notice that a test set with only the five easy questions would have reported a perfect score and hidden the problem. Write questions the way users phrase them, not the way the documents do.

Evaluating the generation half

Feed the generator the correct chunk by hand and check its answers. If they are wrong with perfect context, fix the prompt or the model choice. Then check faithfulness: does each claim in the answer appear in the supplied passages? Ask reviewers to flag unsupported statements, or use a second model as a grader once you have spot-checked its judgements against humans, since graders make mistakes too. Also test abstention with questions the documents do not answer; a good system says so instead of improvising. Keep scores per category so a gain in one area cannot hide a loss in another.

To replay a complaint you need the right evidence logged for every answer. A minimal record, shown with invented values, is enough to tell which stage failed.

request_id:     r-20250611-0042
question:       "Is a cracked jug covered by warranty?"
index_version:  2025-06-10
retrieved:      [w2 (0.41), w1 (0.18)]
prompt_version: answer-v7
model_config:   <name and settings recorded here>
answer:         "No. The glass jug is not covered ... [1]"
user_feedback:  thumbs_down
-> add to test set: question + expected chunk w2

Failure modes and how to recognise them

Common mistakes and the symptom each one causes

• Only testing easy questions. Symptom: perfect offline scores and unhappy users, as in the example above. Fix: mine real queries and include paraphrases.
• Judging only the final answer. Symptom: endless prompt tweaks that never help because the chunk was never retrieved. Fix: log and score retrieval separately.
• Letting test chunks leak into tuning. Symptom: scores rise while real quality is flat. Fix: keep a held-out set you never tune on.
• No unanswerable questions. Symptom: the system never says "I don't know". Fix: include out-of-scope questions and score abstention.
• Never refreshing the index. Symptom: answers slowly drift out of date as policies change. Fix: re-index on document change and monitor the age of the newest chunk.
• Trusting a model grader blindly. Symptom: faithfulness scores look great while humans disagree. Fix: compare grader and human labels on a sample.

How this shows up in real projects and interviews

Mature RAG teams keep a living question set, rerun it on every change to chunking, embeddings, prompts or models, and add every reported failure to it. They log the question, retrieved chunk IDs, prompt version and answer, so a complaint can be replayed. In interviews, asking "how do you know your RAG system is good?" is a favourite, and the answer is the two-half approach above, with retrieval metrics, faithfulness and abstention, a held-out set and ongoing monitoring. Be candid about what you cannot guarantee: no metric proves the absence of wrong answers, which is why high-stakes uses keep a human in the loop. To revisit the foundations, see Large Language Models and Prompt Engineering.