RAG Chunking and Embeddings: Build a Toy Vector Search in Python

A bike-share company indexes its member guide. A rider asks how long a ride can last before extra fees. The answer is in the guide, but the index split the text so that the sentence about the 90-minute limit landed at the very end of one chunk and the sentence about the fee at the start of the next. Neither chunk alone answers the question. Chunking is not plumbing; it decides what the model will ever be able to see.

Chunking: cutting documents into retrievable pieces

A language model cannot take a whole library per question, and a search over whole documents is too coarse: a 40-page manual matches almost every query a little and none well. So documents are split into chunks, each small enough to be specific and large enough to make sense alone.

There is no universally correct chunk size. Small chunks make precise matches but lose surrounding context; large chunks keep context but dilute the match and cost more tokens per retrieved passage. A practical starting point is a few hundred tokens with modest overlap, then adjust against a test set. Whatever you choose, store metadata with each chunk (source file, section title, date, permissions), because you will need it for citations, filtering and debugging.

id:        bike-guide-late-fees-02
source:    member-guide.pdf, section "Late fees"
updated:   2025-02-11
audience:  all-members
text:      "Late fees. A ride longer than 90 minutes is charged an
            extra fee for every additional 30 minutes. ..."
embedding: [0.012, -0.087, 0.301, ... ]   <- produced by an embedding model

Embeddings, with a toy you can run

To search by meaning you need a way to turn text into numbers so that similar texts get similar numbers. That is an embedding: a vector for each chunk, compared with a measure such as cosine similarity. The script below builds a deliberately simple version: a sparse TF-IDF vector, where each word is weighted higher if it is rare across the chunks. It also does paragraph-based chunking with overlap, so you can see the boundary effect from the opening story.

import math
import re
from collections import Counter

DOC = """Riverside Bike Share: member guide

Unlocking a bike. Hold your membership card against the reader on the rear mudguard until the light turns green. The lock releases after two seconds.

Returning a bike. Push the front wheel into any empty dock and wait for the green light. If the light stays red, lift the wheel out and push it in again.

Late fees. A ride longer than 90 minutes is charged an extra fee for every additional 30 minutes. Fees are added to your next monthly invoice.

Reporting damage. If a bike has a flat tyre or a loose chain, press the red fault button on the dock and the bike is taken out of service."""

def split_paragraphs(text):
    return [p.strip() for p in text.split("\n\n") if p.strip()]

def chunk(paragraphs, max_words=40, overlap_words=8):
    """Pack paragraphs into chunks of about max_words, repeating the tail of the previous chunk."""
    chunks, current = [], []
    for para in paragraphs:
        words = para.split()
        if current and len(current) + len(words) > max_words:
            chunks.append(" ".join(current))
            current = current[-overlap_words:]          # carry some context forward
        current.extend(words)
    if current:
        chunks.append(" ".join(current))
    return chunks

def tokens(text):
    return re.findall(r"[a-z0-9]+", text.lower())

def build_idf(chunks):
    n = len(chunks)
    df = Counter(word for c in chunks for word in set(tokens(c)))
    return {w: math.log((1 + n) / (1 + d)) + 1 for w, d in df.items()}

def embed(text, idf):
    """A toy 'embedding': a sparse TF-IDF vector stored as a dict."""
    tf = Counter(tokens(text))
    return {w: c * idf[w] for w, c in tf.items() if w in idf}

def cosine(a, b):
    dot = sum(v * b.get(w, 0.0) for w, v in a.items())
    na = math.sqrt(sum(v * v for v in a.values()))
    nb = math.sqrt(sum(v * v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

chunks = chunk(split_paragraphs(DOC.split("\n\n", 1)[1]))   # drop the title line
idf = build_idf(chunks)
vectors = [embed(c, idf) for c in chunks]

print(f"{len(chunks)} chunks")
for i, c in enumerate(chunks):
    print(f"  #{i} ({len(c.split())} words) starts: {c[:45]}...")

queries = ["how do I get the bike lock to open", "extra charge for keeping a bike too long",
           "my tyre is flat", "puncture"]
for q in queries:
    qv = embed(q, idf)
    score, best = max((cosine(qv, v), i) for i, v in enumerate(vectors))
    print(f"\nQ: {q}")
    if score == 0:
        print("   no chunk shares a single word with the query")
    else:
        print(f"   best chunk #{best} (score {score:.2f}): {chunks[best][:60]}...")

4 chunks
  #0 (25 words) starts: Unlocking a bike. Hold your membership card a...
  #1 (39 words) starts: turns green. The lock releases after two seco...
  #2 (34 words) starts: the wheel out and push it in again. Late fees...
  #3 (37 words) starts: Fees are added to your next monthly invoice. ...

Q: how do I get the bike lock to open
   best chunk #0 (score 0.34): Unlocking a bike. Hold your membership card against the read...

Q: extra charge for keeping a bike too long
   best chunk #2 (score 0.23): the wheel out and push it in again. Late fees. A ride longer...

Q: my tyre is flat
   best chunk #3 (score 0.30): Fees are added to your next monthly invoice. Reporting damag...

Q: puncture
   no chunk shares a single word with the query

Read the output closely. The first two questions find the right guidance because they share distinctive words with it (lock in the first, extra in the second). Chunk 2 starts with the tail of the previous paragraph, which is the overlap doing its job; it is also why chunks can begin mid-sentence, which is acceptable for retrieval but worth cleaning if you show chunks to users. The "my tyre is flat" question lands on the damage chunk. And the last question, "puncture", scores zero everywhere, because no chunk contains that exact word, even though a flat tyre and a puncture are the same problem.

What real embeddings change

Real embeddings come from a trained embedding model, a neural network that maps a piece of text to a dense vector of a few hundred to a few thousand numbers. It is trained so that texts with similar meaning end up close together, regardless of the exact words. That is how "puncture" can retrieve a chunk about flat tyres, which the toy cannot do. You do not build these by hand: you call an embedding model from a provider or run an open one yourself, then store the vectors in a vector index or database and search by nearest neighbours. For a few thousand chunks, a plain in-memory brute-force comparison is often fast enough; specialised approximate indexes matter at larger scale.

Embeddings are strong on meaning and weak on exact strings. Part numbers, error codes, names and rare acronyms can be missed by pure vector search, which is why many systems combine it with keyword search, covered in the next lesson.

Common mistakes and the symptom each one causes

• Chunks cut mid-answer. Symptom: the right document is retrieved but the model says the information is missing, because the other half is in the next chunk. Fix: split on structure, add overlap, test with questions whose answers straddle boundaries.
• Chunks too large. Symptom: vague matches and prompts full of irrelevant text. Fix: smaller chunks or a re-ranking step.
• Chunks too small. Symptom: a retrieved sentence with no subject ("It must be returned within 24 hours") that the model cannot interpret. Fix: include the section heading in each chunk's text.
• Mixing embedding models. Symptom: results that look random after a partial upgrade. Fix: re-embed everything together and record the model name in the index.
• Dropping metadata. Symptom: you cannot cite sources, filter by date or explain a bad result. Fix: store source, section and date with every chunk.
• Embedding messy text. Symptom: navigation menus, footers and page numbers match queries. Fix: clean and de-duplicate before chunking.

How this shows up in real projects and interviews

Teams spend more time on chunking and data cleaning than on any model choice, and a surprising share of "the AI is bad" tickets turn out to be "the chunk never contained the answer". Interviewers ask how you would chunk a given kind of document (a legal contract, a code repository, a table-heavy PDF) and expect you to talk about structure, overlap, metadata and testing with real questions, rather than naming one magic size. They may also ask how embeddings are compared; answer with cosine similarity or dot product over normalised vectors, and say that approximate nearest-neighbour indexes trade a little accuracy for speed at scale. Next, what to do with the nearest chunks once you have them.