Advanced chunking: preserve the evidence a question needs

A cancellation rule is retrieved without the exception in the next paragraph. How can a highly relevant chunk still lead to the wrong answer? Chunking must preserve enough evidence to answer the question, not merely match its words.

Before you begin: You understand embeddings, retrieval ranking, and basic document chunking.

Chunking decides what the retriever can select. It therefore affects correctness, not just text formatting. The useful unit is the smallest piece that can be found reliably and expanded into sufficient evidence.

Separate three units

The source unit is the original document or record. The search unit is the piece indexed for matching. The answer context is what the model receives after retrieval, filtering, and expansion.

These can differ. Small child chunks may match a specific question well, while a parent section supplies the exception or definition needed to answer. Expanding a child is useful only if the parent is relevant, fits the context budget, and is permitted for the same user.

Use a real parser when structure matters. A regular expression that treats every line starting with # as a Markdown heading can split inside a code fence. A table broken into plain fragments can lose which number belongs to which column.

Start with a baseline you can inspect

This example splits the body of one already-parsed section into overlapping word windows. It deliberately does not parse Markdown or count model tokens. Its purpose is to show metadata propagation and boundary behavior without dependencies.

# Runnable: Python 3, standard library.
def split_section(document, version, section, text, size=8, overlap=2):
    if size <= 0 or not 0 <= overlap < size:
        raise ValueError('Require 0 <= overlap < size')
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = min(start + size, len(words))
        chunks.append({
            'id': f'{document}:{version}:{section}:{start}',
            'parent': f'{document}:{version}:{section}',
            'word_start': start,
            'word_end': end,
            'text': ' '.join(words[start:end]),
        })
        if end == len(words):
            break
        start = end - overlap
    return chunks

text = 'Members may cancel early. Guest events follow separate rules.'
chunks = split_section('policy', 'v2', 'cancellation', text)
assert chunks[-1]['word_end'] == len(text.split())
assert all(c['parent'] == 'policy:v2:cancellation' for c in chunks)
assert split_section('policy', 'v2', 'empty', '') == []
print(chunks)

The loop stops once the final word is covered, avoiding a redundant trailing overlap-only chunk. Whitespace is normalized, so these word offsets are not original character offsets. A citation system should preserve source locations separately rather than pretending the reconstructed text is byte-identical.

For production limits, count with the relevant tokenizer, preserve semantic structures, and record the parser and splitter versions. Changing either can change chunk IDs and require reindexing.

Score complete support instead of one relevant hit

For a guest-event cancellation question, label two required evidence units: rule and exception. A retrieval result containing only rule has a relevant hit but incomplete support. Returning five overlapping copies of the rule still leaves the same gap.

This complete local program compares fictional retrieval results. It measures the evidence labels supplied here, not a real retriever's quality.

# Runnable: Python 3, standard library.
required = {"rule", "exception"}
strategies = {
    "small_windows": ["rule", "rule"],
    "parent_section": ["rule", "exception", "unrelated"],
    "scoped_expansion": ["rule", "exception"],
}
for name, retrieved in strategies.items():
    unique = set(retrieved)
    coverage = len(required & unique) / len(required)
    complete = required <= unique
    duplicates = len(retrieved) - len(unique)
    print(name, coverage, complete, duplicates)
assert not required <= set(strategies["small_windows"])
assert required <= set(strategies["scoped_expansion"])

Add a second question requiring a table's row label and column unit. This makes the evaluation sensitive to structural boundaries as well as paragraph overlap.

Metadata must survive expansion

Keep document identity, source version, section or page location, content hash, and access metadata. Resolve permissions before text reaches a reranker or generator. When expanding a child, authorize the parent too; a permitted paragraph must not unlock a restricted neighboring section.

Do not let retrieved old versions silently outrank current authoritative records. Version selection belongs in ingestion and retrieval rules, not only in a prompt asking the model to “be current.”

Evaluate boundaries with real questions

Create questions whose supporting evidence is labeled at source level. Include answers spanning adjacent paragraphs, table rows, exceptions, and short references such as “this rule.” Keep the questions, embedding model, and retrieval settings fixed while comparing chunking strategies.

Measure whether the required evidence appears, how early it appears, how much duplicate context is returned, and whether the final answer uses the full evidence. A hit on one relevant fragment is insufficient when the question needs two fragments together.

Overlap can recover a boundary fact, but it can also fill top results with near-duplicates. Parent expansion can restore meaning, but it can also add distractions and consume the prompt budget. Tune these choices with measured failure cases rather than a universal token-size rule.

Exercise: two adjacent chunks separately contain the cancellation rule and its exception. Retrieval repeatedly returns only the rule. Name two experiments and one risk for each.

Next, improve which candidates reach the model after they have been chunked correctly.

Sources

Sentence Transformers' retrieve-and-rerank guide explains the retrieval pipeline around these units. Retrieval-Augmented Generation supplies the broader retrieval-plus-generation setting. The splitter and policy example here are original teaching baselines.

Continue: Re-ranking and Hybrid Search.