Where should you split a document without breaking its meaning?

A policy says, “Refunds are available within seven days. This does not apply to downloaded materials.” Your splitter puts the second sentence in another chunk. A retrieved first chunk can now support an overbroad answer. The boundary changed what evidence the model could see.

Before you begin: Understand retrieval records and context budgets.

Define the unit of retrieval

A chunk is a segment indexed and retrieved as a unit. Small chunks can focus on a specific fact but lose definitions, exceptions, or referents. Large chunks preserve more context but may dilute relevance, consume more prompt space, and include distracting material.

There is no universal best chunk size. The source structure, questions, embedding model's input limit, and answer context budget all matter. Treat a size as a hypothesis to evaluate, not a rule that every document must obey.

Respect structure before counting

Headings, paragraphs, lists, table headers, and code blocks often carry meaningful boundaries. A recipe's ingredient list and instruction steps have different structures from a policy clause. A table row without its column headers can be misleading even if it fits neatly into a token budget.

Preserve the section title and source locator with each chunk. If a passage says “these conditions,” retain or retrieve the conditions it refers to. A parent-child arrangement can index smaller pieces while returning a larger surrounding section when needed.

Examine overlap explicitly

Overlap repeats some material between neighboring chunks. It can preserve a sentence that crosses a boundary, but it also increases storage and can make retrieval return near-duplicates. Repetition is not a substitute for understanding document structure.

This complete Python 3 example splits a sequence of words. It deliberately does not claim token-aware or semantic chunking. Use it to inspect overlap behavior before choosing a real tokenizer-based method.

# Runnable: Python 3, standard library.
def windows(words, size, overlap):
    if size <= 0 or overlap < 0 or overlap >= size:
        raise ValueError('Require size > overlap >= 0')
    result = []
    start = 0
    while start < len(words):
        end = min(start + size, len(words))
        result.append((start, end, ' '.join(words[start:end])))
        if end == len(words):
            break
        start = end - overlap
    return result

words = 'A B C D E F G H I'.split()
assert windows(words, 5, 2) == [(0, 5, 'A B C D E'), (3, 8, 'D E F G H'), (6, 9, 'G H I')]
print(windows(words, 5, 2))

The offsets refer to words in this example. A real ingestion system needs offsets that reliably locate the original text or page, especially after normalization.

Test with questions near boundaries

Create questions requiring a definition and its example, a rule and exception, a table row and header, and a code function with its explanation. For each, inspect whether the retrieved context contains all necessary parts.

Measure retrieval recall and answer support, not only the number of chunks. A strategy that produces more chunks may improve or worsen evidence coverage. Deduplicate or diversify results when overlapping chunks crowd out other relevant sources.

Keep indexing and display separate

The text embedded for search may include a section title or short contextual prefix. The displayed citation should still make clear which words came from the original source and which were added for indexing. Do not present a generated chunk summary as an exact quotation.

Record splitter version, size measure, overlap, and normalization with the index. Changing chunking changes record boundaries and may require re-embedding and citation remapping.

Choose a repair

Refund answers repeatedly omit the downloaded-material exception. The exception is always in the next chunk. Would doubling the database's query speed solve the problem?

Next, hybrid search will combine exact wording and learned similarity without pretending their raw scores share the same scale.

Sources

The Sentence Transformers retrieval guide demonstrates passage-level retrieval. Lost in the Middle motivates testing how supplied context is used rather than equating more context with better answers.

Continue to the next lesson.