RAG Chunking: Preserve Evidence Boundaries Before Tuning Chunk Size

Reviewed October 6, 2026. The local example uses Python 3.12.14 and original synthetic text. No embedding service, language model, or retrieval benchmark was called.

A retrieval system returns a paragraph saying a refund is available, but omits the sentence restricting that policy to one product category. A smaller chunk made the text easier to retrieve while separating the claim from its qualification. Increasing overlap can help, but it can also return the same evidence several times and crowd other sources out of the context.

Chunking is therefore an evidence-design decision. The goal is not simply to divide a document into pieces that fit a model input limit. A useful chunk needs enough context to support a claim, traceable identity, and a relationship to surrounding material when the answer requires more than one passage.

Practical rule: preserve document and section boundaries where possible, carry source metadata with every chunk, measure actual model-token budgets, and evaluate retrieval against answer-supporting evidence. Treat chunk size and overlap as experiment parameters, not universal best practices.

A chunk needs a source identity

At minimum, preserve document ID, source version, section label, and position within the original text. A display title helps readers interpret the passage, while a stable identifier lets the system link overlapping chunks back to the same source evidence.

A retrieved passage may be accurate for an old policy and wrong for the current question. Publication or effective-date metadata therefore belongs to the retrieval contract when time matters. An access-control rule also needs to remain attached to the source; chunking must not make restricted passages discoverable to an unauthorized reader.

The example demonstrates identity and span bookkeeping. It does not implement access enforcement or a live policy database. Those requirements belong to the real system's retrieval boundary and need separate tests.

Separate structural splitting from size splitting

Start with the document's natural units: headings, paragraphs, table rows, code blocks, or numbered procedures. A table's header and footnote can be necessary to interpret its values. Splitting a row from its unit or a procedure from its prerequisite can damage evidence even if every character survives somewhere.

Then subdivide oversized units under a measured budget. The Azure AI Search chunking guide describes fixed-size, structural, semantic, and combined approaches. Its suggested sizes are starting points for that environment, not proof of the optimum for every corpus and question.

Characters, words, and model tokens are different units. The ratio varies with language, code, punctuation, and tokenizer. Count using the tokenizer associated with the actual embedding or generation model, including any metadata text inserted into its input.

A reproducible splitter that exposes its limits

The example receives already separated sections. It uses character windows to make positions directly inspectable. It does not claim sentence awareness, token-budget compliance, or semantic optimality. Those are explicit extension requirements.

def chunk_sections(document_id, version, sections, size=20, overlap=5):
    if type(size) is not int or type(overlap) is not int:
        raise ValueError("size and overlap must be integers")
    if size <= 0 or not 0 <= overlap < size:
        raise ValueError("require size > 0 and 0 <= overlap < size")
    chunks = []
    for section_index, section in enumerate(sections):
        text = section["text"]
        start = 0
        while start < len(text):
            end = min(start + size, len(text))
            chunks.append({
                "id": f"{document_id}:{version}:{section_index}:{start}-{end}",
                "document_id": document_id, "version": version,
                "section": section["heading"], "section_index": section_index,
                "start": start, "end": end, "text": text[start:end],
            })
            if end == len(text):
                break
            start = end - overlap
    return chunks

sections = [{"heading": "Scope", "text": "ABCDEFGHIJKLMNOPQRSTUVWXYZ"},
            {"heading": "Exceptions", "text": "1234567890"}]
chunks = chunk_sections("policy-01", "v1", sections, size=10, overlap=3)
assert [(c["section_index"], c["start"], c["end"]) for c in chunks] == [
    (0, 0, 10), (0, 7, 17), (0, 14, 24), (0, 21, 26), (1, 0, 10)]
assert chunks[0]["text"][-3:] == chunks[1]["text"][:3]
assert len({c["id"] for c in chunks}) == len(chunks)
assert all(c["version"] == "v1" for c in chunks)
assert chunk_sections("empty", "v1", [{"heading": "Empty", "text": ""}]) == []
for bad_size, bad_overlap in [(0, 0), (10, 10), (10, -1), (True, 0)]:
    try:
        chunk_sections("policy-01", "v1", sections, bad_size, bad_overlap)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid window policy accepted")
print([(c["id"], c["text"]) for c in chunks])

The input contract requires section dictionaries with heading and text strings. This small function validates its window parameters, not every possible external document shape. Parse and validate a real document before passing it to this boundary.

The Exceptions section starts a separate coordinate space; no window crosses into it. In a real corpus, the heading could be included in the embedding text and stored separately for display. Adding it changes the token budget, so budget the complete model input rather than only the passage body.

Overlap helps continuity and creates duplication

Shared text can preserve a sentence or qualification near a boundary. However, more overlap increases stored text and can produce multiple highly similar retrieval hits. If three hits repeat one paragraph, the answer context may have less diversity than a nominal top-three count suggests.

Deduplicate or merge retrieval evidence using source identity and spans where appropriate. Keep the display provenance for any merged passage. Do not count repeated overlapping chunks as independent documents confirming a claim.

A fixed overlap also does not guarantee that an entire logical qualification survives. A long sentence, nested list, or table annotation can exceed the shared region. Test the actual evidence relationship: does the retrieved material include the scope needed to support the answer?

Evaluate chunks through the questions they must answer

Create a held-out question set with expected supporting passages. Include single-section facts, answers requiring multiple sections, date-specific policy questions, ambiguous requests, and questions the corpus cannot answer. Record evidence IDs at the appropriate level so multiple overlapping windows around one passage do not artificially inflate success.

Keep retrieval evaluation separate from answer evaluation. A model can generate a wrong answer from correct evidence, or a plausible answer from insufficient evidence. Measure the stages independently before deciding chunking is the cause.

If you compare chunk sizes, hold the corpus snapshot, embedding model, retrieval settings, and question split stable or document the changes. A larger chunk may increase a simple hit metric while consuming more context and including more irrelevant text. Report the tradeoff rather than announcing a universal winner.

Plan for updates and inspectable citations

When a document changes, old chunks need a defined retirement or version-selection policy. A source URL alone may point to a newer document than the passage originally embedded. Preserve a snapshot identifier or another reproducible version reference when the task requires auditability.

An answer citation should locate the actual support, not merely a document with a similar title. Keep chunk identity internal if it is unreadable to users, but provide a useful source title, section, date, and passage location. The provenance must remain available for review.

The simple character splitter is a baseline for understanding these relationships. Improve it only under a concrete corpus and question contract, then demonstrate whether the changes preserve evidence and improve the measures that matter.

Sources and reproducibility

Primary references: Azure AI Search document chunking and Text Split skill parameters. The complete example runs locally without dependencies or services. Its size values count characters; it is explicitly not a tokenizer, embedding benchmark, or complete retrieval application.