Chunking: versioned evidence boundaries measured end to end

Chunking decides which evidence units are indexed and later placed into context. Boundary, overlap, metadata, parser, and embedding choices interact; no token size or strategy wins for every corpus and task.

Mental model

• RAG retrieves external passages and conditions generation on them; evidence-unit design affects what can be retrieved. Evaluate chunks as part of the full retrieval and answer pipeline.
• Long-context model performance can vary with the position of relevant information. Do not assume more or longer chunks guarantee evidence use.
• Late chunking derives chunk representations after long-context encoding and reports empirical results on selected tasks. Present it as a candidate to benchmark, not a universal recall improvement or coreference guarantee.
• Unstructured documents chunking strategies preserve different element and size semantics. Version parser and chunker behavior instead of treating a token count as a standard.

Guarantees and boundaries

• Every chunk carries source identity, offsets, processing version, and access scope.
• Overlap can reduce some boundary losses but cannot guarantee complete context.
• Canonical source retention enables deterministic reprocessing; model outputs remain non-deterministic.

Diagram scenarios

Structure-aware indexing. Chunks retain source offsets, headings, ACLs, and a reproducible processing version.

Boundary and context failure. A valid-looking chunk can omit the antecedent, table header, code definition, or policy qualifier needed to answer.

Shadow test a new strategy. Late, semantic, recursive, or fixed chunking remains an empirical choice tied to the model and corpus.

Untrusted content and deletion. Documents can contain prompt injection or sensitive text; chunking does not make them trustworthy.

Architecture decision

Preserve document structure and stable source offsets, version the parser and chunker, and compare strategies with retrieval plus answer-grounding evaluations. Treat retrieved text as untrusted data, enforce access controls per chunk, and keep a reprocessing path from the canonical source.

Failure modes

• Parser changes can silently alter chunk IDs and duplicate an index.
• One chunk can cross access-control boundaries or contain malicious instructions.
• A retrieval metric can improve while answer attribution or cost worsens.

Operational checklist

• Build evaluation questions that exercise tables, code, headings, coreference, and long dependencies.
• Compare chunk policies on the same corpus and model versions.
• Apply ACL and deletion at source and derived-object layers.
• Store citations to exact source regions, not only chunk text.

Related concept

Primary sources

1. https://arxiv.org/abs/2005.11401
2. https://arxiv.org/abs/2307.03172
3. https://arxiv.org/abs/2409.04701
4. https://docs.unstructured.io/open-source/core-functionality/chunking