System Design Cases
Chunking Strategies for RAG
Chunking strategies for RAG: how to split documents into chunks for embedding and retrieval. Demonstrates three strategies side-by-side — fixed-size with overlap (cheap but loses coreferences at chunk boundary), recursive markdown-aware (respects document structure, keeps headers in metadata), and late chunking (Jina 2024 — embed whole doc with long-context model first, then split the token embeddings, preserving global context for pronouns and cross-references). ADR — Choose chunker by content type: - Markdown / HTML / docs: recursive structure-aware splitter (RecursiveCharacterTextSplitter / MarkdownHeaderTextSplitter). Header path -> metadata for filtering and citations. - Code: AST-based (tree-sitter) — split on functions/classes, never mid-symbol; embed signature + docstring + body together. - PDF / DOCX / slides: layout-aware (Unstructured.io, Marker) — preserve tables and section breaks. - Long narrative with many coreferences: late chunking — only viable if you have a long-context embedding model (Jina, Voyage); +20% recall on anaphora. - Default for unknown / mixed text: recursive 500/50 with sentence-aware fallback. - Avoid: one-size-fits-all 500-token splitter on heterogeneous corpus, naive split with zero overlap, chunk_size > embedding model max (silent truncation). - Always: enrich every chunk with metadata payload (source_url, section_path, doc_type, date, permissions) for pre-ANN filtering and citations. Trade-off summary: too small loses surrounding context (LLM can't answer); too big dilutes the embedding signal (ANN recall drops). Sweet spot for general RAG is 300-500 tokens with 10-20% overlap, plus parent-child for technical docs that need both precise retrieval and wide context.
Chunking: versioned evidence boundaries measured end to end
Chunking decides which evidence units are indexed and later placed into context. Boundary, overlap, metadata, parser, and embedding choices interact; no token size or strategy wins for every corpus and task.
Mental model
- RAG retrieves external passages and conditions generation on them; evidence-unit design affects what can be retrieved. Evaluate chunks as part of the full retrieval and answer pipeline.
- Long-context model performance can vary with the position of relevant information. Do not assume more or longer chunks guarantee evidence use.
- Late chunking derives chunk representations after long-context encoding and reports empirical results on selected tasks. Present it as a candidate to benchmark, not a universal recall improvement or coreference guarantee.
- Unstructured documents chunking strategies preserve different element and size semantics. Version parser and chunker behavior instead of treating a token count as a standard.
Guarantees and boundaries
- Every chunk carries source identity, offsets, processing version, and access scope.
- Overlap can reduce some boundary losses but cannot guarantee complete context.
- Canonical source retention enables deterministic reprocessing; model outputs remain non-deterministic.
Diagram scenarios
Structure-aware indexing. Chunks retain source offsets, headings, ACLs, and a reproducible processing version.
Boundary and context failure. A valid-looking chunk can omit the antecedent, table header, code definition, or policy qualifier needed to answer.
Shadow test a new strategy. Late, semantic, recursive, or fixed chunking remains an empirical choice tied to the model and corpus.
Untrusted content and deletion. Documents can contain prompt injection or sensitive text; chunking does not make them trustworthy.
Architecture decision
Preserve document structure and stable source offsets, version the parser and chunker, and compare strategies with retrieval plus answer-grounding evaluations. Treat retrieved text as untrusted data, enforce access controls per chunk, and keep a reprocessing path from the canonical source.
Failure modes
- Parser changes can silently alter chunk IDs and duplicate an index.
- One chunk can cross access-control boundaries or contain malicious instructions.
- A retrieval metric can improve while answer attribution or cost worsens.
Operational checklist
- Build evaluation questions that exercise tables, code, headings, coreference, and long dependencies.
- Compare chunk policies on the same corpus and model versions.
- Apply ACL and deletion at source and derived-object layers.
- Store citations to exact source regions, not only chunk text.