7 Best Chunking Strategies for RAG Pipelines in 2026

Most RAG failures trace back to bad chunking. A Vectara study published at NAACL 2025 tested 25 chunking configurations across 48 embedding models and found that chunking configuration had as much or more influence on retrieval quality as the choice of embedding model. Get chunking wrong by one bracket, and context precision drops 15-30%.

The reason is straightforward. Embedding models compress a chunk into a single vector. If the chunk mixes two unrelated topics, the vector represents neither well. If the chunk cuts a paragraph mid-sentence, the embedding loses the complete thought. Every downstream component, from vector search to reranking to generation, inherits whatever the chunking step produces.

Chunk sizes between 256 and 512 tokens generally outperform larger chunks for question-answering retrieval. But "generally" hides important nuance. Factoid queries (Who founded OpenAI?) perform best at 256-512 tokens. Multi-hop analytical queries (Compare the funding strategies of three AI labs) benefit from 512-1,024 tokens. The FloTorch February 2026 benchmark tested seven strategies across 50 real academic papers and found a 15-percentage-point accuracy gap between the best and worst approaches, all using the same embedding model and retrieval pipeline.

Before picking a strategy, identify your dominant query pattern and document type. The decision tree below maps both to a starting configuration.

Decision Flowchart

1. Are your documents structured with clear headings and sections? If yes, start with document-structure chunking (#4 below).
2. Are your queries mostly fact-based lookups? Start with recursive splitting at 256-512 tokens (#1).
3. Are your queries analytical, requiring multi-paragraph context? Start with recursive splitting at 512-1,024 tokens (#1) or page-level chunking (#5).
4. Do you have a heterogeneous corpus with mixed document types? Try semantic chunking (#2) with a minimum chunk floor of 200 tokens.
5. Is your corpus small, high-value, and worth the compute cost? Consider LLM-driven chunking (#6) or contextual retrieval (#7).

Recursive character splitting is the benchmark-validated default for most RAG pipelines. It works by trying a sequence of separators in priority order: double newlines, single newlines, sentences, then spaces. Each separator preserves progressively less document structure, so the algorithm uses the most meaningful break it can find within the target chunk size.

In the FloTorch February 2026 benchmark, recursive splitting at 512 tokens scored 69% end-to-end answer accuracy across 50 academic papers, outperforming every more expensive alternative. Chroma's research confirmed 85-89% retrieval recall at 400-512 tokens.

Recommended configuration:

• Chunk size: 512 tokens (start here, tune later)
• Overlap: 50-100 tokens (10-20% of chunk size)
• Separators: `["

", " ", ". ", " ", ""]`

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["

", "
", ". ", " ", ""]
)
chunks = splitter.split_text(document_text)

When to use it: This is your starting point. If you have no prior data about your query patterns or document types, begin here and benchmark before trying anything more complex.

Limitations: It ignores semantic boundaries entirely. A paragraph about pricing might get merged with a paragraph about security features if they happen to be adjacent and fit within the token budget.

Semantic chunking splits text based on meaning rather than character count. It embeds each sentence, measures cosine similarity between consecutive sentences, and creates a new chunk boundary when similarity drops below a threshold. The result is chunks that represent coherent topics.

The performance story is more complex than most guides suggest. Chroma's testing showed semantic chunking reaching 91.9% retrieval recall, beating recursive splitting's 85-89%. But FloTorch's end-to-end benchmark told a different story: semantic chunking scored only 54% answer accuracy compared to recursive splitting's 69%.

The gap comes from fragment size. Semantic chunking without a minimum floor produces chunks averaging 43 tokens in FloTorch's testing. These tiny fragments retrieve well (the vector matches are precise) but give the LLM too little context to generate a good answer.

The fix: Set a minimum chunk size of 200 tokens. When semantic boundaries produce fragments below this floor, merge adjacent fragments until the minimum is met.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    embeddings=OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95
)
chunks = chunker.split_text(document_text)

When to use it: Heterogeneous corpora where documents mix topics, formats, and writing styles. The extra embedding cost pays off when fixed-size splitting would regularly cut across topic boundaries.

Limitations: Requires embedding every sentence during ingestion, which adds cost and latency. Without a minimum chunk floor, it produces fragments too small for generation.

Skip the chunking pipeline and start searching your documents

Fast.io auto-indexes uploaded files for RAG with citations. 50 GB free storage, 5,000 AI credits per month, no credit card required.

Sign Up Free

Sentence splitting is the simplest semantic-aware approach. It uses natural language boundaries (periods, question marks, newlines) instead of arbitrary character counts. Paragraph splitting groups sentences into their original paragraph units.

This approach sits between recursive splitting and semantic chunking in both complexity and performance. It respects basic document structure without the embedding cost of semantic analysis.

Sentence splitting works well for:

• FAQ databases where each question-answer pair is self-contained
• Chat logs and conversational data
• Documents with short, independent paragraphs

Paragraph splitting works well for:

• Blog posts and articles with clear paragraph structure
• Legal documents with numbered clauses
• Technical documentation with distinct sections

The main risk is variance in chunk size. Some paragraphs are 20 tokens, others are 500. This inconsistency can cause problems during retrieval because embedding models behave differently at extreme lengths. If you use paragraph splitting, add a maximum size cap and split oversized paragraphs with recursive character splitting as a fallback.

Practical tip: Combine paragraph splitting with overlap by repeating the last sentence of each chunk at the start of the next. This preserves cross-paragraph references without the computational overhead of semantic analysis.

Document-structure chunking uses the inherent organization of a file, such as headings, sections, tables, and code blocks, to define chunk boundaries. Instead of treating a document as flat text, it parses the format first and chunks along structural lines.

This approach dominates for any corpus with reliable formatting: Markdown files, HTML pages, legal filings, technical manuals, API documentation. NVIDIA's 2024 benchmark found page-level chunking (a simpler version of structure-aware splitting) achieved 0.648 accuracy with the lowest standard deviation across financial documents, meaning it performed consistently regardless of document content.

Implementation varies by format:

• Markdown/HTML: Split on heading levels (h1, h2, h3), keeping each section with its heading as context
• PDFs: Use layout detection to identify sections, tables, and figures before chunking
• Code: Split on function or class boundaries, keeping docstrings with their functions

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers = [
    ("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),
]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
chunks = splitter.split_text(markdown_text)

When to use it: Structured documents where the formatting carries semantic meaning. Technical docs, legal contracts, research papers with clear section hierarchies.

Limitations: Requires format-specific parsers. Unstructured text (plain .txt files, raw OCR output) has no structural signals to exploit. Also, some sections may far exceed optimal chunk size and still need recursive splitting within each structural unit.

Late chunking flips the standard pipeline. Instead of chunking first and embedding each chunk independently, it embeds the entire document through a transformer, then segments the resulting token-level embeddings into chunks afterward. Each chunk's vector carries context from the full document, not just its local window.

Redis's April 2026 benchmarks showed late chunking producing roughly 3% average relative improvement over naive chunking on long-document retrieval across four BeIR datasets. That sounds modest, but the improvement concentrates on exactly the queries where standard chunking fails: ones that require understanding how a passage relates to the broader document.

Late chunking works particularly well for documents approaching 8,000 tokens, where cross-references between sections are common (technical specs, research papers, policy documents).

Requirements:

• An embedding model with a context window large enough for your full documents (Jina's embeddings-v3 supports up to 8,192 tokens)
• Higher memory and compute at ingestion time
• No benefit for short documents where the context window already covers the full text

When to use it: Long documents with heavy cross-references where passages refer to definitions, acronyms, or context established elsewhere in the document. If your documents are under 1,000 tokens, the overhead is not worth it.

Limitations: Documents exceeding the embedding model's context window cannot use this approach. Ingestion cost is higher because the full document passes through the transformer before splitting.

LLM-driven chunking uses a language model to decide where to split. The model reads the document and identifies natural boundaries based on topic shifts, argument structure, and semantic coherence. Agentic chunking extends this by having an LLM decompose content into atomic propositions, each representing a single fact.

Redis's Pseudo-Instruction Chunking (PIC) benchmark showed Hits@5 of 58.4, compared to 54.5 for fixed-size and 56.0 for semantic chunking. That is a meaningful improvement for high-stakes retrieval where every percentage point matters.

The cost tradeoff is steep. Every document requires one or more LLM calls during ingestion. For a corpus of 10,000 documents, that is 10,000+ API calls before a single query runs. Redis's benchmarks also showed adding a reranker (often paired with LLM-driven chunks) increased latency by 9.2x, from 0.22 seconds to 2.02 seconds per query.

When to use it: Small, high-value corpora where retrieval accuracy directly impacts business outcomes. Legal discovery, medical research, financial analysis. The per-document cost is justified when wrong answers are expensive.

When to skip it: Large corpora, real-time applications, or any pipeline where ingestion speed matters. Recursive splitting at 512 tokens with a reranker often matches LLM-driven chunking at a fraction of the cost.

Hybrid Approach

A practical middle ground: use recursive splitting for initial chunking, then run a lightweight classifier to flag chunks that span multiple topics. Only send those flagged chunks through an LLM for re-splitting. This targets the expensive operation where it adds the most value.

Contextual retrieval prepends a short summary to each chunk before embedding. An LLM reads the full document and generates a 1-2 sentence context header for every chunk, explaining where it fits in the larger document. The chunk "Revenue grew 12% in Q3" becomes "This passage is from the Q3 2025 earnings report of Acme Corp, discussing quarterly financial performance. Revenue grew 12% in Q3."

This approach attacks the core weakness of all chunking strategies: isolated chunks lose document-level context. A chunk about "the third requirement" means nothing without knowing what document and section it came from.

Redis's April 2026 analysis noted that contextual retrieval reduces retrieval failures when combined with hybrid search and reranking, though it requires an LLM call per chunk during ingestion, similar to LLM-driven chunking.

When to use it: Pair it with recursive or structure-based chunking on corpora where chunks frequently reference information from other parts of the document. It works well as an augmentation layer on top of simpler strategies rather than a standalone approach.

Limitations: Same cost profile as LLM-driven chunking. Every chunk requires an LLM call during ingestion. On a 10,000-document corpus with 20 chunks per document, that is 200,000 API calls. Best reserved for high-value collections where the retrieval accuracy gain justifies the ingestion cost.

Most of this guide assumes you are building and tuning a custom RAG pipeline. But for teams that need document Q&A without managing embeddings, chunk sizes, and vector databases, workspace-level auto-indexing removes the chunking decision entirely.

Fast.io's Intelligence Mode auto-indexes files when you upload them to a workspace. Documents, PDFs, spreadsheets, and presentations are chunked, embedded, and indexed automatically. You ask questions through the workspace chat, and the system returns answers with citations pointing to specific source passages.

This matters for the chunking discussion because it represents a different tradeoff: you give up control over chunk size and strategy in exchange for zero pipeline maintenance. For teams running AI agents that need persistent, searchable file storage, this is often the right call. The agent uploads files via the Fast.io MCP server, enables Intelligence on the workspace, and immediately has RAG capabilities without writing a single line of chunking code.

When custom chunking still wins: If your retrieval accuracy requirements demand tuning chunk sizes per document type, or you need to benchmark strategies against your specific query patterns, build the pipeline yourself. The strategies in this guide give you that control.

When auto-indexing wins: If your team's bottleneck is not retrieval precision but getting documents searchable at all. Fast.io's free agent plan includes 50 GB storage and 5,000 monthly AI credits with no credit card required, so testing it costs nothing.

For structured data extraction from documents, Metadata Views let you define custom fields and extract typed data (dates, amounts, names) from PDFs and images without building a separate parsing pipeline.