Engineering Notes

Chunking Methods

RAG Chunking Methods for Production (Engineers Edition)

If your RAG system underperforms, it’s rarely the model. It’s almost always the chunking.

Chunking determines recall, grounding quality, latency, and cost. Get it wrong and you get hallucinations, missed context, bloated indexes, and noisy retrieval.

Below is a practical breakdown of chunking methods used in production systems — when to use them, tradeoffs, and implementation notes.

1. Fixed-Size Chunking (Baseline)

Method
Split text into N-token windows (e.g., 512 tokens) with optional overlap (e.g., 50–100 tokens).

Why it works

• Simple

• Predictable embedding size

• Fast ingestion

• Works surprisingly well for unstructured prose

Where it fails

• Breaks semantic boundaries

• Splits tables/code mid-structure

• Context bleeding across overlaps

Implementation Notes

• Use token-aware splitting (not character-based).

• Tune overlap based on domain:

  • Legal/docs → higher overlap (75–150 tokens)

  • FAQs → minimal overlap (0–50 tokens)

• Monitor:

  • Retrieval hit rate

  • Context window waste (unused tokens sent to LLM)

When to use

• MVPs

• Homogeneous long-form content

• Low engineering bandwidth

2. Semantic Chunking (Embedding-Aware Splits)

Method
Split text at semantic boundaries using:

• Sentence embeddings + similarity threshold

• Sliding window clustering

• Topic shift detection

Instead of “every 500 tokens,” split when cosine similarity drops.

Why it works

• Preserves topical coherence

• Improves precision@k

• Reduces noisy retrieval

Tradeoffs

• Slower ingestion

• Harder to tune thresholds

• Risk of overly large chunks if topics are broad

Implementation Pattern

1. Sentence tokenize

2. Embed each sentence

3. Compute similarity between adjacent sentences

4. Break when similarity < threshold

5. Enforce min/max token limits

When to use

• Knowledge bases

• Long technical docs

• Multi-topic documents

3. Structure-Aware Chunking (HTML / Markdown / Docs)

Method
Split based on document structure:

• Headings

• Sections

• Bullet groups

• Table boundaries

• Code blocks

Why it works

• Aligns with how humans navigate content

• Maintains logical grouping

• Excellent for product docs & wikis

Example
Instead of:

[512 tokens arbitrary split]

You chunk as:

H2: Authentication
  - Description
  - Code example
  - Error cases

Engineering Notes

• Parse DOM for HTML

• Preserve header hierarchy in metadata

• Store path context:

  • doc > section > subsection

When to use

• Confluence

• Notion

• Developer docs

• API references

4. Table-Aware & Code-Aware Chunking

Naive chunking destroys:

• CSV tables

• JSON schemas

• SQL

• Source code

Best Practice

• Treat tables as atomic units

• Optionally generate:

  • A natural language summary

  • Column descriptions

• Embed both:

  • Raw table chunk

  • Structured summary chunk

For code:

• Chunk by function/class

• Store file path + symbol name in metadata

• Avoid splitting functions

Why this matters
Most enterprise RAG failures come from broken structured data ingestion.

5. Metadata-Enriched Chunking (Underused, High Impact)

Chunking isn’t just splitting text.

At ingestion time, you can attach:

• Source system

• Document type

• Owner

• Created/updated timestamps

• Section path

• Product area

• Access controls

Advanced pattern:
Generate semantic tags at ingestion using an LLM:

This chunk is about: billing, subscription cancellation, refunds

Then retrieval becomes:

• Hybrid search (vector + metadata filters)

• Scoped retrieval

• Top-k per source

This dramatically reduces hallucination risk.

6. Hierarchical Chunking (Multi-Resolution Retrieval)

Instead of one index:

Create:

• Level 1: Document summaries

• Level 2: Section-level chunks

• Level 3: Fine-grained chunks

Retrieval Flow:

1. Retrieve top documents

2. Narrow to top sections

3. Pull fine-grained chunks

Benefits:

• Better recall

• Lower context waste

• Scales to large corpora

This is essential beyond ~1M chunks.

7. Adaptive Chunking (Query-Aware Retrieval)

Emerging approach:

Instead of static chunk size:

• Retrieve broader chunks for exploratory queries

• Retrieve fine-grained chunks for specific factual queries

This can be implemented via:

• Query classification

• Dynamic top-k

• Multi-stage reranking

Chunking becomes part of retrieval orchestration, not just ingestion.

Key Tradeoffs

Production Failure Modes

Where chunking breaks in real systems:

• Connectors update schema → ingestion silently fails

• Document templates change → structure-aware parser breaks

• New doc types appear → chunking rules mismatch

• Table-heavy data embedded as plain text → unusable retrieval

• Over-chunking → index explosion + latency spike

• Under-chunking → hallucination due to context dilution

Chunking must be versioned and monitored.

Metrics You Should Track

At minimum:

• Retrieval hit rate

• % of grounded answers

• Avg tokens sent to LLM per query

• Context utilization ratio

• Chunk recall overlap (duplicate retrieval)

• Latency per stage

Chunking decisions directly impact all of these.

Practical Recommendation

If you’re building a serious RAG system:

• Start fixed-size + overlap.

• Move to structure-aware ASAP.

• Add metadata enrichment.

• Introduce hierarchical retrieval when corpus grows.

• Treat chunking as a first-class system component — not preprocessing glue.

In mature systems, chunking is not a static step. It’s part of the retrieval architecture.

Most engineers over-invest in models and under-invest in chunking.

The leverage is in the split.

About Twig

Ship Production RAG — Faster

Twig is the AI engineering platform for teams building RAG and agentic systems. We automate ingestion, smart chunking, indexing, and evals — so you can move from prototype to production up to 80% faster.

👉 Schedule a 30-min call with a Twig engineer and ship faster.