Production RAG Pipelines: Fix Ingestion and Chunking First

Your RAG demo answered every question correctly. Three weeks after shipping, it started returning confident, wrong answers to the exact same questions on real documents. The model didn't get worse — the pipeline that feeds it did. Most teams debug the LLM first and the ingestion pipeline last, which is backwards: ingestion is where retrieval quality is actually decided.

Retrieval starts at the document, not the query

A production RAG pattern worth following treats ingestion as four distinct jobs: preserve structure, remove noise, extract metadata, and prepare the document for retrieval. Skip any one of them and the damage compounds downstream — no amount of prompt tuning recovers a document that was flattened into a blob of text before it reached the chunker. A contract with sections, tables, and headers is not the same input as a plain-text dump, and treating it that way is how a clear clause like "active invoices must be closed before upgrading" turns into a retrieval hit that looks relevant and answers the wrong question.

Chunking is where most systems quietly fail

Fixed-size chunking — cut every N tokens, move on — is the default in most tutorials, and it's the first thing to replace. It's fine for prototypes and raw plain text, but it breaks on contracts, tables, and code-heavy documents, because it cuts sentences and headers apart from the content they describe. Sliding-window chunking with overlap reduces the damage but doesn't fix the underlying design problem. The stronger pattern is structure-aware chunking: split on headings for Markdown, semantic tags for HTML, clauses for contracts — so each chunk carries its own context instead of arriving as a stripped fragment.

Parent-child chunking: search small, answer big

One production pattern stands out for balancing precision and context: index small child chunks for matching, but hand the LLM the larger parent chunk once a child is retrieved. Small chunks are easier to match precisely against a query; large chunks give the model enough surrounding context to generate a complete, non-hallucinated answer. Hierarchical chunking takes this further, keeping document, section, paragraph, and sentence levels available at once, so a broad query can pull a section while a citation can point to one sentence.

Metadata has to be born at ingestion, not bolted on later

Metadata — document type, section, version, language, last-updated time — is what lets a system answer "show me the latest policy" or "ignore deprecated versions" instead of drowning every query in undifferentiated semantic soup. Illustrative example: tagging a chunk with {"title": "Pricing Policy", "section": "Upgrading Plans", "version": "2.1"} at ingestion time is what later lets the retriever filter and cite it correctly — adding that metadata after the fact, once everything is already embedded as one long blob, is far more expensive to fix.

What to check before you blame the model

The common failure chain looks like this: text gets extracted too early, structure is lost, chunking splits the wrong boundaries, metadata comes in shallow or missing, and retrieval returns something "relevant" but incomplete — which the LLM then answers with full confidence. If your RAG system is hallucinating or missing obvious context, audit ingestion and chunking before you swap embedding models or upgrade the LLM. It's usually the cheaper fix, and it's usually the actual cause.