RAG Deep Dive: Retrieval That Actually Works

Bad chunking is the number-one reason RAG fails in production. Fixed token counts (512 tokens per chunk) split mid-sentence, mid-table, and mid-policy - producing chunks that are meaningless without surrounding context.

Chunk by semantic boundaries instead: sections, paragraphs, individual tickets, FAQ entries, or policy clauses. Each chunk should be self-contained enough to answer at least one specific question. Include metadata on every chunk: source document, date, section title, customer ID, product version. Metadata enables filtering ('only search policies updated after 2025-01-01') that dramatically improves precision.

For structured documents (contracts, manuals), use a hierarchical approach: store both section-level chunks for broad questions and paragraph-level chunks for specific details. When a user asks about 'Section 4.2 liability caps,' you want the exact paragraph, not the entire contract.

Vector-only search misses exact matches - SKUs, policy numbers, proper nouns, legal citations, and error codes. Keyword/BM25 search misses semantic similarity - 'refund policy' vs. 'money-back guarantee.' Production RAG needs both.

A practical hybrid pipeline: run vector search and keyword search in parallel, merge results with reciprocal rank fusion, then rerank the top 20 candidates with a cross-encoder or LLM reranker. This adds 100-300ms latency but improves recall significantly on real queries.

Tune the blend for your domain. E-commerce agents need strong keyword search (product IDs, SKUs). Legal agents need strong semantic search (conceptual similarity across different phrasings). Support agents need both (error codes are exact; troubleshooting steps are semantic). Test with 50+ real queries from your target users, not synthetic benchmarks.