Bad chunking is the number-one reason RAG fails in production. Fixed token counts (512 tokens per chunk) split mid-sentence, mid-table, and mid-policy — producing chunks that are meaningless without surrounding context.

Chunk by semantic boundaries instead: sections, paragraphs, individual tickets, FAQ entries, or policy clauses. Each chunk should be self-contained enough to answer at least one specific question. Include metadata on every chunk: source document, date, section title, customer ID, product version. Metadata enables filtering ('only search policies updated after 2025-01-01') that dramatically improves precision.

For structured documents (contracts, manuals), use a hierarchical approach: store both section-level chunks for broad questions and paragraph-level chunks for specific details. When a user asks about 'Section 4.2 liability caps,' you want the exact paragraph, not the entire contract.

Vector-only search misses exact matches — SKUs, policy numbers, proper nouns, legal citations, and error codes. Keyword/BM25 search misses semantic similarity — 'refund policy' vs. 'money-back guarantee.' Production RAG needs both.

A practical hybrid pipeline: run vector search and keyword search in parallel, merge results with reciprocal rank fusion, then rerank the top 20 candidates with a cross-encoder or LLM reranker. This adds 100-300ms latency but improves recall significantly on real queries.

Tune the blend for your domain. E-commerce agents need strong keyword search (product IDs, SKUs). Legal agents need strong semantic search (conceptual similarity across different phrasings). Support agents need both (error codes are exact; troubleshooting steps are semantic). Test with 50+ real queries from your target users, not synthetic benchmarks.

RAG without evaluation is guesswork. Build a test set of 50+ real questions with expected source documents and acceptable answer ranges. Run this test set weekly as you add content, change chunking, or swap embedding models.

Track three metrics: recall@k (did the right source document appear in the top 5 results?), answer faithfulness (did the generated answer stick to retrieved content, or did it hallucinate?), and answer relevance (did it actually address the question?). Recall@k below 80% means your retrieval is broken. Faithfulness below 90% means your prompt needs tighter constraints.

Add every production failure to your test set. When a customer asks a question and gets a wrong answer, that question becomes a permanent regression test. Over 3-6 months, this dataset becomes your most valuable asset — it captures exactly where your knowledge base and retrieval pipeline fall short.