Building a Production RAG System: Lessons from Real Users

Why RAG is harder than the demos suggest

Every RAG tutorial ends the same way: embed some documents, stuff the top-k chunks into a prompt, and marvel at the answer. Production is a different sport. When real students started asking real questions, our first version confidently answered from the wrong chapter about a third of the time.

This post covers the three changes that took us from demo-quality to production-quality.

1. Chunking is a product decision, not a preprocessing step

We started with fixed 512-token chunks. The problem: educational content has structure — definitions, worked examples, exercises — and naive chunking cuts straight through it.

What worked:

• Structure-aware splitting on headings and semantic boundaries
• Parent-document retrieval: embed small chunks, but hand the LLM the surrounding section
• Metadata filters (subject, grade level) applied before vector search

2. Hybrid search beats pure vectors

Pure embedding search failed on exact terms — formula names, code identifiers, dates. Adding BM25 keyword search and merging with reciprocal rank fusion fixed most of it:

const results = rrf([vectorResults, bm25Results], { k: 60 });

3. You cannot improve what you do not evaluate

We built a small eval set of 200 real questions with graded answers. Every retrieval change runs against it in CI. Retrieval precision went from 61% to 88% over six weeks — and we could prove it.

Takeaways

1. Invest in chunking before you invest in a fancier model.
2. Hybrid retrieval is table stakes.
3. An eval set of even 100 questions changes how you ship.