Retrieval-augmented generation grounds a language model in your documents: find relevant passages, put them in the prompt, generate an answer from them. It is the standard answer to "the model does not know our policies", and it works.
The part that gets attention
Model selection, prompt structure, output formatting. These matter least. Swapping models moves answer quality by a few percent.
The part that decides the outcome
Retrieval. If the right passage is not in the top results, no model recovers it. And retrieval quality is mostly determined by things that look unglamorous: how documents are chunked, whether headings survive extraction, whether the policy from 2019 is clearly marked as superseded, whether the same term means two things in two departments.
Where it fails predictably
- Questions requiring synthesis across many documents. Retrieval returns passages, not conclusions.
- Anything needing arithmetic over records. Use a query, not a model.
- Corpora with contradictions. The system will confidently pick one, and it may not be the current one.
Measure retrieval separately
Evaluate whether the correct passage appears in the top results, independently of what the model then says. Teams that only measure end-to-end answer quality cannot tell which half to fix.