RAG systems retrieve the wrong documents because semantic similarity is not the same thing as relevance. The retriever returns whatever text sits closest to your query in vector space, which frequently means a long, keyword-dense, thoroughly worded document that happens to be four years out of date beats the crisp two-page memo that replaced it. The model then answers accurately from the wrong source, which is worse than answering badly.
Most teams try to fix this at the generation layer, tightening prompts and swapping models. That almost never works, because the model cannot reason about documents it was never handed. If the correct chunk was not in the top five results, no instruction anywhere in the system will conjure it back.
The Five Failure Modes Behind Bad Retrieval
Chunking is the first and most common. Split a document at a fixed 500 tokens and you cut a table in half, separate a condition from its exception, and strip the heading that gave a paragraph its meaning. That orphaned chunk still embeds fine and still gets retrieved, it just no longer says what the original said.
Embedding mismatch is the second. Your users ask questions in operational language (“can I expense a client dinner”) while your documents are written in policy language (“reimbursable business entertainment expenditure”). General-purpose embedding models handle that gap unevenly, and domain vocabulary makes it worse. Medical, legal, and engineering terminology all suffer here, and industry benchmarks generally show noticeable accuracy gaps between general and domain-adapted embeddings on specialist corpora.
Then there is duplicate and near-duplicate content, where nine versions of the same policy compete and the retriever has no basis for preferring the current one. Missing metadata compounds it, since without effective date, owner, and status fields there is nothing to filter or rank on. And finally the queries themselves, which are often short, ambiguous, or referential (“what about the other one”) in ways that no single embedding can resolve.
Fixing Chunking Before You Touch Anything Else
Semantic chunking beats fixed-length splitting in almost every case. Split at structural boundaries, headings, sections, table edges, so each chunk is a complete thought rather than an arbitrary window. Most enterprise content works well somewhere in the 300 to 800 token range, but the right number depends on document type rather than a global setting.
Overlap matters more than people expect. Ten to twenty percent overlap between adjacent chunks preserves the context that would otherwise be severed at the boundary, and it costs almost nothing in storage terms. Without it, the sentence that qualified the previous paragraph disappears from both chunks.
Context enrichment is the cheapest high-impact change available. Prepend the document title, section heading, and effective date to every chunk before embedding. Now the retriever knows a paragraph about withdrawal limits came from the 2026 account terms rather than the 2021 ones, and so does the model when it writes the answer.
Different content needs different treatment. Tables should be extracted as structured data with headers repeated per row, not flattened into prose. Slide decks need the speaker notes joined to the slide. Contracts benefit from clause-level chunking because clauses are the natural retrieval unit. Handling all of these with one pipeline is the single most common reason enterprise RAG underperforms its pilot.
Ranking, Filtering, and Hybrid Search
Vector search alone misses exact matches. Product codes, error numbers, policy IDs, and proper nouns are exactly the things users search for and exactly the things embeddings handle poorly. Hybrid search, combining dense vector similarity with BM25 keyword matching, usually delivers a substantial retrieval improvement for very little engineering effort, and it is the first thing to add when precision is poor.
Reranking is the next layer. Retrieve twenty candidates cheaply, then run a cross-encoder over them to reorder by actual relevance to the query. It adds latency (typically a few hundred milliseconds) and it consistently lifts the quality of what reaches the model. Most teams working through how to improve RAG accuracy for enterprise AI find that reranking plus metadata filtering delivers more than any embedding model upgrade.
Metadata filtering is where recency finally becomes controllable. Filter by status before ranking, so superseded documents never enter the candidate set at all. Filter by department, region, or product line when the query context supports it. This is not a retrieval trick, it is content governance made queryable, and it requires the metadata to exist in the first place.
Query rewriting closes the loop on ambiguous input. Expand abbreviations, resolve pronouns against conversation history, and generate two or three query variants to search in parallel. Short queries improve dramatically. Vague ones improve enough to matter.
How Requirements Change by Industry and Content Type
Legal and compliance work needs precision above everything, since a plausible but wrong clause is worse than no answer. These deployments run aggressive filtering, clause-level chunking, and usually surface the source passage alongside the answer so a human can verify in seconds.
Technical support and engineering corpora face the format problem hardest. Manuals in PDF, diagrams, spec tables, version-specific procedures across a dozen product releases. Version metadata is not optional here, because the correct answer for firmware 4.2 is wrong for 4.4 and both documents exist.
Customer-facing knowledge bases are relatively clean but change constantly, so the governance rhythm matters more than the pipeline sophistication. Financial services carry the audit requirement, meaning every retrieval needs logging with enough detail to reconstruct why a given answer was produced six months later. Smaller corpora under a few thousand documents often perform well with modest tuning, which is exactly why pilots on curated content mislead so badly.
Measuring Whether Any of This Worked
Build an evaluation set before you change anything. A hundred real questions from actual users, each with the correct source document identified by someone who knows the content. Without this you are guessing, and every subsequent change becomes an argument about vibes.
Measure retrieval separately from generation. Recall at five tells you whether the right chunk made it into the candidate set. Precision tells you how much noise came with it. If recall is poor, no generation improvement will help, and knowing which of the two is broken saves months of work aimed at the wrong layer.
Then instrument production, because the evaluation set ages. Track queries returning nothing useful, escalation rates, thumbs-down feedback, and the specific questions that consistently fail. Those failures cluster, and the clusters point directly at content gaps rather than model problems.
The thing worth planning for now is that retrieval quality degrades quietly as content grows, so a system tuned to 92 percent recall on fifty thousand documents will not hold that at two hundred thousand without re-tuning. Put a quarterly evaluation run on the calendar with a named owner, and treat any drop below your threshold as a content operations task rather than an AI one. That is where the fix almost always lives.