Retrieval Is the Hard Part — Building RAG That Actually Works

The demo is always the same. Someone loads a pile of documents into a vector database, wires it to a language model, asks a question, and gets a fluent, correct-sounding answer. Everyone nods. The project gets funded on the strength of that afternoon.

Then it meets real questions and real documents, and the cracks show. It answers confidently from the wrong section. It misses the one paragraph that actually mattered. It invents a policy that sounds exactly like something the company would write but doesn't exist. The fluent demo has become a liability nobody quite trusts.

The difference between those two systems is almost never the model. It's retrieval — the part nobody demos, because retrieval isn't impressive to watch. It's just correct or it isn't.

RAG is a search problem wearing a chat interface

Retrieval-augmented generation has a simple shape: when a question comes in, find the most relevant material from your own content, and hand it to the model as context so it answers from that rather than from its training. The generation step gets all the attention. The retrieval step decides whether it works.

   question ─▶ ┌───────────┐   top passages   ┌───────────┐ ─▶ grounded answer
              │ retrieval │ ───────────────▶  │    LLM    │     (+ citations)
              └───────────┘                   └───────────┘
                    ▲
             your indexed content

Here's the uncomfortable consequence of that diagram: the model can only be as good as what retrieval feeds it. If the passage that answers the question never makes it into the context, there is no prompt clever enough and no model expensive enough to recover. It will answer from what it was given, confidently, and be wrong.

So the engineering effort belongs where the failures are. In my experience, when a RAG system gives a bad answer, retrieval is at fault far more often than generation.

If the right chunk didn't make it into the context window, nothing downstream can fix the answer. Retrieval is not a preprocessing step you get for free — it's the product.

Chunking: the decision made carelessly that ruins everything downstream

Before anything can be retrieved it has to be split into pieces and embedded. This step feels mechanical, so people do it mechanically — "split every 500 tokens" — and quietly poison the whole system.

Fixed-size splitting cuts through the middle of ideas. A sentence that begins "This does not apply when…" gets severed from the condition that follows, and now your database contains a chunk that says the opposite of the truth. Retrieval faithfully finds it. The model faithfully repeats it.

Chunk on the document's own structure instead — headings, sections, paragraphs — so each piece is a coherent unit of meaning:

// Split on semantic boundaries, keep a little overlap so context that
// straddles a boundary isn't lost, and keep the heading with its body.
function chunkDocument(doc: ParsedDoc): Chunk[] {
  return doc.sections.flatMap((section) =>
    splitByParagraph(section.body, { maxTokens: 512, overlapTokens: 64 }).map(
      (text) => ({
        text,
        // Carry structural context INTO the chunk so it's self-describing.
        heading: section.heading,
        source: doc.title,
        url: doc.url,
      }),
    ),
  );
}

Two details do most of the work. Overlap — carrying a little of the previous chunk into the next — stops an idea that spans a boundary from being lost by both sides. And carrying the heading into the chunk means a paragraph that only makes sense under "Refund exceptions" still knows it's about refund exceptions when it's retrieved in isolation.

Pure vector search is not enough

The default RAG design embeds the question, embeds every chunk, and returns the chunks whose vectors are nearest. Semantic search is genuinely powerful — it finds "how do I get my money back" when the document says "refund policy." But on its own it has a blind spot that matters enormously in practice: it's bad at exact things.

Order number INV-4471. A product SKU. A person's surname. An error code. Semantically these are near-meaningless — the embedding for INV-4471 isn't usefully close to the embedding for INV-4472 — so vector search shrugs at exactly the queries where the user knows precisely what they want.

The fix is hybrid search: run classic keyword search (which is superb at exact matches) alongside vector search (which is superb at meaning), and combine the results.

A common and effective way to merge the two rankings is Reciprocal Rank Fusion — it rewards documents that rank highly in either list without needing the two scores to be on the same scale:

// Reciprocal Rank Fusion: blends keyword and vector rankings robustly.
function fuse(keyword: string[], vector: string[], k = 60): string[] {
  const score = new Map<string, number>();
  for (const list of [keyword, vector]) {
    list.forEach((id, rank) => {
      score.set(id, (score.get(id) ?? 0) + 1 / (k + rank));
    });
  }
  return [...score.entries()].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

The first time I add hybrid search to a struggling RAG system, the improvement is usually dramatic — because the failures were disproportionately the exact-match queries that pure vector search was quietly dropping.

Reranking: cheap retrieval finds candidates, expensive scoring picks winners

Vector and keyword search are built for speed over millions of chunks, and they buy that speed with a coarse notion of relevance. So a good pattern is two stages: retrieve broadly and cheaply to get maybe the top 30 candidates, then rerank those few with a slower, far more accurate model that reads the question and each candidate together and scores true relevance.

  question ─▶ hybrid search ─▶ top 30 candidates ─▶ reranker ─▶ top 5 ─▶ LLM
              (fast, coarse)                        (slow, precise)

Because the reranker only ever looks at a handful of candidates, you can afford its cost — and it reliably pushes the genuinely-best passages to the top, where they'll fit in the context budget you actually send to the model. Retrieving five chunks straight from vector search versus reranking thirty down to the best five is often the difference between "usually right" and "reliably right."

Grounding: make the model answer from the text, or admit it can't

Now the model has good context. The last job is to make sure it actually uses it rather than falling back on its training — and that it declines gracefully when the answer genuinely isn't there.

Most of this is prompt discipline, and it's worth being explicit:

const systemPrompt = `
Answer ONLY using the provided context. Follow these rules exactly:
- If the context does not contain the answer, say you don't know. Do not guess.
- Cite the source of each claim using its [number].
- Do not use knowledge from outside the context, even if you're confident.

Context:
${retrieved.map((c, i) => `[${i + 1}] (${c.source}) ${c.text}`).join("\n\n")}
`;

The single most valuable behaviour you can engineer is a confident "I don't know." A RAG assistant that admits the limits of its content is trustworthy. One that fills gaps with plausible invention is worse than useless — it teaches people they can't rely on any of its answers, including the correct ones. Citations reinforce this: when every claim points back to a source passage a human can open and check, wrong answers get caught, and right answers get trusted.

You can't improve what you don't measure

Everything above is testable, and if you don't test it you're flying blind. Build a set of real questions with known-correct answers and known-correct source passages, and measure two things separately:

• Retrieval quality — did the right passage make it into the retrieved set at all? (If not, no generation fix will help.)
• Answer quality — given good context, was the final answer correct and grounded?

Separating these tells you where a failure lives. A wrong answer with the right passage retrieved is a generation problem. A wrong answer with the right passage missing is a retrieval problem — and now you know which knob to turn. Without this split, teams thrash: they swap models to fix what was a chunking bug, and rewrite prompts to fix what was a search bug.

The unglamorous truth

RAG is sold as an AI problem. In practice it's mostly a search problem with a language model on the end. The parts that decide whether it works — how you chunk, whether you search hybrid, whether you rerank, whether you ground and cite and measure — are classic information-retrieval engineering, not prompt magic.

That's good news, actually. It means the outcome is under your control. A retrieval-augmented system that's built with care is one of the most genuinely useful things you can put in front of a business: it answers from your knowledge, shows its work, and knows what it doesn't know. That's not a demo. That's infrastructure — and it's the part worth paying for.