Why RAG fails in production and how to fix it

A RAG feature usually breaks in front of a user the first time someone asks where an answer came from. Which document justified it, which version of that document was current when the answer was given, and why did retrieval pick that passage over the one that actually contained the answer? A demo built from a chunker, an embedding call and a prompt that says "answer from the context" can't answer any of those, and that's the same reason it invents a contract clause or quotes a policy revision that was replaced three months ago.

The model is rarely the root cause. When an answer is wrong, the right passage was usually never retrieved, or it was retrieved next to an outdated copy, or the prompt let the model fill a gap the evidence didn't cover. So the fixes sit around the model: keep provenance in the data model, retrieve with more than one signal, make the model bind every claim to evidence you can check, and measure retrieval on its own before anyone tunes a prompt.

Everything below runs on Postgres with pgvector. Whether a separate vector database would serve you better is its own question, covered in the post on pgvector 0.8 versus a dedicated vector database.

Provenance has to be in the tables, because you can't reconstruct it from logs later. Documents get explicit versions, chunks point at a document version and carry their position in it, and every vector records the model that produced it:

 1create extension if not exists vector;
 2create extension if not exists unaccent;
 3
 4create text search configuration simple_unaccent (copy = simple);
 5alter text search configuration simple_unaccent
 6  alter mapping for hword, hword_part, word with unaccent, simple;
 7
 8create table documents (
 9  id bigint generated always as identity primary key,
10  tenant_id bigint not null,
11  source_ref text not null,
12  version int not null,
13  checksum bytea not null,
14  superseded_at timestamptz,
15  unique (tenant_id, source_ref, version)
16);
17
18create table chunks (
19  id bigint generated always as identity primary key,
20  document_id bigint not null references documents (id),
21  tenant_id bigint not null,
22  is_current boolean not null default true,
23  page int,
24  char_start int not null,
25  char_end int not null,
26  content text not null,
27  embedding_model text not null,
28  embedding vector(384) not null,
29  tsv tsvector generated always as (to_tsvector('simple_unaccent', content)) stored
30);
31
32create index chunks_tsv on chunks using gin (tsv);
33create index chunks_embedding on chunks using hnsw (embedding vector_cosine_ops);
34create index chunks_tenant on chunks (tenant_id) where is_current;

A new version of a document gets new chunk rows, and the old ones are marked is_current = false instead of being updated in place. That's what lets you answer "why did the answer change between April 3 and April 29": the retrieval log points at chunk IDs, the chunk IDs point at a document version, and nothing they point at has been rewritten. The page and character offsets let a review screen mark the exact passage, and embedding_model stops a half-finished re-embed from silently mixing two models in one index.

The text search configuration is worth a moment if your corpus isn't English. The english configuration that most examples use stems English words and drops English stop words, which mangles Czech, German or mixed content. Postgres ships snowball configurations for about thirty languages, and Czech isn't among them, so for Czech or mixed corpora a simple configuration with unaccent is a reasonable default: it lowercases, strips diacritics so "záruka" matches "zaruka", and doesn't stem. It also won't match "záruční" to "záruka", and if you need that, you'll need a Czech ispell or Hunspell dictionary on top. Wrapping unaccent into a text search configuration, as above, is what makes it usable in a generated column, because to_tsvector with an explicit configuration is immutable while calling unaccent() directly isn't.

Invoice numbers and other identifiers tokenize in ways you should check before relying on them. With this configuration, INV-000207 becomes the tokens inv and -000207, and websearch_to_tsquery parses a search for it the same way, so exact identifier search works. Run to_tsvector on a few real identifiers from your data and look at the output.

Dense vectors are good at paraphrase and bad at exact strings. They'll find "payment is due within thirty days" for a question about payment terms, and they'll miss the chunk that contains SKU A-4471-B because it looks like every other chunk with a product code in it. Full-text search has the opposite profile. Running both and combining the results catches most of what either one misses.

The combination is where implementations usually go wrong. A weighted sum like 0.45 * ts_rank_cd(...) + 0.55 * (1 - (embedding <=> $2)) looks reasonable and has two problems. The scores aren't on comparable scales: ts_rank_cd isn't bounded and depends on document length and term frequency, while cosine distance lives between 0 and 2, so the weights don't mean what they appear to mean. And because the expression has to be computed for every row before sorting, neither the GIN index nor the HNSW index can be used, so every query scores every chunk the tenant owns.

Reciprocal rank fusion avoids both problems by ignoring scores entirely. Each retriever returns its own top-k using its own index, and a document's fused score is the sum of 1 / (k + rank) over the lists it appears in. Cormack, Clarke and Büttcher introduced it at SIGIR 2009 with k = 60, and the paper found results weren't sensitive to the exact value:

 1prepare hybrid(text, vector, bigint) as
 2with lexical as (
 3  select id, row_number() over (order by rank desc, id) as r
 4  from (
 5    select c.id, ts_rank_cd(c.tsv, query) as rank
 6    from chunks c, websearch_to_tsquery('simple_unaccent', $1) query
 7    where c.tsv @@ query
 8      and c.tenant_id = $3
 9      and c.is_current
10    order by rank desc
11    limit 50
12  ) l
13),
14semantic as (
15  select id, row_number() over (order by dist, id) as r
16  from (
17    select c.id, c.embedding <=> $2 as dist
18    from chunks c
19    where c.tenant_id = $3
20      and c.is_current
21    order by c.embedding <=> $2
22    limit 50
23  ) s
24)
25select id,
26       l.r as lexical_rank,
27       s.r as semantic_rank,
28       coalesce(1.0 / (60 + l.r), 0) + coalesce(1.0 / (60 + s.r), 0) as rrf_score
29from lexical l
30full outer join semantic s using (id)
31order by rrf_score desc, id
32limit 12;

The lexical branch only ranks rows that match tsv @@ query, which the GIN index finds, and the semantic branch is a plain order by ... limit that HNSW can serve. Running it on 100,000 chunks spread over 50 tenants with Postgres 18 and pgvector 0.8.7, EXPLAIN showed a BitmapAnd of the tenant index and the GIN index for the lexical side. For the semantic side, at 2% of rows per tenant, the planner chose the tenant btree and an exact sort over the HNSW graph, which is the right call at that size. With a filter that matches most rows, it switches to an index scan on the HNSW index. Set hnsw.iterative_scan = relaxed_order for the transaction either way, so that when the graph is used with a filter it keeps scanning until it has 50 matches instead of returning a handful; the pgvector post covers why that matters.

The id tiebreakers make the ordering deterministic, which matters because you want the same query against the same snapshot to produce the same candidates when you're debugging. Log both ranks for every returned chunk. A chunk with a lexical rank and no semantic rank tells you something different from the reverse, and that's what you'll want when someone asks why a result showed up.

If you add a cross-encoder reranker on top, give it the fused top 30 or so and keep its version in the log too. A reranker is the one place where an extra model call usually pays for itself, because it reads the query and the passage together.

Free-form prose with citations sprinkled on afterwards is hard to verify. A structured contract is easier: the model returns a list of claims, and every claim lists the chunk IDs that support it. Validate the shape in code, then check that every cited ID was actually in this request's retrieval set, because models do cite IDs they saw in an earlier turn or make them up:

 1from typing import Literal
 2from pydantic import BaseModel, Field, ValidationError
 3
 4
 5class Claim(BaseModel):
 6    text: str = Field(min_length=1)
 7    evidence_ids: list[int] = Field(min_length=1)
 8
 9
10class Answer(BaseModel):
11    claims: list[Claim]
12    answer: str
13
14
15class Verdict(BaseModel):
16    label: Literal["supported", "contradicted", "insufficient"]
17
18
19def check_citations(answer: Answer, retrieved: dict[int, str]) -> list[str]:
20    problems = []
21    for i, claim in enumerate(answer.claims):
22        missing = [e for e in claim.evidence_ids if e not in retrieved]
23        if missing:
24            problems.append(f"claim {i}: cites chunks not in this retrieval: {missing}")
25    return problems
26
27
28def parse_verdict(raw: str) -> str:
29    try:
30        return Verdict.model_validate_json(raw).label
31    except ValidationError:
32        return "insufficient"  # fail closed: an unparseable verdict never counts as support
33
34
35def gate(verdicts: list[str], max_insufficient: float = 0.2) -> bool:
36    if "contradicted" in verdicts:
37        return False
38    return verdicts.count("insufficient") / max(len(verdicts), 1) <= max_insufficient

A claim with an empty evidence_ids list fails validation with Pydantic's own message (List should have at least 1 item after validation, not 0), which is the kind of error you want in a log. Existence isn't support, though. A cited chunk can be real and still not say what the claim says, which is where a verifier comes in: a second, narrow model call per claim that sees only the claim and the cited text and returns one of three labels.

 1{
 2  "task": "Decide whether the evidence supports the claim. Use only the evidence.",
 3  "labels": ["supported", "contradicted", "insufficient"],
 4  "rules": [
 5    "Do not use outside knowledge.",
 6    "A numeric mismatch is a contradiction.",
 7    "If the evidence does not settle the claim, answer insufficient."
 8  ],
 9  "claim": "The SLA guarantees 99.95% uptime.",
10  "evidence": "Section 4.2: Service availability target is 99.9% per calendar month.",
11  "output": {"label": "one of the labels"}
12}

Use your provider's structured output mode so the reply is JSON, and still parse it defensively: parse_verdict treats anything it can't parse as insufficient, so a broken verifier response can never turn into a pass. The verifier should be a different call from the generator, ideally a smaller and cheaper model, because asking the generator to grade its own answer mostly measures how confident it sounds.

The gate in the code is a starting point: any contradicted claim blocks the answer, and so does a share of unsupported claims above 20%. What "blocks" means depends on the product. An internal tool can show the answer with the unsupported claims marked, while anything that triggers an action, sends an email or updates a record should fall back to quoting the evidence or go to a human review queue that shows the question, the retrieved chunks, the draft and the verdicts on one screen. For actions, the post on why tool calls aren't actions describes an approval queue that fits behind this gate. For clear-cut fields such as amounts, dates and IDs, add deterministic checks next to the verifier: if a claim states an invoice total, the cited chunk should contain that number.

A lot of what users ask an assistant is a database query in disguise. "Show unpaid invoices from February for Acme" has an exact answer in your tables, and sending it through retrieval and generation turns a solved problem into a probabilistic one. Put a classifier in front, even a crude one, and route by task. Questions over structured data go to parameterized queries against known views, extraction from fixed templates goes to a parser plus schema validation with only the ambiguous fields sent to a model, and only the genuinely open questions over unstructured documents take the RAG path.

If you let a model write SQL for the open-ended cases, run it under a database role that can only read the views it's meant to see, with a statement timeout, and treat the generated SQL as untrusted input. The read-only role is what makes this safe, while parsing the statement to reject anything but a single select is a useful early error on top of that.

The routing also gives you a graceful way to degrade. When the generation path misbehaves after a model update or a provider outage, the assistant can fall back to returning the top retrieved passages with their sources, which is less pleasant than a fluent answer and much better than a fabricated one.

If the right chunk isn't in the top 10, no prompt will fix the answer, so measure retrieval first and separately. Build the evaluation set from real queries, sampled from search logs, support tickets and whatever people actually type, and label the chunks or at least the documents that answer each one. Then track recall@k for the retrieval stage on its own, and citation coverage plus unsupported-claim rate for the generation stage. Set thresholds per use case, because a policy assistant for operations and a tool that summarizes sales notes can tolerate very different error rates, and break the numbers down by query type so a regression on identifier lookups can't hide behind an improvement on long questions.

Log every retrieval event: the query, the normalized query, the embedding model, the candidate chunk IDs with their lexical and semantic ranks, the fused order, the chosen citations, the verifier labels and the final outcome. With that log you can tell whether a bad answer came from full-text search missing a SKU, from vector search preferring a similar but wrong policy, or from generation stretching past the evidence, and each of those has a different fix.

Evaluation sets age as documents change and users invent new shorthand, so rerun the evaluation whenever chunking, the embedding model, the fusion settings, the parser or the access-control logic changes, and refresh the query sample every few months. Changing the embedding model is the biggest of those changes, and the post on rolling out a new embedding model in pgvector covers how to do it without losing the ability to roll back.