The answer was wrong, and retrieval was not the only reason.
When a retrieval pipeline answers badly, the usual response is to blame the vector database. Usually it is something narrower: the passage was never found, or it was found and buried, or it was in the context and the prompt ignored it. This reads eight questions against the passages that actually answer them and works out which of those happened, one question at a time.
SimulationThe handbook, the questions and the retrieval results are written for this demonstration, and the diagnosis is real code running in your browser rather than a model. Nothing is connected to a vector store. The production build reads real query logs and your own relevance labels, and re-runs a failing question against a candidate fix.
8 questions were asked. 88 per cent of them retrieved the passage that answers them, and 50 per cent ended with an answer that used it. 3 answers were given without the passage in hand, and 3 of the faults are in the pipeline rather than in the source material.
Question by question
How long is paternity leave, and is it paid?
The right passage was ranked 1 of 5 and the answer used it.
What the pipeline answered
Paternity leave is four weeks at full pay, taken in up to two blocks within six months of the birth.
GroundedThe passage that answers itranked 1
Paternity leave is four weeks at full pay, and it can be taken in two blocks within six months of the birth.
The fix
Nothing to fix on this query.
Retrieval order5 results
- 1People Handbook, Parental leave0.74answers it
- 2People Handbook, Parental leave0.61
- 3People Handbook, Annual leave0.44
- 4People Handbook, Sickness0.31
- 5Expenses Policy, Receipts0.18
The faded entries fall after the 5th result, because that is what the pipeline passes to the model. A passage down there was retrieved and then thrown away.
Fix these, most first
Grouped by root cause rather than by question, and ordered by how many questions each one settles. A repair that fixes one question is a patch; a repair that fixes three is a change worth making.
- 1Assemble the context deliberatelysmall change
Put the strongest passage first and the next strongest last, drop the padding, and keep the passage count to what the prompt actually uses.
fixes Do I need a purchase order for a 9,000 pound contract?
- 2Require a quote, and require a refusal without onesmall change
The answer must quote the passage it relied on. If no passage supports the claim, the correct output is a refusal, not a confident sentence.
fixes Can I install a tool on my work laptop myself?
- 3Add keyword retrieval beside the embeddingsmedium change
Embeddings are good at meaning and poor at exact strings. A limit, a threshold or a policy name is an exact string, so run both and merge the results.
fixes What is the hotel limit for a stay outside London?
- 4Write the missing sectionmedium change
The pipeline refused correctly, so this is a gap in the handbook rather than a bug in the pipeline.
fixes What is the policy on bringing a dog to the office?
How the diagnosis works
Two facts per question drive everything: which passage answers it, and where that passage sat in the ranking. The cause is read off those two facts, so the tool cannot report a cause the data does not support.
The five causes
- Not retrieved: the passage was not in the results at all, so retrieval is the problem.
- Below the cut: retrieved, but ranked under the passages that were passed on.
- In the context and unused: it was passed on and the answer did not use it.
- Prompt: the passage was near the top and the answer hedged or reversed it.
- Coverage: nothing in the corpus answers the question, and refusing was correct.
What it does not claim
- It needs the relevance judgement. Without a person marking which passage answers each question, there is nothing to measure against.
- It reads the query log it is given, and it cannot see the embeddings, the index or the model's sampling.
- A coverage gap is reported as a gap in the source material rather than as a pipeline fault, because that is what it is.
- It does not change the pipeline. It says which change is worth making, and why.