An answer is only as useful as its evidence.
Separating retrieval failures from answer failures in a document assistant.
Follow the evidence through the system
Consider a document assistant answering a question about an internal policy. Before it can compose a useful response, it needs to find the relevant passage, preserve its meaning, and distinguish the current policy from an older version. A fluent answer cannot repair evidence that never reached the model.
Inspect the retrieved passages alongside the final answer. This separates two questions: did the system find enough information, and did it use that information correctly? Keeping those questions separate makes a failure easier to diagnose.
Make retrieval inspectable
For each test question, retain the source identifiers, relevant metadata, retrieved text, and ranking order. Avoid logging sensitive material unnecessarily; the trace itself needs appropriate access and retention rules.
Ask a reviewer whether the evidence could support a correct answer. If not, inspect ingestion, extraction, chunk boundaries, document versions, and access filters before changing the answer prompt. An extraction problem and a ranking problem need different fixes.
Treat citations as something to check
A citation is useful when a reader can open the referenced passage and verify the associated claim. A document title alone may be too coarse, especially when the document is long or internally inconsistent.
Include test cases where the documents do not contain an answer. Define an acceptable response for those cases: explain the missing evidence, ask for clarification, or route to a person. A confident guess should not count as success simply because it reads well.
Change one part, then compare
Keep a fixed set of questions while trying a retrieval change. Compare evidence quality and answer quality separately, and record what became worse as well as what improved. Add newly discovered failures to a regression set.
The aim is a traceable relationship between a technical change and a useful result. That relationship is more valuable than a collection of impressive answers selected after the fact.