Template
Hallucination Triage Sheet
A wrong answer is usually a hole in the context rather than a lying model, so the first move is to paste the answer in and re-run.
This is the sheet you fill in when a model returns something wrong, before anybody edits a prompt. Its only job is to name which of six causes you are looking at, because five of them have different fixes and one of them is not a bug at all.
Triage pays off in batches. A single wrong answer is faster to just fix.
Why this exists
The reflex fix is a line in the prompt telling the model not to make things up. It almost never works, because the most common cause is not invention — it is that the correct answer was never in the context. The model was filling a hole, and a stern instruction does not fill it.
Without triage, every wrong answer becomes another sentence in the prompt. The prompt grows, starts contradicting itself, and the failure rate sits where it was. Then somebody proposes a bigger model, which is the expensive version of the same guess. Both moves feel like progress because neither one required you to find out what happened.
How to use it
Run these in order and stop at the first one that answers.
- Paste the correct answer into the context and re-run. Right now? It is a retrieval gap. Fix the index, the chunking, or the query — not the prompt.
- Still wrong with the answer in front of it? Ask the same question alone, with minimal context. Correct alone but wrong in the full task means a prompt gap: the instruction, the ordering, or the sheer volume of context is defeating it.
- Wrong even alone, every time? That is a model limit. The options are decomposition, a stronger model, or removing the feature. Not a better sentence.
- Check the source document. If what it cited is stale or wrong, the model reported it faithfully. This is a data ticket, and it is the class that gets misfiled most often as a model problem.
- Check the reporter. Some share of any triage queue is answers that were correct. Log them. The count tells you something about expectations and documentation, not about the model.
- What is left is ambiguity. The question had two defensible readings and nobody had chosen one.
Ambiguous is not a wastebasket. Closing a row as ambiguous requires a written rule saying which reading wins, and that rule goes in two places — the prompt and the eval set. An ambiguous row that produces no rule will be reported again next month by somebody else, and you will triage it again.
Where it goes wrong
Triage names the class. It does not price the fix. A confirmed model limit has only expensive options, and this sheet will hand you the same verdict every week until somebody is allowed to choose between paying for it and dropping the feature. If nobody holds that authority, triage quietly becomes a well-organized list of things that will never be fixed.
Test one has a trap worth knowing. Pasting the answer in can make the model right for the wrong reason — it may be copying your pasted text rather than reasoning over it. That is good enough for classification, since you only needed to know the hole existed. It is not evidence that better retrieval will be sufficient.
Do not use this where wrong is undefined. Tone, length, helpfulness and house style are not hallucinations; they belong in an eval with a rubric, and forcing them through a triage taxonomy produces confident nonsense. Do not use it for anything you can reproduce in one line of code either — if a date parses as the wrong month, that is a bug, and it does not need a class.
Copy and Clear become available when the worksheet is ready.
HALLUCINATION TRIAGE SHEET —
Triage run by: Date:
Batch size: Source of rows: complaint queue / spot check
A. INTAKE — one row per wrong answer
id
date_reported
reported_by
the_question verbatim, including the typo
what_it_said verbatim output, do not paraphrase
what_was_right the correct answer, and who says so
source_expected the document that should have answered it
context_captured yes | no if no, reproduce before triaging
model
prompt_ver invoice-extract.json.v4.2.0
severity cosmetic | misleading | acted-on | unsafe
class (filled in by section B)
fix_owner
eval_row_added | n/a
B. THE TESTS — run in order, stop at the first that answers
T1 Paste the correct answer into the context. Re-run.
Now correct? -> class = RETRIEVAL GAP
T2 Ask the same question alone, minimal context.
Correct alone, wrong in the task? -> class = PROMPT GAP
T3 Run T2 three times.
Wrong every time, even alone? -> class = MODEL LIMIT
T4 Open the source document it used.
Document stale, wrong, or contradicts another source?
-> class = BAD SOURCE (a data ticket, not an AI ticket)
T5 Re-read what_was_right against the source of truth.
Output actually correct, reporter mistaken?
-> class = NOT A DEFECT (log it; do not silently close it)
T6 Nothing above fired.
Two defensible readings of the question?
-> class = AMBIGUOUS
C. WHAT EACH CLASS ACTUALLY COSTS
RETRIEVAL GAP fix: chunking, index coverage, query rewriting,
the document was never ingested at all.
tell: the answer exists somewhere you can point to.
cost: usually the cheapest real fix.
PROMPT GAP fix: instruction wording, ordering, cutting context,
splitting one call into two.
tell: it knows the answer, it just did not use it.
cost: cheap, but re-run the eval — this is where
quiet regressions come from.
MODEL LIMIT fix: decompose the task, stronger model, drop it.
tell: fails alone, repeatedly, with everything given.
cost: money or scope. Needs a decision-maker.
BAD SOURCE fix: the document, and whoever owns it.
tell: the output faithfully matches a wrong source.
cost: not yours. Route it out.
NOT A DEFECT fix: expectation, documentation, or the UI's framing.
tell: the disagreement is about what was asked.
cost: none, but a rising count is a real signal.
AMBIGUOUS fix: a WRITTEN RULE naming which reading wins.
tell: two reasonable people read it differently.
cost: one decision, once — if you actually write it.
D. CLOSING AN AMBIGUOUS ROW — all four or it is not closed
- Written rule: "When , the answer is ."
- Decided by: on
- Rule added to the prompt (which version: )
E. BATCH ROLLUP — the numbers worth reporting
rows triaged
retrieval gap (expect this to be the largest)
prompt gap
model limit
bad source
not a defect
ambiguous ambiguous rows closed with a rule:
severity acted-on or unsafe: -> escalate these first, by severity
not by class
Two things to say out loud when you report this:
- If retrieval gap is small and model limit is large, stop editing
prompts. You have a scope problem, not a tuning problem.
- If not-a-defect is large, the feature is unclear about what it does.
That is a design fix, and no prompt change will touch it.
F. WHEN NOT TO OPEN THIS SHEET
- One complaint. Just fix it.
- You cannot reproduce it and did not capture the context.
- The complaint is about tone, length or style. That is an eval.
- The bug is deterministic and in your own code.