Evaluate an AI answer contract before paying for model calls
Run an offline retrieval baseline, inspect its real failure, and catch six deliberately contaminated responses with a transparent evaluation harness.
Reproducible DWKD demonstration · Offline baseline · No model provider used
“It gave a convincing answer” is a weak acceptance test for an AI application. Before choosing a model, decide what an answer must contain, what it may cite, when it should decline, and how you will detect an unsupported claim.
This build makes those decisions executable. It runs a deliberately simple retrieval baseline against fictional policy documents, then tests six deliberately contaminated responses. The baseline passes seven of eight authored cases and fails one paraphrase. All six contaminated fixtures are rejected. You can inspect every question, response and failure code.
This is not a live language-model evaluation. No provider API is called. The baseline uses word overlap and literal quotations; the contaminated responses were written specifically to exercise the checker. The value is an inspectable starting contract, not a claim about the quality or safety of any model.
Download the documents, evaluation cases, responses, code and tests
Learn: separate retrieval, evidence and answer quality
The fixture contains four public policy documents covering dispatch, returns, support hours and access to order reports. A fifth invented report is labelled for operations staff. The public answering function filters by that label before ranking documents. It then counts meaningful shared words, requires at least two overlaps, and declines if the highest scores tie.
A returned answer is the selected document’s text, with its source ID and exact quotation. That is intentionally restrictive. It lets us check whether a citation exists, whether the quotation is really in that document, and whether the response adds text outside the extractive contract. The authored test case also names which source is relevant or whether the question should be declined.
Question + permitted documents
↓
Public-source filter
↓
Word-overlap retrieval
↓
Quote with citation OR decline
↓
Schema + source + quote + case checks
↓
Pass OR specific failure codesBuild: begin with a successful answer and a real failure
For “What are human support hours?” the baseline retrieves the support document and returns:
Human support is available Monday to Friday, 09:00–17:00 UTC. No weekend coverage is promised.
The source exists, the quotation matches it exactly, it is available to the public audience, and the test case expects that source. The response passes this narrow contract.
For “Can I phone someone on Sunday?” the system declines: “The supplied public documents do not support an answer.” That is a failure in the authored test. The support document is relevant, but the baseline does not connect “phone someone” to “human support” or Sunday to weekend coverage. The recorded code is unnecessary_abstention.
Leaving this failure visible is useful. It tells us what an improved retrieval method would need to solve. Replacing the wording in the test just to get a perfect score would hide the limitation. Adding a synonym rule may improve this example, but it would still need evaluation on new, independently written questions.
Show: a real source is not enough
| Contamination | Check that rejects it |
|---|---|
| An invented citation promises a 90-day return window. | unknown_source |
| A real returns document is cited for a fabricated “always free” quotation. | quote_not_in_source |
| A valid support quotation is followed by a made-up Sunday service promise. | answer_outside_extractive_contract |
| A literal quotation exposes the operations-only fixture. | unauthorized_source |
| A support-hours question is answered with a returns-policy quote. | wrong_evidence |
| The response declines an answerable dispatch question. | unnecessary_abstention |
These six rejections are deterministic test results on authored fixtures. They do not establish a hallucination rate, prompt-injection resistance, security certification or real-world false-positive rate. An attacker was not interacting with a deployed model. The point is to make distinct failure categories visible instead of compressing everything into one quality score.
What the numbers actually say
| Measure | Result |
|---|---|
| Baseline cases | 8 authored questions |
| Baseline passes | 7 |
| Baseline failures | 1, an unnecessary abstention on a paraphrase |
| Deliberately contaminated responses | 6 rejected out of 6 |
| Model API calls and billed model tokens | 0 |
| Model provider charge for this run | $0.00; local compute and authoring time are not costed |
Zero provider charge does not mean zero operating cost. This program does work on your computer, and someone must design and review its cases. It also tells us nothing about production latency, model pricing or how much a real application will cost. A live comparison would need recorded provider/model versions, actual token usage, current rates, retries, request timing and an evaluation set separated from development examples.
Run and inspect it
Extract the ZIP and use Python 3.10 or newer. No dependencies, credentials or network connection are needed after download.
New to running Python? Extract the ZIP and open its README.md first. It walks through finding the folder, opening a terminal, Windows commands and common errors. You can also inspect results.json without running code. These examples use fictional data and need no API key or paid service.
python3 evaluate.py
python3 -m unittest test_demo.py -vInspect documents.json, cases.json and contaminated-responses.json before reading the summary. The output includes every response and its failure codes; results.json contains the baseline. Copy the fixture folder, write a new paraphrase, and run python3 evaluate.py --fixtures your-fixtures. Do not add private information to this public example.
The boundaries that remain
The quotation checker is not a general fact checker. A short quotation can be technically present yet misleading because it omits context. The case-specific source check does not prove that an arbitrary question is fully answered. Exact extraction rejects legitimate paraphrases, and the small authored set is visible to the implementation; it is not held-out evidence of generalization.
The audience label illustrates where document filtering belongs, but it is not production authentication. A real application needs identity-derived permissions, permission checks before retrieval and delivery, document lifecycle handling, and independent security testing. This program executes no tools and demonstrates no agent sandbox.
The next investment should be chosen from observed failures: improve retrieval, write stronger cases, add a reviewed model adapter if necessary, and compare whether the extra capability justifies its cost. A model should enter a system with a testable contract already waiting for it.
Explore AI and automation consulting or Work with Kenny to define an answer contract, build a reproducible baseline and evaluate the failure modes that matter to your workflow.
Related builds: audit the data before trusting its conclusions and make the surrounding workflow recoverable.