How to evaluate an AI tool in a week

A short evaluation can expose a poor fit before a team commits to a larger pilot. This five-day plan is an initial screening exercise, not a substitute for the validation a consequential deployment needs.

Decide what would make you reject the tool before testing it. A longer evaluation can also say no; its value depends on the question and the evidence, not its duration.

Day one: pick one task and define acceptable

Not “evaluate the tool”. One specific, repeated task the tool would do.

Then, before anything is run, write down what an acceptable output looks like. Specifically enough that two people scoring the same output would agree. “Extracts all four fields correctly, and flags rather than guesses when a field is absent” is a standard. “Good quality” is not.

Choose a narrow task and collect an initial set of representative cases. Twenty is a manageable starting exercise, not a statistically sufficient sample for every task or failure rate.

Day two: collect an initial set of approved cases

Use public, synthetic or properly approved material until privacy, contractual and security requirements permit real data in the selected service and account. Do not upload sensitive work merely because it makes a realistic test.

Include routine cases, difficult cases and known failures. A twelve/five/three split can organize an initial twenty-case exercise, but choose coverage from the task and extend it where a failure matters.

Score the initial cases carefully and note missing coverage. A larger, well-designed evaluation may be necessary; twenty cases are a starting exercise, not a generally sufficient alternative.

Day three: run them, and time the checking

Run all twenty. Then score each one against the standard you wrote on day one, and — this is the part people skip — time how long verification takes.

Measure accepted output quality and the effort needed to verify it. A higher score with slower checking may or may not be worthwhile; compare total cost, error severity and the manual baseline.

Note the shape of the failures too, not just the count. Failures that are obvious are much cheaper than failures that look plausible, and the second kind should weigh heavily against a tool regardless of how rare they are.

Day four: test the things that break it later

Probe possible problems in the transition to routine use, while checking failure modes specific to this task.

Give it something out of scope and see whether it declines or confabulates. A tool that answers confidently about a document it was not given is a liability at volume.

Have somebody else run five cases — ideally somebody who was not in the demo. Much of a pilot’s success lives in one person’s undocumented technique, and you want to find that out now.

Test the actual data shape with permitted material. Use approved records or synthetic equivalents with realistic fields and malformed inputs; preserving realism does not override data restrictions.

Ask what happens when it is wrong. Is the error visible? Does anyone find out? What does undoing it involve?

Day five: decide against the threshold

Compare the results with the acceptance criteria you declared before testing. Record what you tested, the outcomes, verification effort, unacceptable failures and the decision in a short review.

At the end, decide to reject, revise the test, or advance to a bounded pilot. Broad adoption needs representative validation plus the privacy, security, reliability and operational checks required for the use case.

“Not yet” is the most useful outcome and the least used. It converts a rejection into a condition, which is a far better thing to hand a vendor and a far better thing to revisit.

The mistakes that invalidate the week

Letting the vendor run it. They will choose the cases, and they will choose well. The entire value of this exercise is that the cases come from your work and include the ones that are awkward. Have them demonstrate by all means — then run your twenty yourself.

If reviewers disagree, inspect the disputed cases and clarify the criteria. Disagreement can reveal ambiguity or a missing requirement; do not discard it as information-free.

Only one person running it. A single evaluator’s technique gets baked into the result without anybody noticing. Have a second person run a handful of the same cases cold; the gap between them is a real finding about how much the tool depends on skill.

Testing the model rather than the workflow. What matters is the whole path — getting the input in, getting the output somewhere useful, and checking it. A tool with an excellent model and an awkward integration can easily be worse in practice than a mediocre one that fits how people already work.

What this deliberately leaves out

Worth naming, so nobody mistakes this for a procurement process.

This screen does not complete procurement, integration planning or a security and compliance assessment. Resolve the requirements for the proposed test before running it.

Check data handling, permissions and any required approvals before sending data or giving a tool access. Use approved synthetic or public examples while unresolved questions are reviewed. Deeper procurement work can follow a promising screen, but required protections cannot wait until after exposure.

What the short version should produce

A bounded screen should produce evidence, a decision and a record of what remains untested.

A useful evaluation produces a decision tied to evidence, including a clear no when requirements are not met. The five-day limit keeps this first screen bounded; it does not cap the work needed to validate a high-risk system.

Save the cases, criteria, settings, outcomes and unresolved questions. They become the starting point for the next test, or the documented reason not to proceed.

Technical review, October 10, 2026. References: OpenAI: evaluation best practices; NIST: AI Risk Management Framework; OpenAI: API data controls.

Try this

Write the criteria for advancing to a bounded pilot, including unacceptable failures, verification effort and necessary approvals. Test an initial case set, record its gaps, and plan further validation before broad adoption.