Evaluate AI on your task, including the failures

Define success in observable terms

For extraction, success may mean finding every invoice total with the correct currency. For a summary, it may mean preserving all decisions and unresolved questions. For a customer draft, it may mean factual accuracy, appropriate tone, and no unauthorized promises. Avoid a vague score such as “sounds smart.”

Create a small set of realistic examples. Include the ordinary cases you expect most often and the unusual cases that would be costly if mishandled. Keep examples separate from any demonstrations used to tune the instructions when you need an honest estimate of generalization.

Inspect several dimensions

Accuracy is only one dimension. Measure omissions, unsupported additions, formatting failures, and whether the system knows when evidence is insufficient. Also record total elapsed time, usage cost, and the human effort needed to approve the result.

For an agent, evaluate actions and permissions as well as its explanation. Did it operate on the right file? Did it stop before a consequential step? Did it report a failed tool call accurately? An eloquent final message cannot compensate for an incorrect external action.

A useful test set for a document assistant

Use a straightforward question, a question with an exception, a conflict between two documents, a missing answer, and a misleading instruction embedded in a document. Write the required behavior for each. The last case checks whether the system treats source content as untrusted data rather than as a new command.

Create an evaluation rubric for this assistant. Include factual correctness, source support, handling of unknowns, and permitted actions. Suggest realistic failure cases. Do not score a response higher merely because it is longer or more confident.

Have a knowledgeable person inspect a sample of outputs. If another model helps grade them, verify that grading process too. Automated evaluators can share blind spots or reward superficial features.

Use results to make a decision

Report the actual sample size and conditions. A handful of successful trials is a pilot result, not a universal reliability rate. Repeated runs may be needed for tasks whose outputs vary. Focus additional testing on concrete uncertainty or risk rather than building a test collection with no decision attached.

Keep the configuration, model identifier when available, date, and examples used. Recheck the important cases after an update. Use the evidence to decide whether to deploy, narrow the scope, add review, or choose another method. Evaluation earns its value by changing what you do next.