Production AI evaluation: test the workflow, not just the model

Model benchmarks are useful for comparison. They do not tell you whether an enterprise workflow is ready for production.

A production system includes prompts, retrieved data, tools, business rules, permissions, model responses, evaluations, and human decisions. Failure can enter at any point. Testing only the model leaves most of the system unexamined.

Begin with the job to be done

Define what successful completion means in operational terms. For a contract workflow, success may require the correct clause, the correct interpretation, a traceable source, and an appropriate next action. For an engineering workflow, it may require working code, passing tests, policy compliance, and an independent review.

The evaluation should reflect the job, not a generic idea of intelligence.

Build a representative test set

Use examples that reflect the range of real work: common cases, difficult cases, incomplete inputs, conflicting records, unusual formats, and known failure modes. Protect confidential information and use approved data-handling methods.

The test set should include cases the system must refuse, defer, or escalate. A system that always produces an answer may be less safe than one that knows when evidence is insufficient.

Score multiple dimensions

Accuracy alone is rarely enough. Depending on the workflow, evaluate:

• Correctness and completeness
• Grounding in the provided source
• Format and schema adherence
• Tool selection and tool output
• Policy compliance
• Appropriate uncertainty and escalation
• Latency and cost
• Reviewer acceptance

Some measures can be automated. Others require expert judgment. High-stakes workflows usually need both.

Test components and the full path

Component tests help isolate problems. Test retrieval separately from interpretation, tool execution separately from planning, and formatting separately from factual quality.

Then run end-to-end tests. A retrieval component may perform well alone but return the wrong information after permissions, filters, or orchestration are applied. A model may produce a correct recommendation that the workflow writes to the wrong record.

Production readiness depends on the whole path.

Evaluate change continuously

Models, prompts, data, APIs, policies, and user behavior change. A system that passed before launch can drift.

Run evaluations before material changes and monitor performance after deployment. Track the version of the prompt, model, retrieval process, tool, and policy used for each result. When a failure occurs, the team should be able to reproduce the conditions.

Include operational and business measures

Technical quality is necessary but insufficient. Measure whether the workflow improves cycle time, throughput, exceptions, rework, or the relevant business KPI. Also measure the human-review burden.

A workflow can achieve high answer quality and still be too slow, too expensive, or too difficult for users to adopt. Those are production failures too.

Create release gates

Define the minimum evidence required to move from development to limited release and from limited release to broader use. Gates may include evaluation thresholds, security review, approved data access, human-control design, rollback procedures, monitoring, and named ownership.

Release gates should be proportional to risk. They should create a clear route to production, not an indefinite waiting room.

Evaluation gives the enterprise a way to improve without guessing. It shows whether a change made the system better, where the workflow fails, and whether the cost and risk are justified by the result.

The unit of evaluation is not the model. It is the work the company expects the complete system to perform.