How to Evaluate an AI Agent Before Production: A Practical B2B Framework

An AI agent completes the demo successfully.

It finds the correct account, summarizes the history, updates the CRM, and drafts a useful response.

The team approves the pilot.

Then production introduces conditions that were absent from the demonstration: a required field is missing; two systems disagree; the API times out; the same event is delivered twice; the customer asks an ambiguous question; a document contains a malicious instruction; approval expires while the process continues; and the agent retries an action that already succeeded.

The agent was not necessarily evaluated incorrectly.

It may not have been evaluated at all.

A successful demonstration proves that one execution path can work.

Production readiness requires evidence that the system behaves acceptably across normal, ambiguous, adversarial, and partially failed conditions.

An AI agent should be evaluated at five levels: Task outcome - did it complete the business task correctly? Execution trace - did it choose the right tools and actions? Safety and policy - did it remain inside its permissions? Economics - did it complete the task at an acceptable cost and speed? Operational resilience - did it handle missing data, failures, retries, and escalation correctly?

The unit of evaluation should be the completed business task, not the quality of a single model response.

Why standard LLM evaluation is not enough

A chatbot may be evaluated mainly on its final answer.

An agent can produce a good final answer while behaving incorrectly during execution.

It may retrieve restricted data; use the wrong tool; send invalid arguments; update the wrong account; repeat an irreversible action; conceal uncertainty; exceed its cost limit; and reach the correct result through an unsafe path.

Agent evaluation therefore has to inspect both the result and the trace that produced it.

Current official evaluation guidance increasingly describes the process as a combination of test-case design, execution, trace generation, scoring, simulation, and failure analysis. Google's agent-evaluation documentation, for example, includes multi-turn scenarios and simulated tool failures such as service errors and latency spikes. (Google Cloud Documentation)

Start with a written task contract

Before creating a test suite, define the task precisely.

A useful task contract includes:

Without this contract, a team may disagree about whether the same agent run was successful.

One person evaluates answer quality.

Another evaluates speed.

Another assumes that updating the CRM was permitted.

The agent cannot be measured against requirements that were never defined.

Evaluate the business result first

The primary metric is not "the response sounded good."

It is task success.

Examples: Was the correct customer identified? Was the correct invoice matched? Was the ticket routed to the right queue? Was the requested report generated from the approved data source? Was the record updated accurately? Was the task escalated when evidence was insufficient?

A task should be marked successful only when all mandatory conditions are satisfied.

For high-impact workflows, partial success may still be failure.

An agent that prepares the correct refund amount but applies it to the wrong account has not achieved 90% success.

It has created an incident.

Metrics that reveal whether the agent works

Task success rate

Task Success Rate = Correctly Completed Tasks / Total Evaluated Tasks

Success criteria should be deterministic where possible.

For example: correct record; correct field value; correct route; correct approval state; no prohibited action; and required evidence attached.

Tool-selection accuracy

Did the agent choose the appropriate tool?

An agent may have access to CRM search; billing lookup; knowledge retrieval; email draft; customer update; and approval request.

Selecting a semantically related but operationally incorrect tool should be counted as an error even when the final output appears reasonable.

Tool-argument validity

A correct tool with incorrect parameters is still a failed action.

Evaluate required arguments; account identifiers; dates; amounts; status values; scope; recipient; idempotency key; and approval token.

Structured outputs and schema validation reduce risk, but validation must also verify business meaning.

A syntactically valid amount of $100,000 may still violate the agent's approval limit.

Policy-compliance rate

Measure whether the agent: stayed within permitted systems; avoided restricted information; obtained required approval; respected action limits; stopped when evidence was insufficient; avoided prohibited actions; and handled external instructions as untrusted content.

OWASP recommends least privilege, human oversight for high-risk actions, limits on recursion and retries, validation of external inputs, and structured adversarial testing before production. (OWASP Cheat Sheet Series)

For a prohibited irreversible action, the acceptable unauthorized-execution count in testing should be zero.

Human-correction rate

Correction Rate = Tasks Requiring Material Human Changes / Tasks Reviewed

Do not count formatting preferences as material corrections.

Track changes that affect: decision; recipient; amount; classification; evidence; system action; and customer commitment.

A high correction rate means the agent may be shifting work rather than eliminating it.

Escalation quality

An agent should not attempt to solve every case.

Evaluate whether it escalates: when required information is missing; when records conflict; when confidence is insufficient; when the action exceeds its authority; when the request is unusual; when tools fail; and when policy interpretation is required.

Both under-escalation and over-escalation matter.

Under-escalation creates risk.

Over-escalation eliminates the economic value of automation.

Duplicate-execution rate

Agents must be tested against repeated messages, retries, and delayed responses.

The system should not: issue two refunds; create two invoices; send duplicate emails; create duplicate CRM records; and reopen a completed task.

Idempotency is not only an integration concern. It is part of agent correctness.

Latency and task duration

Measure time to first useful action; total task duration; tool waiting time; number of model iterations; human approval delay; and retry delay.

A task may be accurate but operationally unusable when it takes longer than the manual process.

Cost per completed task

Cost per Successful Task = (Model + Tool + Infrastructure + Review Cost) / Successfully Completed Tasks

Include model calls; retrieval; reranking; external APIs; observability; human review; failed attempts; retries; and engineering support.

Cost per request hides failures.

Cost per completed task exposes them.

Build the test suite around failure, not only success

A representative evaluation suite should contain several scenario classes.

Normal cases

Routine tasks with complete, consistent data.

Purpose: establish baseline performance.

Ambiguous cases

Requests with multiple plausible interpretations.

Expected behavior: ask for clarification or apply a documented rule.

Missing-data cases

Required information is absent.

Expected behavior: identify the missing evidence and stop or escalate.

Conflicting-data cases

CRM, billing, and support systems disagree.

Expected behavior: follow the source-of-truth policy rather than selecting the most convenient answer.

Tool-failure cases

Simulate: timeout; authentication failure; rate limit; malformed response; partial update; and temporary service outage.

Expected behavior: retry only where safe, preserve state, avoid duplicates, and escalate when limits are reached.

Duplicate-event cases

Deliver the same trigger more than once.

Expected behavior: recognize that the task has already been completed.

Stale-state cases

Change a record after the agent begins but before it acts.

Expected behavior: revalidate critical data before execution.

Adversarial cases

Place malicious or misleading instructions inside: email; PDF; CRM note; webpage; support ticket; and retrieved document.

Expected behavior: treat external content as data, not as authority to override system policy.

Unauthorized-request cases

Ask the agent to perform an action outside its permissions.

Expected behavior: refuse or route for authorized approval.

Partial-failure cases

The first action succeeds and the second fails.

Expected behavior: preserve an accurate state, avoid repeating the successful action, and provide a recoverable escalation.

Create risk-based acceptance criteria

There should not be one universal performance threshold for every agent.

Classify tasks by potential impact.

A high average task-success rate cannot compensate for one critical unauthorized action.

Results should therefore be reported by scenario and severity - not only as one blended percentage.

Log the complete execution trace

For every evaluated run, retain: Scenario ID; Input and relevant context; Agent version; Model and configuration; Instruction and policy version; Retrieved evidence; Tool calls and parameters; Permission decisions; Approval requests and responses; Errors and retries; Final action; Duration and cost; Evaluation result; and Failure category.

This creates three benefits: failed behavior can be reproduced; releases can be compared; and previously fixed failures can become regression tests.

NIST's Generative AI profile organizes risk management around the functions Govern, Map, Measure, and Manage. For an agent program, evaluation is the bridge between mapping intended behavior and managing actual production risk. (NIST)

Use a staged production rollout

Passing an offline test suite is not the end of evaluation.

Production introduces real users, changing data, integrations, latency, unexpected wording, and organizational behavior.

A safer rollout uses four stages.

Shadow mode

The agent evaluates live tasks but cannot act.

Compare its proposed decisions with actual human decisions.

Measure agreement; false positives; false negatives; missing evidence; and escalation behavior.

Recommendation mode

The agent presents recommendations to employees.

Humans perform the action.

This reveals whether the recommendation is useful and whether the agent provides enough evidence.

Approval mode

The agent prepares the action and executes it after explicit approval.

Measure approval rate; correction rate; reviewer time; expired approvals; rejected actions; and post-execution errors.

Bounded production

The agent acts autonomously only for a narrow, low-risk category.

Use account allowlists; amount limits; tool limits; rate limits; time limits; spend limits; automatic escalation; and immediate shutdown controls.

Autonomy should expand one boundary at a time.

Production monitoring is continuous evaluation

After launch, monitor: task success; correction and reversal; escalation; tool failure; retries; duplicate actions; unauthorized attempts; average task cost; latency; abnormal behavior; performance by scenario type; and performance by model or agent version.

Averages can conceal important failures.

For example, a 95% success rate may include 99% on routine cases; 60% on ambiguous cases; and 40% when one system is unavailable.

The aggregate number looks healthy.

The operating system is not.

A production-readiness scorecard

Use the following structure to reach a go, limited-go, or no-go decision.

Mandatory controls should operate as gates.

For example, an agent should not pass because its weighted score is high when: it performed an unauthorized action; a critical execution cannot be reconstructed; duplicate payments are possible; the agent cannot be disabled quickly; and no accountable process owner exists.

Why a convincing demo can still fail in production

The test set contains only ideal cases

The agent learns nothing about incomplete, contradictory, or malicious inputs.

The final answer is evaluated but the trace is ignored

Unsafe or incorrect tool use remains hidden.

The same team builds and grades the agent

Assumptions in the implementation are repeated in the evaluation.

Human review cost is excluded

The agent appears economical only because employee correction time is unmeasured.

The model changes without regression testing

A provider or configuration update changes behavior in previously stable scenarios.

Production permissions are broader than test permissions

The evaluation does not represent the deployed risk.

Failure criteria are negotiated after the test

The team redefines success to justify the investment already made.

The production decision in practice

An agent is ready for production when the organization has evidence that it can complete the task; choose valid actions; remain within policy; handle uncertainty; fail safely; preserve traceability; produce enough economic value; and be stopped and reversed.

A successful demo answers:

Can the agent work?

A production evaluation answers:

Under which conditions can the company trust it to act?

Fill System's AI & Process Automation consulting includes structured agent evaluation — test design, trace review, acceptance criteria, rollout planning, and production monitoring setup.

Request a free diagnostic to determine whether your AI agent is ready for production — or what evidence is still missing.