AI Agent Workflow Testing: A Terminal-State Release Gate

An agent opens the right record, finds the right policy, drafts the right document, and clicks through the right screens. The trace looks competent. Most subtasks receive credit. The job still ends in the wrong state.

That is the uncomfortable signal in two recent healthcare-agent benchmarks. HealthAdminBench reports a best full-task success rate of 36.3%, even though the strongest subtask result reaches 82.8%. CHI-Bench reports a best task-resolution rate of 28.0%, with no tested configuration above 20% on its stricter three-run reliability measure. These are simulated healthcare environments, not production deployments, and the numbers should not be generalized as a universal agent failure rate. But the pattern is useful far beyond healthcare: a sequence of locally plausible actions is not the same thing as a completed workflow.

For a small product team, the immediate change is simple: stop using step completion, a polished final message, or one successful demo as the release decision. Define the terminal state of the job, run the whole workflow repeatedly from a known start state, and require evidence that the intended state—not merely the intended conversation—was reached.

This field note turns that principle into a release gate. It is a proposed experiment protocol, not a report of testing we performed. You can adapt it to refunds, onboarding, support escalations, account changes, procurement, compliance review, or any other agent that reads policy and changes real state.

What the new benchmarks actually establish

HealthAdminBench evaluates computer-use agents across four simulated interfaces: an electronic health record, two payer portals, and a fax system. Its 135 expert-defined tasks cover prior authorization, appeals and denials, and durable-medical-equipment orders. The authors decompose those tasks into 1,698 verifiable subtasks. The best end-to-end configuration completes 36.3% of full tasks; a different configuration records the highest subtask score, 82.8%.

CHI-Bench stresses a different execution surface. It gives an agent a clinical case, a large managed-care handbook, and a simulator spanning healthcare applications and MCP tools. The agent has to move a case to a terminal status while producing role-specific artifacts. Across the reported harness and model configurations, the best resolves 28.0% of tasks. When reliability is measured across three trials, no configuration clears 20%; when the entire workload is run in one continuous session, reported performance falls to 3.8%.

These results establish four bounded facts:

The test environments are long-horizon, policy-rich, and multi-system.
Subtasks can score much better than complete workflows.
Harness and interaction design affect results, so this is not a model-only comparison.
Repeating a workflow can expose fragility hidden by a single pass.

They do not establish that every agent product succeeds 28% or 36.3% of the time, that healthcare automation is impossible, or that one named model has a universal reliability level. CHI-Bench is a new preprint with an associated open repository and dataset; HealthAdminBench is also a recent benchmark paper. Their simulations omit parts of live operations, and their task distributions are not your users’ distribution.

The transferable finding is narrower and more useful: if your metric rewards the path while your customer depends on the destination, your release gate is measuring the wrong object.

Three terms that prevent a misleading review

Teams often say “the agent completed the task” while referring to three different things. Separate them before writing an eval.

Subtask success means an intermediate requirement was satisfied: the record was found, a field was extracted, a policy was retrieved, or a draft was produced. Subtasks are valuable diagnostic signals. They tell you where the workflow breaks. They are not proof that the job finished.

Terminal-state success means the environment ends in an explicitly acceptable state. The database fields, external system status, user-visible artifact, audit entry, and follow-up queue all agree with the intended outcome. A workflow may have more than one valid terminal state—completed, correctly escalated, or safely declined—but “the agent stopped” is not one of them.

Repeated reliability asks whether the same scenario reaches an acceptable terminal state across multiple independent trials. The original τ-bench paper introduced pass^k to make this distinction visible: a system that occasionally succeeds can still be unsuitable when users need the same job to work consistently. The exact metric you choose can differ, but report single-run task success and repeated success separately.

One more distinction matters: a terminal-state check is not the same as forcing one canonical trajectory. There can be several correct paths. The evaluator should constrain outcomes, required evidence, and forbidden actions without requiring the agent to click buttons in one prewritten order.

A concrete small-team scenario

Imagine a SaaS company launches an agent that handles a customer request: “Cancel the annual upgrade I bought this morning and keep my original monthly plan.” The agent can read billing policy, inspect the account, call billing tools, update CRM notes, and email the customer.

A step-based demo might grade these items:

recognized cancellation intent;
found the correct customer;
retrieved the refund policy;
called a refund function;
drafted a confirmation email.

The trace can pass four of five while leaving a costly mess. The refund could be created against the wrong charge. The annual subscription could remain active. The monthly plan might not be restored. The CRM could say “resolved” while the billing provider says “pending.” The email could promise money that no system has returned. A retry might issue a second refund.

The terminal-state definition is different:

the targeted annual charge has exactly one refund with an allowed status;
the annual subscription is canceled and cannot renew;
the prior monthly plan is active with the correct renewal date;
the account has one audit record linking request, decision, tool calls, and resulting IDs;
the customer message describes the observed state, not an assumed future state;
if any required transition fails, the case is marked unresolved and routed to a human;
a rerun is idempotent and cannot duplicate money movement or customer messages.

This is the core Build Lab move: translate “handle cancellation” into observable state before evaluating a prompt, model, or agent framework.

Why high subtask scores can coexist with broken jobs

Long workflows do not fail like a quiz. Errors interact with state.

First, small misses compound. If ten required stages each succeed 95% of the time and failures were independent, all ten would succeed together only about 60% of the time. Real agent errors are not independent, so this arithmetic is an illustration, not a reliability forecast. One bad identity match can corrupt every later action; one stale policy choice can make an otherwise flawless execution invalid.

Second, some actions mutate the world. Reading a record and sending a refund are not equivalent steps. The SABER paper separates mutating from non-mutating actions and studies “decisive deviations”—early action-level divergences that flip the outcome. A harmless-looking wrong parameter at a mutating step may be more important than ten correct retrieval steps.

Third, handoffs create hidden contracts. One role produces an artifact that another role interprets. The first role may “complete” its work while omitting a field the next role requires. CHI-Bench deliberately includes multiple roles and role-specific artifacts because the handoff is part of the task, not administrative decoration.

Fourth, a polished response can disagree with backend state. Language models are optimized to produce coherent text. A confident “done” is therefore weak evidence. The evaluator must read the systems of record, not grade the tone of the final message.

Finally, retries change the problem. A second run is not simply another sample if the first run mutated state. Without reset or idempotency, a retry can convert a recoverable miss into a duplicate charge, duplicate ticket, conflicting approval, or repeated notification.

The terminal-state release card

Before you build fixtures, write one card for each workflow. This is the reusable artifact for the release review.

Field	What to record	Cancellation example
Workflow ID	Stable name and version	`billing.cancel-upgrade.v3`
User goal	Outcome in the user’s words	Cancel today’s annual upgrade; retain monthly plan
Known start state	IDs, statuses, balances, timestamps	Monthly plan replaced by annual charge at 09:14 UTC
Acceptable terminal states	Every valid completion or safe exit	Completed; escalated with no money movement; safely declined
Required assertions	State that must be true together	One refund, annual canceled, monthly active, audit linked
Forbidden states	Conditions that fail immediately	Duplicate refund; both plans active; “resolved” while pending
Mutating actions	Calls that need tighter checks or approval	Refund, cancel subscription, restore plan, send email
Evidence bundle	IDs and snapshots retained after the run	Charge, refund, subscription, audit, message IDs
Retry rule	Reset or idempotency behavior	Same request key; no second mutation or email
Human gate	Conditions that suspend automation	Identity mismatch, partial provider failure, policy ambiguity
Budget	Maximum tool calls, time, tokens, retries	20 calls, 5 minutes, one repair attempt
Version pins	Model, prompt, tools, policy, evaluator	Exact model ID and hashes for all mutable components

Write assertions as machine-checkable conditions where possible. “The CRM looks right” is weak. “Case C-1042 has status escalated, a non-empty reason, no refund ID, and one review-queue entry” is testable.

Do not force binary completion when the product supports legitimate safe exits. A correct escalation is often a valid terminal state; silent abandonment is not. A correct refusal can pass when policy prohibits the requested action; a generic refusal should fail when the agent had enough evidence to proceed.

The card is also a design review. If the team cannot name the system of record, valid final states, or retry behavior, the workflow is not ready for autonomous execution regardless of model quality.

Build fixtures from real workflow shapes

Start with five to twelve fixtures, not a hundred synthetic variations. Each fixture should represent a distinct state transition or failure boundary.

Use this mix:

Happy path: complete evidence, ordinary policy, all dependencies available.
Missing fact: the agent must ask for information or escalate rather than guess.
Conflicting systems: CRM, database, and provider disagree about current state.
Policy edge: the request sits just inside or outside an eligibility window.
Partial mutation: one system changes successfully and the next call fails.
Duplicate request: the same user intent arrives twice or a worker retries.
Unsafe request: identity, authorization, or scope is insufficient.
Stale artifact: a handoff document reflects an older version of the case.

Derive fixtures from production traces only after removing or transforming sensitive data and obtaining the necessary authorization. When production traces are unavailable, use domain-expert scenarios and label them as designed cases. Do not call them representative until real traffic supports that claim.

For every fixture, store the initial snapshot separately from the expected snapshot. Make reset deterministic. If an external sandbox cannot be reset, wrap it with unique test accounts and idempotency keys, then record cleanup as an assertion. A fixture that leaves residue can contaminate the next trial and create false conclusions.

AppWorld illustrates the engineering cost of controllable evaluation environments: its authors describe a multi-app simulated world, a large execution engine, hundreds of APIs, and an extensive test suite. Small teams do not need to recreate AppWorld. They do need enough control to know the start state, observe the final state, and rerun the case without hidden leftovers. Its open-source implementation is useful as a reference for how explicit app state and task evaluation can be structured.

Run the gate as a repeated, versioned experiment

One green run is a demo. A release gate needs repeated trials.

For each fixture, run at least three independent trials from the same known start state. Use fresh agent context for each trial unless persistent context is part of the product being tested. Record every version and budget. Do not silently give the candidate more retries, time, or tokens than the baseline.

A minimal experiment has two arms:

Current production: the model, prompt, tools, policy bundle, and orchestration already shipped.
Candidate: exactly one declared change where practical.

If several components must change together, name the bundle and accept that the experiment identifies the bundle’s effect, not the contribution of each component. Hold back at least a small set of fixtures from prompt and harness iteration. Otherwise, the team can tune to the review set and mistake memorization of cases for product improvement.

Calculate and report:

single-run terminal-state success;
three-run reliability per fixture;
forbidden-state rate;
safe-escalation rate;
duplicate-mutation rate under retry;
median and tail tool calls, latency, and cost;
recovery success after an injected partial failure.

Do not average away a forbidden state. A candidate with higher overall success but one duplicate-refund failure may be worse for release. Define hard blockers before looking at results.

The current τ-bench repository also provides a useful warning about versioning: a July 2026 grading update notes that affected banking results before and after a task fix are not comparable without careful pinning or rescoring. Your evaluator, fixtures, and expected states are product dependencies. Version them like code.

Score the state, then use the trace for diagnosis

The strongest evaluator is usually a deterministic assertion against an authoritative system. If the job is “create one approved refund,” query the refund and subscription records. If the job is “route this case for review,” query the queue, assignment, reason, and notification state.

Some outputs cannot be reduced to an exact database comparison. A customer message may need to be accurate, complete, and appropriately cautious while allowing multiple valid wordings. Use layered evaluation:

deterministic assertions for IDs, counts, statuses, permissions, and forbidden actions;
schema checks for required fields and evidence references;
rubric-based review for semantic qualities that cannot be deterministically encoded;
sampled human review for high-impact or uncertain cases.

If you use an LLM judge, do not let it infer the whole truth from the agent’s final message. Give it the scenario, relevant observed state, explicit rubric, and trace evidence. Calibrate it against human-reviewed cases and keep a disagreement sample.

Proxy State-Based Evaluation proposes an intermediate option for teams that cannot afford a fully deterministic backend: infer a structured proxy state from the complete interaction trace, then judge it against explicit scenario constraints. The paper reports promising agreement and ranking stability in its setting. The boundary is important: a proxy state remains inferred. Use it to accelerate evaluation where direct state access is unavailable, not to overrule authoritative records when they exist.

Once the outcome is scored, use the trajectory diagnostically. Mark the earliest decisive deviation, the last known good state, the first mutating action after divergence, and whether a guard could have stopped the cascade. This turns a red fixture into a repair plan instead of a vague “agent got confused” note.

Treat recovery as part of success

An agent that succeeds only when every dependency is healthy is not ready for a real workflow. Inject failures deliberately:

a tool times out after a successful mutation but before its response returns;
a policy service returns an old version;
an attachment upload succeeds while the case update fails;
a downstream system returns pending longer than expected;
the worker restarts between two mutating steps;
the user repeats the request after receiving no confirmation.

For each fault, define the expected safe terminal state. Sometimes that is completion after reconciliation. Sometimes it is an unresolved case with no additional mutation and a complete handoff to a human. “Agent produced an apology” is not recovery if the system remains inconsistent.

Store a recovery evidence bundle: original request ID, idempotency key, attempted mutations, provider responses, reconciliation query, final state, notifications, and human owner. The bundle should let a reviewer answer what changed, what did not change, and what must happen next without replaying hidden chain-of-thought.

This is where a small team often finds the highest-value product work. Better idempotency, explicit pending states, reconciliation jobs, and typed tool responses can improve reliability more than another prompt paragraph.

Set a release decision before seeing the score

Use a short, explicit decision table. Adjust thresholds to the risk of your product; these are example rules, not universal standards.

Signal	Ship	Limited canary	Block
Forbidden-state events	0	0	Any event
Duplicate mutation on retry	0	0	Any event
Three-run reliability	Meets team threshold on every critical fixture	Misses only low-impact fixtures with human gate	Misses any critical fixture
Safe escalation	Correct owner and evidence	Minor message issue, state correct	Dropped case or false “resolved”
Candidate vs production	No critical regression	Bounded improvement experiment	Critical regression or incomparable setup
Evidence completeness	All version and state IDs present	One noncritical observability gap	Cannot reconstruct outcome

The purpose is not to manufacture a green score. It is to make the trade visible before enthusiasm for a new model or prompt changes the standard.

For a canary, limit both users and actions. Start with read-only assistance or draft generation, then allow low-risk mutations, then higher-impact changes only after evidence accumulates. Keep the human gate operational; a canary that routes every edge case to an unattended queue is not safe because the automation percentage is small.

NIST’s AI RMF Core calls for documented test sets, metrics, tools, real-world monitoring, and post-deployment risk tracking. You do not need a compliance program to use that discipline. The release card, fixture versions, decision table, and production reconciliation metrics form a compact evidence trail a five-person team can maintain.

Watch the same terminal states after launch

Offline fixtures are necessary but incomplete. Production introduces user language, state combinations, latency, permissions, and dependencies the test set did not cover.

Monitor outcome signals, not just agent activity:

cases marked complete while a required external status is pending;
user-facing confirmations without matching provider IDs;
duplicate mutations sharing the same intent or idempotency key;
workflows abandoned without a valid terminal state;
escalations with no owner, reason, or evidence bundle;
retries following ambiguous tool responses;
drift by model, prompt, tool schema, policy, locale, and customer segment.

Sample successful cases as well as failures. Silent wrong-state outcomes may not generate errors. Reconcile system-of-record state against agent claims on a schedule appropriate to impact. Keep an emergency path to disable mutations while preserving read-only or drafting functionality.

When a production miss appears, add a sanitized fixture only if it represents a stable product requirement. Avoid turning every one-off anomaly into a brittle test. Record why the fixture exists, which incident or user need it represents, and who can retire it.

Where this protocol fits—and where it does not

This release gate fits agents that perform multi-step work across tools, policies, people, or mutable systems. It is especially useful when a locally correct action can still leave the overall job incomplete: customer support, finance operations, account administration, onboarding, procurement, security review, and internal approvals.

It is less useful for a single, reversible suggestion with no external state change. A writing assistant that produces a draft may need factuality, usefulness, and style evaluation more than terminal database assertions. A deterministic workflow that never calls a model should use ordinary integration and property tests. A research agent with no known correct endpoint may need evidence coverage and epistemic calibration rather than one expected final state.

The protocol also has limits:

Designed fixtures can miss real user variation.
A clean terminal state can hide a policy violation in the trajectory; keep forbidden-action checks.
A correct outcome can still deliver a harmful or confusing user experience.
LLM judges and proxy-state evaluators introduce their own model and prompt dependencies.
Three trials expose some inconsistency but do not estimate rare-event risk.
Healthcare benchmark results do not transfer numerically to another domain.
High-impact workflows may require legal, security, privacy, clinical, or compliance review beyond this technical gate.

The goal is not to prove an agent is “safe.” It is to replace one weak claim—“the demo worked”—with a bounded, repeatable claim: under these versions, fixtures, budgets, and trials, the system reached these acceptable terminal states without these forbidden outcomes.

The 48-hour change a small team can make

Do not begin by buying an evaluation platform or rebuilding a research benchmark.

On day one, choose one production workflow with at least one state-changing action. Write its terminal-state release card. Identify five fixtures: happy path, missing fact, policy edge, partial mutation, and duplicate request. Add direct assertions against the systems of record. Pin the current model, prompt, tool schema, policy bundle, and evaluator.

On day two, run the current production configuration three times per fixture from clean state. Do not tune while measuring the baseline. Mark terminal success, forbidden states, safe escalation, recovery, cost, and the earliest decisive deviation. Then decide which single change deserves a candidate run.

At the next release review, show the terminal-state matrix before the best transcript. If the team cannot reproduce the start state, observe the final state, or explain a retry, keep the agent in draft, read-only, or human-approved mode.

The practical lesson from the new healthcare benchmarks is not that agents are useless. It is that impressive local competence can coexist with unreliable delivery. Build the product around the job’s final state, and let the trace explain how you got there—not whether you arrived.

Your AI agent passed the step checks—and still failed the workflow