Your Voice Agent Ended the Call. Did It Resolve the Problem?

A shorter support call can be a useful improvement. It can also be the first half of a longer problem: the caller hangs up, checks their account, finds nothing changed, then opens a ticket or calls again. A dashboard can accurately report lower call duration while the product creates more work.

Vapi’s October 26 explanation of traffic splitting makes this a timely measurement question. Its version controls let builders expose a limited share of calls to a changed assistant. The product decision is what to count after that exposure. If the proposed benefit is resolution, the observation should continue beyond the call that received the new prompt.

This article proposes an experiment for a small support product. It is not a test we ran: no Vapi account, customer calls, routing configuration, or paid model was used. The scenario, follow-up window, rehearsal cases, and worksheet are proposed design choices. Their purpose is to make a particular failure observable, not to claim that any voice provider improves support outcomes.

1. Separate a controlled rollout from evidence of improvement

Vapi’s October 26 product interview describes gradually exposing calls to assistant versions and discusses outcome-based automation as a future direction. That is a vendor account of the feature and early usage, not an independently measured improvement. The October 12 changelog already lists traffic splitting and version-aware logs; October 26 should not be described as the feature’s first availability.

A canary limits exposure while a team watches for operational harm. An outcome experiment asks whether an assigned experience changes a defined result. The same routing feature can support both, but ten uneventful calls do not establish a resolution benefit. A prompt can avoid crashes and still misunderstand a return policy.

For this experiment, first use a canary to check that calls reach the intended versions and existing safeguards work. Then start a separately recorded comparison with stable allocation and enough independent callers for the intended decision. Do not combine changing canary percentages into a single convenient result.

The decision to write down is narrow: should one closing-prompt change replace the current prompt for eligible order-status callers? Keep the voice, model, tools, business policy, and human escalation access the same. This gives the team an interpretable intervention rather than an unexplained bundle of improvements.

2. Start with an issue whose outcome exists outside the transcript

Imagine a small subscription business whose assistant handles delivery-status questions. Version A reads the available status and ends with an open invitation to ask more. Version B checks whether the status actually answers the caller’s question and, when it does not, offers the same existing human-support route. Both versions have identical authority and access to order data.

The hypothesis is that B increases verified resolution of the caller’s first eligible issue without increasing harmful promises or total support effort. This is an illustrative business, not a customer case. The experiment does not require the agent to issue refunds, change addresses, or invent delivery dates.

Before enrollment, define a resolution rule for each supported issue type. For “where is my parcel?”, an accurate status may resolve the issue when the caller only wants tracking information. “The carrier says delivered but I have nothing” is a different issue and may require a support case. A correct tracking sentence alone does not resolve both.

Use a frozen rubric grounded in the business’s actual service policy, order records, and permitted next steps. A reviewer should be able to explain which evidence satisfies the rule without knowing the assistant version. If the business cannot agree on that rule, pause the comparison. Routing more calls will not settle an undefined product promise.

3. Choose the caller before counting their calls

Vapi’s routing documentation says percentages apply to new calls; an ongoing call keeps its version. Repeat-caller stickiness is best effort, based on phone number or SIP username. Web calls and other calls without those usable identifiers can be assigned independently. Target order also affects stickiness. These are important constraints on any longitudinal comparison.

The proposed primary cohort therefore starts with callers whose routing identity and authorized support identity can be linked reliably. Define eligibility using information available before the changed prompt, such as the existing authenticated order-selection step. Use the first eligible issue per caller during enrollment; do not enroll that caller again in the same experiment. This avoids treating every attempt to solve one problem as a new independent customer.

A phone number is not inherently a person or an account. Shared household numbers, business switchboards, withheld numbers, and multiple devices need an explicit eligibility rule. Keep known ambiguous identities in a separate feasibility report. Do not remove a caller after seeing a bad outcome merely because their identity became inconvenient.

Microsoft’s pre-experiment guidance emphasizes stable identifiers, interference, and power at the randomization-unit level. Here, the first observed assigned version defines the caller’s analysis group. Record subsequent crossovers, but retain that caller in their original group. This estimates the deployed assignment policy, including imperfect stickiness; it does not isolate the effect of uninterrupted exposure to one prompt.

4. Keep one fixed follow-up window across channels

For a rehearsal, use a 72-hour window beginning at the first eligible call’s start. This is an author-selected example, not an industry standard. A business with weekend fulfillment or slower ticket responses may need a longer window. Choose the duration from the service cycle before collecting comparison outcomes.

An index issue is the first eligible support problem that starts the window. A repeat contact is another contact about that same problem, whether it arrives by phone, chat, email, or a reopened ticket. A new question about a different order is not automatically a repeat. Define matching rules before reviewing transcripts: account key, order key, issue family, and a human adjudication route for ambiguity.

The initial call ending is not the start of a new clock. Nor does a follow-up call restart the window. Otherwise a version that generates more repeat calls can receive systematically different observation time. Stop enrollment on a fixed date, then wait for the final enrolled caller’s full window to mature before the final comparison.

Microsoft’s cross-product experimentation account explains why outcomes in a second system must be connected to the original assignment. Apply that principle to support channels: preserve the enrolled caller even when no later contact exists. Separately record whether those channels were observable. Absence of a matched event is informative only when the underlying capture and linkage worked.

5. Record resolution, recontact, and uncertainty separately

Use three separate fields. First, did the frozen rubric find a verified resolution? Second, did the same issue produce an unplanned repeat contact within the window? Third, was follow-up coverage complete? A caller can have no observed repeat and still lack evidence of resolution. They may have given up or used an unobserved channel.

For the primary result, count durable verified resolution only when the issue satisfies its resolution rule, has no contradictory same-issue reopening or unplanned repeat contact in the window, and has complete required follow-up evidence. If any required element is unknown, count it as non-success in the primary denominator and show the unknown count separately. This is a conservative operational measure, not proof that all non-successes were dissatisfied.

Record planned follow-ups separately from unplanned retries. A correctly arranged human callback may be the intended resolution pathway for an unsupported issue, but it is not automated resolution. Keep both assisted and automated outcomes visible. Do not improve “deflection” by removing the option to reach a person.

Vapi structured outputs extract information from conversations using AI analysis after calls. They can suggest an issue family or claimed outcome for review. A schema-valid resolved: true is still an extracted claim. Verify against the appropriate business record and rubric; keep the extractor and rubric identical across versions. Audit a sample of positive labels as well as uncertain or failed cases so an optimistic extractor cannot silently define success.

6. Build an issue ledger beside the call dashboard

The Traffic tab compares calls by version, including duration, cost, latency, ended reasons, and configured versus actual shares. It also explains how fixed windows can span older splits, and that calls without a version are absent from its views. That dashboard is useful operational evidence. An issue cohort needs a second view with a different denominator.

Create one enrollment row per eligible caller and link all relevant call and support events to it. Keep the first version, the allocation epoch, and subsequent exposures separate. An epoch is a period with an unchanged split, target order, and experiment configuration. A new prompt, reordered targets, or changed allocation ends it.

The following is a data contract to adapt, not Vapi API fields or executable configuration:

experiment: delivery-closing-prompt
cohort_key: pseudonymous-authorized-caller-key
index_issue: pseudonymous-order-key-plus-issue-family
eligibility_rule_revision: required-before-start
first_call_id: required
first_assigned_version: required
allocation_epoch: required
window_started_at: required
window_ends_at: required
exposures: [] # call IDs, actual versions, times
support_events: [] # channel, issue match, planned/unplanned
resolution_rule_revision: required
verified_resolution: unknown # yes / no / unknown
unplanned_same_issue_recontact: unknown
followup_coverage: unknown # complete / incomplete
crossed_version: unknown
durable_verified_resolution: unknown
total_support_cost: unknown
adjudication: pending

Vapi’s call-log reference documents call IDs, version information, ended reasons, artifacts, and tool-call inspection, with availability depending on configuration. Confirm those records exist in your account before enrollment. Do not treat this worksheet as a promise that one export already joins every channel.

Use pseudonymous linkage keys, minimum necessary outcome fields, restricted access, and the team’s existing retention policy. A hashed phone number remains linkable data. This experiment provides no reason to retain extra recordings or combine customer identities without an authorized purpose.

7. Rehearse the joins before exposing customers

Run a synthetic rehearsal using staff-controlled identities and fictional orders. Start with twelve cases: an accurate answer, an incorrect answer, a normal transfer, an abandoned call, an email repeat, a phone repeat, a planned callback, an unrelated new issue, an unknown identity, a missing version, a late support event, and a cross-version repeat. These are coverage cases, not twelve observations that establish performance.

Write the expected ledger state before generating each event. The missing-version case should enter the intake audit and block attribution; it must not disappear because the version dashboard omits it. The late-event case should demonstrate event-time handling: a contact inside the window remains relevant even if its log arrives afterward. Choose an ingestion grace period before the final freeze.

Next, consider an A/A rehearsal with two otherwise identical published configurations and staff-controlled traffic. Its job is to expose broken version attribution, missing joins, inconsistent extraction, or asymmetric logging. It does not prove the vendor’s randomization algorithm, population balance, or production follow-up coverage. Preserve the before-and-after records so someone other than the implementer can recompute the flags.

If the current system cannot link the intended channels or identify assignment at the first eligible exposure, stop at feasibility. A per-call operational comparison remains possible, with its narrower claim. Do not label it an experiment in durable resolution. This stopping rule prevents a small team from accumulating an attractive dashboard around an outcome it cannot observe.

8. Audit allocation without confusing calls and callers

Sample ratio mismatch, or SRM, means the observed allocation differs from the expected allocation beyond what the chosen randomization model explains. Microsoft’s SRM diagnosis guidance treats it as a data-quality symptom requiring investigation, not a reason to reinterpret a favorable outcome.

Keep two audits. The operational audit reconciles all incoming call records against versioned calls, unversioned calls, eligibility decisions, and failed intake. The experiment audit checks unique newly enrolled callers within each fixed epoch, not all calls pooled together. Returning callers produce correlated calls; if treatment creates more retries, a skewed call share can itself be an outcome rather than proof that assignment was broken.

Before claiming a causal difference, verify how first assignments are generated for your eligible population and whether their expected allocation supports the planned SRM test. The public sticky-routing description does not provide a complete randomization proof. If that mechanism cannot be established, label the work a monitored observational pilot and avoid causal language. Do not calculate a binomial p-value on repeated calls simply because the dashboard shows a configured percentage.

Use pre-exposure channel and issue strata to inspect unexpected imbalance or missingness. Record unknown assignments separately; they cannot be put into whichever arm makes totals look balanced. Unresolved assignment loss, treatment-dependent linkage failure, or unexplained cohort imbalance blocks the benefit claim. Fix instrumentation and start a new recorded cohort rather than deleting awkward rows.

9. Read the result with the original denominator

Let N be all enrolled callers assigned to an arm whose observation windows have matured. The durable-resolution rate is the number meeting every primary success condition divided by N. Failed calls, crossovers, transfers, and missing follow-up evidence stay in N. Report their counts so a conservative result does not hide an observability problem.

An illustrative arithmetic example: A has 100 enrolled callers, 70 durable successes, and 5 unknown outcomes. B has 100 callers, 74 successes, and 15 unknown outcomes. Primary rates are 70% and 74%; removing unknowns would misleadingly change both denominators and greatly improve B’s apparent result. These numbers are fictional. A four-point difference in such a small sample is not sufficient evidence of improvement on its own.

Report the absolute difference and an uncertainty interval calculated at the actual independent assignment unit. If identities share routing or several callers belong to one account, the design may require clustered assignment and analysis; do not assume 200 rows are 200 independent people. Define the smallest worthwhile improvement, baseline assumption, power, and analysis method before starting. Get statistical help if those assumptions do not fit the routing mechanism.

A tiny canary can detect a conspicuous failure while remaining unable to measure a modest benefit. Low-volume teams can stop after a useful feasibility finding, or run a longer preregistered comparison. The alternative is not to declare a winner from a favorable day. Safety monitoring can stop exposure at any time; benefit decisions follow the predeclared analysis plan, not repeated significance checks.

10. Count work that leaves the phone channel

A useful companion measure is total support effort per enrolled caller during the fixed window. Include the index call, repeats, human callbacks, ticket handling, and corrections. Also show cost per durable resolution, with the full arm’s support cost in the numerator and durable successes in the denominator. If there are no successes or cost data are incomplete, report that limitation instead of a fabricated efficiency number.

Use a declared costing convention for human work. Scheduled staffing cost and incremental handling time are different quantities; keep the same convention across arms. Separate telephony and model charges from staff minutes when a trustworthy common monetary conversion is unavailable. A lower voice bill can coexist with more manual work.

Microsoft’s research on metric-interpretation pitfalls describes why favorable local metrics and changing metric populations can mislead product decisions. Here the original enrollment cohort anchors the comparison even when the work moves elsewhere. Duration, turn latency, and transfer rates explain a result; none substitutes for the chosen resolution outcome.

11. Make the promotion decision explicit

Complete this worksheet before starting. Replace every threshold with a value justified for the business; blank thresholds mean the protocol is not ready.

Choose among four outcomes: repair instrumentation; retain A; extend the comparison under its predeclared plan; or promote B for the tested population. If assignment remains unverified, the decision may be an operational rollout based on bounded evidence, but the report must not claim a randomized causal win.

After stopping enrollment or rolling back, keep collecting the already-enrolled cohort’s follow-up evidence. A rollback changes future exposure, not the historical denominator. Record emergency changes and their timing; if they change the intervention, end the epoch. Never preserve experimental purity at the expense of a caller needing help.

12. Know what this experiment cannot tell you

This design fits a repeatable, low-risk support issue with an independent outcome rule and observable follow-up channels. It is unsuitable when shared identities cannot be disentangled, outcomes take months, or the business cannot observe where unresolved work goes. It is not a substitute for domain-specific review of medical, financial, emergency, or other consequential calls.

The 72-hour example cannot establish long-term satisfaction, retention, or the absence of later recurrence. Even complete logging within named channels does not reveal every silent abandonment. A deployment-policy comparison includes real crossover behavior; it cannot automatically attribute a difference to prompt wording alone. No source here independently validates Vapi’s feature effectiveness, and no experiment results are claimed.

The practical first step is small: choose one issue family and see whether yesterday’s call can be connected to its real outcome without asking the assistant to grade itself. If that connection fails, repair it before increasing exposure. If it works, traffic splitting becomes a way to learn whether the change resolved a problem rather than merely ended a conversation.

References

1. Vapi: Traffic splitting is live, October 26, 2026 — product interview; vendor account, not outcome validation.
2. Vapi: What’s New, week of October 12, 2026 — earlier availability record.
3. Vapi: Traffic splitting — routing, identity, target order, and version pinning.
4. Vapi: Compare versions on the Traffic tab — metrics, windows, and unversioned-call boundary.
5. Vapi: Structured outputs — AI extraction from conversations.
6. Vapi: Call logs — available call records and inspection surfaces.
7. Microsoft Research: Patterns of Trustworthy Experimentation, pre-experiment stage — hypotheses, identity, and randomization-unit design.
8. Microsoft Research: A/B Testing Across Products — linking downstream outcomes to original assignment.
9. Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing — assignment and data-quality diagnosis.
10. Dmitriev et al.: A Dirty Dozen, KDD 2017 — original research on misleading metric interpretations.