Your voice agent has more load than its CPU graph shows

A realtime voice agent can look idle one second before it fails.

Imagine a backend holding 40 connected calls. Most users are listening or thinking, so CPU sits at 35%. A launch dashboard says there is room. Then a product demo ends, users speak at roughly the same time, several background tool calls return, and one instance starts draining for a deploy. The system did not suddenly acquire 40 new customers. It merely had to honor work it had already promised.

That promise is the unit most load tests miss.

Google’s new engineering note on session-aware load balancing makes the operational distinction explicit: QPS measures arrivals, CPU and memory measure current pressure, and active sessions measure committed concurrency. OpenAI’s account of continuous interaction in GPT-Live shows why that commitment is getting harder to approximate. A session can listen and speak simultaneously, keep growing context, hand off between model instances, and wait on asynchronous reasoning or tools without ceasing to exist.

This is not an API benchmark of GPT-Live. OpenAI says the GPT-Live API is upcoming, while its existing GPT-Realtime model is generally available over WebRTC, WebSocket, and SIP. I did not run production traffic or a provider comparison for this article. The artifact below is a proposed failure drill that a small team can apply to its current realtime stack and rerun when a new model, transport, region, or routing policy ships.

The immediate change is simple: stop approving realtime capacity from request rate and average latency alone. Add a session ledger, measure setup and media separately from background work, force cleanup failures, rehearse drain and reconnect, and promote only when the system closes every promise it opens.

What changed: one conversation now contains several clocks

Turn-based voice products often resemble a chain of bounded jobs: record, transcribe, generate, synthesize, play. A continuous system has a different shape. OpenAI describes GPT-Live as a full-duplex model that decides many times per second whether to speak, listen, pause, interrupt, or invoke a tool. Deeper work can move to a background model while the live conversation continues.

That creates several clocks inside one user-visible call:

• the connection clock, from join intent through signaling, media readiness, disconnect, and final cleanup;
• the media clock, covering packet arrival, jitter, decode, playback, interruption, and silence;
• the conversation clock, including provisional transcripts, finalized turns, context growth, and compaction;
• the delegation clock, from tool or reasoning request through result, cancellation, and reconciliation;
• the infrastructure clock, covering routing, autoscaling, pod termination, deploys, and regional failure.

A healthy average can hide a broken clock. Fast model output does not repair a slow media setup. Smooth audio does not prove a cancelled tool stopped. A clean disconnect in the browser does not prove the server decremented its session counter. A low CPU sample during silence does not mean the instance can accept another full set of simultaneous speakers.

OpenAI’s engineering account says a long voice session can grow beyond a context limit while model instances also scale up and down. Its production design warms a replacement, prefills the current context, runs old and new paths in parallel, and cuts over after the replacement is ready. Treat that as a first-party architecture report, not a promise that every API or vendor provides the same handoff. The useful general lesson is that a live call may reserve future compute even while its current utilization is small.

Define capacity as a promise, not a snapshot

For a small team, realtime capacity has at least three layers:

1. Observed load: CPU, memory, network, model utilization, queue depth, and current tool work.
2. Committed load: sessions the system has accepted and must be ready to serve if users speak, interrupt, or receive background results.
3. Recovery load: temporary overlap created by reconnect, failover, handoff, context prefill, retry, or drain.

QPS is useful for the first layer. It says almost nothing about the duration or latent work of the second. Autoscaling based only on the first can also miss the third until recovery is already consuming the spare capacity intended to save the service.

Do not replace CPU with a session count. Google explicitly argues for a hybrid signal because sessions have unequal cost. A silent translation call, a noisy support call with frequent interruptions, and a tool-heavy troubleshooting call are three different workloads even if each contributes one active connection.

Use a calibrated pressure score instead:

committed_pressure =
    active_sessions
  + speaking_weight * speaking_sessions
  + delegation_weight * sessions_with_background_work
  + recovery_weight * sessions_handing_off_or_reconnecting

The weights are not universal constants. Start with deliberately conservative values, replay your own mix, compare the score with saturation and tail latency, and version the result. A scoring model that is not tied to traces is just a more complicated guess.

Start with one product-shaped scenario

Do not begin with thousands of identical synthetic sockets. Begin with a scenario whose failure would be recognizable to a user.

Suppose a small team is preparing a voice onboarding assistant. A normal call lasts four to eight minutes. The agent explains setup steps, listens while the user works, handles interruption, and delegates account-specific checks to a background service. If the check runs long, the agent should acknowledge the wait without inventing a result. If the connection drops, the user may reconnect once. If the service deploys, existing calls should drain or move according to a documented policy.

Create six session roles:

The mix matters more than raw volume. Run 12 to 30 sessions first, not because that proves production scale, but because it keeps every trace reviewable. After the state machine is correct, expand concurrency until you locate a real capacity knee.

Build a session ledger before a load generator

Every accepted call should create one authoritative record. A dashboard counter is derived state; the ledger is the evidence used to explain it.

realtime_session_receipt:
  drill_id: "voice-onboarding-v1"
  session_id: "sess-018"
  fixture_role: "tool-waiter"
  client_region: "ap-northeast"
  transport: "webrtc"
  provider_model: "pin-exact-model-or-service-version"
  admitted_at: null
  media_ready_at: null
  first_playable_audio_at: null
  last_media_at: null
  disconnect_observed_at: null
  cleanup_completed_at: null
  owner_instance: ""
  active_counter_delta:
    increment: 0
    decrement: 0
  media:
    jitter_ms_p95: null
    packets_lost_delta: null
    discarded_packets_delta: null
    playback_underruns: null
  conversation:
    interruptions: null
    interruption_cancelled_output: null
    provisional_turns_finalized: null
  delegation:
    started: 0
    completed: 0
    cancelled: 0
    orphaned: 0
  recovery:
    reconnect_attempts: 0
    duplicate_live_sessions: 0
    handoff_overlap_ms: null
  end_reason: ""
  cleanup_status: "pending"

Record timestamps on a shared monotonic basis where possible. Preserve raw events as well as aggregates. A p95 graph cannot tell you whether the one dropped call also left a tool running and a counter inflated.

The browser can supply transport evidence through the W3C WebRTC Statistics API, including received packets, lost packets, jitter, discarded packets, and round-trip time when available. These values require care: packetsLost can behave unexpectedly because it is an estimate, field availability differs, and network metrics may reveal sensitive location or speaking patterns. Collect the minimum needed, document retention, and avoid turning observability into a new privacy leak.

Measure five SLO families, not one latency number

A realtime release packet should keep five families separate.

1. Admission. Measure join intent to accepted session, accepted session to media readiness, rejection correctness when capacity is closed, and the percentage of sessions assigned to an already over-budget backend.

2. Media. Measure time to first playable audio, jitter, packet loss, discarded late packets, playback underruns, and gaps during handoff. Do not substitute server first-byte time for what the user can hear.

3. Interaction. Measure interruption acknowledgement, time until old output stops, transcript finalization delay, false end-of-turn behavior, and whether overlapping speech produces a coherent record. OpenAI’s current description distinguishes a speculative conversation view from an authoritative finalized transcript; that is a useful warning against treating partial text as settled audit data.

4. Delegation. Measure time to acknowledgement, tool completion, cancellation completion, orphaned work, stale results delivered after cancellation, and whether the live path continues while background work is pending.

5. Lifecycle. Measure active-session counter error, cleanup delay, duplicate live sessions after reconnect, drain success, forced termination, state-loss rate, and recovery overlap.

Set thresholds from your product baseline and risk, not from this article. A language-practice companion may tolerate a reconnect that a live payment-confirmation call cannot. A support assistant may allow a long tool wait if it speaks honestly; a simultaneous interpreter may have a far smaller media-gap budget.

The release decision should still require zero invariant violations for events that create uncontrolled state: no negative counters, no double billing, no two active owners for one non-duplicable session, no completed disconnect with orphaned privileged work, and no admission after the backend has declared itself draining.

Run a 30-session staircase with deliberate silence

The first proposed drill uses three 10-session steps. Keep model, region, client build, transport, tool fixture, and routing revision fixed.

Step A: quiet commitment. Admit ten sessions. Keep eight silent, one speaking, and one waiting on a tool. Hold for three minutes. Verify that routing sees ten commitments even while CPU remains modest.

Step B: synchronized activation. Add ten more sessions. Then trigger speech in all previously silent sessions within a five-second window. This is not a natural traffic forecast; it is a safe way to test the promise represented by silence. Capture CPU and memory response, media tail latency, rejected admissions, overloaded assignments, and recovery time.

Step C: lifecycle pressure. Add the final ten-session mixed group. Force four client disconnects, reconnect two, cancel two tool calls, and mark one backend draining. Confirm that new sessions stop landing there while existing sessions follow the stated drain policy.

Repeat the staircase at least three times before inferring a trend. Randomize which backend receives the disruptive events. Keep every failed run; discarded attempts are part of the denominator.

Google’s article recommends varying concurrent sessions, duration, arrival pattern, idle-to-speaking ratio, cancellation and disconnect rate, backend count, and maximum sessions per backend. It also recommends tracking distribution, overloaded assignments, startup tails, time to first stream, dropped sessions, and counter behavior after forced disconnects. Those dimensions are more useful than a fire-and-forget request benchmark for this workload.

Inject the failures that dashboards usually erase

Add one fault at a time before combining them:

WebRTC already has a transport-level concept of ongoing consent. RFC 7675 requires consent to be renewed and says a new session or ICE restart is needed after consent is lost on a candidate pair. That does not complete your application cleanup. It only gives the application a signal it must convert into one idempotent lifecycle transition.

The same distinction applies during deploys. Kubernetes exposes terminating endpoints and supports graceful shutdown, but its Pod lifecycle documentation notes that applications may need explicit session draining and completion. Google Cloud’s connection-draining documentation further warns that another backend has no record of an existing TCP connection and may reset it. Infrastructure can stop new routing and buy time; it cannot invent missing application state.

Make counter accuracy a release invariant

Google’s example increments an active-session counter when a stream begins and decrements it in a finally block. The important idea is not the programming language. It is that every exit path converges on one terminal transition.

Track three independent quantities during the drill:

ledger_open_sessions
runtime_active_sessions
routing_reported_sessions

They will not be identical at every microsecond. They must converge within a defined observation window. Alert on both direction and age:

• runtime higher than ledger suggests leaked or delayed cleanup;
• runtime lower than ledger suggests false capacity or missing ownership;
• routing lower than runtime can overload the backend;
• routing higher than runtime can strand capacity;
• any value without a recent timestamp is unknown, not zero.

Do not “fix” drift by periodically overwriting one counter from another without preserving the discrepancy. Reconciliation must emit a receipt naming the source of truth, affected sessions, correction, and suspected cause. Otherwise the system becomes numerically tidy while the bug remains unauditable.

At high concurrency, also test tracker overhead and contention. A correct global atomic counter can become a hot memory location; a sharded or aggregated design may reduce contention but increase reporting lag. The acceptable trade depends on how often routing reads the signal and how quickly your traffic can spike.

Promotion requires a signed session-capacity receipt

Do not promote because “30 calls worked.” Promote the exact configuration and evidence boundary.

session_capacity_decision:
  fixture_version: "voice-onboarding-v1"
  client_revision: "pin"
  server_revision: "pin"
  routing_revision: "pin"
  provider_model: "pin"
  region: "pin"
  completed_runs: 0
  discarded_runs: 0
  max_tested_concurrency: null
  thresholds:
    media_ready_p95_ms: "team-defined"
    first_playable_audio_p95_ms: "team-defined"
    cleanup_convergence_ms: "team-defined"
    overload_assignment_rate: "team-defined"
  invariants:
    negative_counter: "must-be-zero"
    duplicate_owner: "must-be-zero"
    orphaned_privileged_work: "must-be-zero"
    admission_during_drain: "must-be-zero"
  untested:
    - "multi-region failover unless actually run"
    - "provider outage unless actually run"
    - "GPT-Live API behavior before public availability"
  decision: "hold-until-filled"
  owner: ""
  review_at: ""

Use four outcomes: hold, limited launch, promote, or rollback. A limited launch must name the allowed region, concurrency, session duration, feature set, fallback, and on-call response. “Beta” is not a control unless it changes exposure.

Retest when the model alias, media path, tool policy, context strategy, autoscaler, load balancer, maximum session duration, or deploy process changes. The fact that a browser still connects does not mean the capacity contract stayed the same.

Common failure modes

Silent sessions count as free. They are cheap now but carry an obligation to resume. Under-counting them spends recovery headroom before the spike arrives.

Every session gets equal weight. This hides speaking, delegation, and recovery differences. Use role-aware pressure only after the roles are visible in traces.

The counter decrements twice. Timeout, cancellation, disconnect, and process cleanup race toward the same session. Make the terminal transition idempotent and test the race.

Reconnect creates a second truth. A new socket may be valid while the old server owner still believes it is authoritative. Define session identity, replacement rules, billing, and stale-result handling.

Draining means “the load balancer will handle it.” The network can stop new assignments, but application state, pending tools, and user messaging still need owners.

Only server latency is measured. Users hear jitter, late packets, buffer underruns, and output that does not stop when interrupted. Client evidence belongs in the same receipt.

Averages authorize launch. Realtime failures live in tails and races. Preserve per-session traces for the worst setup, first-audio, cleanup, and recovery cases.

Vendor architecture becomes your guarantee. OpenAI’s relay/transceiver and stateful handoff are valuable first-party evidence about one production design. They do not prove that your provider, account tier, region, or application inherits the same behavior.

Where this drill applies—and where it does not

Use this protocol when a product maintains a bidirectional, stateful stream and users experience setup, interruption, background work, reconnect, or handoff: voice assistants, live translation, realtime tutoring, interactive support, multiplayer AI characters, or world-model interfaces.

It is too heavy for offline transcription, batch speech generation, or one-shot audio files. Those jobs still need latency, quality, rights, and retry tests, but an active-session control plane may add complexity without reducing risk. YBlog’s earlier audio production fixture is the better starting point for generated clips and non-realtime speech.

This drill also does not choose an infrastructure architecture. OpenAI reports that its point-to-point workload uses a stateless relay plus stateful transceiver, keeping ICE, DTLS, SRTP, and session lifecycle in one owner (engineering detail). A multiparty product may need an SFU; a server-to-server product may choose another transport. The test cares about explicit ownership and observable transitions, not copying a diagram.

Privacy and safety remain separate gates. Voice traces can contain sensitive speech, partial transcripts, network metadata, tool results, and inferred behavior. Minimize collection, control access, define retention and deletion, and test consent and escalation flows. A capacity pass does not authorize recording, training use, voice cloning, or high-risk automated action.

Evidence boundaries and unresolved questions

The current evidence is strong on architecture and test dimensions but incomplete on portability.

Confirmed first-party facts include OpenAI’s full-duplex GPT-Live design, background delegation, stateful handoff account, and relay/transceiver design; Google’s active-session tracking and hybrid-balancing proposal; WebRTC’s standardized client metrics and consent-freshness mechanisms; and Kubernetes/Google Cloud termination and draining behavior.

Vendor claims include product preference results, scale, and internal latency improvements. They should not become your expected SLO without a matched run. OpenAI reports that moving a media/inference frontend from Python asyncio to Go made the new p95 frame-delivery smoothness match the previous p50; the workload, measurement, and system are OpenAI’s, not a universal language benchmark.

Unknowns for a given team include provider-side queueing, exact session migration behavior, regional routing, account limits, model-alias changes, audio retention, outage recovery, and the upcoming GPT-Live API’s final surface. Mark those unknown in the receipt. Do not fill them with ChatGPT product behavior or an older Realtime API assumption.

The proposed 30-session staircase has not been run for this article. Its numbers are fixture sizes, not observed limits. A team should lower them if the system or budget is small, or raise them only after traces and cleanup remain reviewable.

A 48-hour shipping plan

Hours 0–4: define the contract. Draw the session state machine. Name the owner of media, conversation state, background work, and cleanup. Choose one product-shaped scenario and six roles. Set retention and access for traces.

Hours 4–12: instrument the ledger. Add monotonic lifecycle timestamps, counter deltas, owner identity, client WebRTC stats, delegation status, reconnect identity, and one terminal reason. Compare ledger, runtime, and routing counts.

Hours 12–20: run the quiet step. Admit ten mixed sessions, keep most silent, and verify that routing sees committed load. Review every receipt before adding concurrency.

Hours 20–30: activate and disrupt. Trigger synchronized speech, forced disconnects, cancellation races, one reconnect, metric staleness, and one draining backend. Keep failed and discarded runs.

Hours 30–38: repeat. Run at least two more staircases. Inspect the worst per-session traces, not only dashboards. Calibrate weights and thresholds from the observed knee and user impact.

Hours 38–44: reconcile. Prove that ledger, runtime, and routing counts converge. Search for orphaned tool work, duplicate owners, negative counters, stale results, and missing terminal reasons.

Hours 44–48: sign or hold. Complete the capacity receipt. If an invariant fails, hold and name the smallest next experiment. If the bounded fixture passes, launch only inside the tested region, concurrency, duration, and feature envelope, with a rollback trigger.

A realtime agent is not healthy because it answers one request quickly. It is healthy when every accepted session remains observable through silence, speech, interruption, background work, disconnect, drain, and cleanup. The shipping question is not “How many requests can we start?” It is “How many promises can we keep—and can we prove that every finished promise actually closed?”

References

1. OpenAI — How we built a realtime system for responsive voice AI in six months
2. OpenAI — Introducing GPT-Live
3. OpenAI — How OpenAI delivers low-latency voice AI at scale
4. OpenAI API — GPT-Realtime model
5. Google Developers Blog — Scaling realtime AI agents with session-aware load balancing
6. W3C — Identifiers for WebRTC’s Statistics API
7. IETF RFC 7675 — STUN Usage for Consent Freshness
8. IETF RFC 8827 — WebRTC Security Architecture
9. Kubernetes — Pod lifecycle and termination flow
10. Kubernetes — Explore termination behavior for Pods and endpoints
11. Google Cloud — Connection draining
12. Google Cloud — Backend services, session affinity, and timeouts