Voice Agents and the 300ms Budget
A latency-first architecture for phone-grade voice: streaming STT, partial LLM, and TTS interleave — with real numbers.
Voice Agents and the 300ms Budget
A latency-first architecture for phone-grade voice: streaming STT, partial LLM, and TTS interleave — with real numbers.
Humans reply to each other in conversation with a gap of roughly 200–300 milliseconds. That number is wired deep — it holds across languages and cultures — and it's the number your voice agent is judged against on every single turn. Under ~500ms feels human. Around 800ms feels like talking to a machine that's thinking. Past 1.5 seconds, callers start saying "hello?" — and every "hello?" mid-generation is a turn-taking failure you now have to recover from.
Text chat forgives latency; a spinner buys you seconds. Voice forgives nothing. So when we build phone-grade voice agents, we don't start with the prompt or the personality. We start with a latency budget, and every architectural decision gets charged against it.
Where the time actually goes
Here's the anatomy of a turn in a conventional stitched pipeline, with 2026 numbers:
| Stage | Typical range |
|---|---|
| Endpointing (detecting the caller stopped) | 100–300ms |
| Streaming STT finalization | 100–300ms |
| LLM time-to-first-token | 350–1,000ms |
| TTS time-to-first-audio | 90–200ms |
| Network hops between vendors + telephony | 50–200ms |
| Total (voice-to-voice) | ~700ms–2.0s |
Run each stage sequentially — wait for full transcript, wait for full LLM response, then synthesize — and you land at 1.5–2 seconds on a good day. That's a demo, not a product. Industry benchmarks in 2026 put standard sequential STT→LLM→TTS pipelines at 800–1,500ms, while well-engineered streaming pipelines and speech-to-speech models get voice-to-voice into the 400–800ms range, with the best speech-to-speech systems touching 160–400ms.
The gap between 1,500ms and 500ms isn't a faster model. It's architecture: never let any stage wait for the previous one to finish.
The interleaved pipeline
The core principle: every stage streams, and every stage starts on partial input from the stage before it.
Streaming STT with aggressive partials. Modern streaming STT (Deepgram's Nova-3 class engines run ~150ms streaming latency and hold up on noisy telephone audio) emits interim transcripts while the caller is still talking. Don't wait for the final transcript — feed stable partials to the LLM speculatively. If the final transcript matches the partial (it usually does), you've already banked 200–400ms of LLM thinking time. If it doesn't, discard and re-run; the occasional wasted call is cheap against the latency win on every other turn.
Smart endpointing over fixed silence timers. A fixed 700ms silence threshold donates 700ms of your budget to dead air on every turn. Semantic endpointing — small models that judge from prosody and content whether the caller is done or just pausing — cuts endpoint decisions to 100–200ms and, just as importantly, stops the agent from barging in on a caller who paused mid-thought to find an account number.
First sentence fast, think later. Structure generation so the opening clause never depends on slow work. The agent can say "Sure, let me pull that up—" the moment intent is clear, while the actual lookup runs behind it. This isn't a filler-word hack; it's honest turn-taking, the same thing a human rep does. Recent work on speculative tool-calling formalizes the same idea: kick off likely tool calls asynchronously during generation instead of serializing them after it.
Sentence-chunked TTS interleave. Never synthesize the full response. Stream LLM tokens out, cut at the first sentence boundary, ship it to TTS immediately, and keep synthesizing chunk-by-chunk while the first sentence is already playing. With current fast TTS (Cartesia's Sonic class delivers first audio in 40–60ms), the caller hears voice while the model is still writing sentence three. This one change routinely converts a 1,200ms perceived latency into 500ms — the model's total generation time barely changed, but the caller stopped experiencing it.
Add it up: ~150ms endpointing, LLM already warm on speculative partials so ~250–400ms to first sentence, ~50–90ms to first audio, plus network. That's a P50 in the 450–650ms range on a stitched pipeline — phone-grade, with full control over your models and tools.
What about speech-to-speech models?
The 2026 fork in the road. Native speech-to-speech models (the GPT-4o Realtime lineage, Gemini Live) skip transcription entirely and hit 300–800ms voice-to-voice, sometimes lower, with better prosody and emotion handling than any stitched pipeline — the model hears frustration instead of reading a transcript of it.
The trade-offs keep them from being our default for business-critical phone work: you give up the ability to swap components independently, you get less deterministic control over exactly what the agent says (a compliance problem in regulated verticals), tool-calling mid-conversation is still clunkier than in text-first models, and per-minute costs run higher. Our current rule: speech-to-speech for experiences where emotional naturalness is the product; stitched-but-streaming for workflows where control, auditability, and cost matter — which is most enterprise phone traffic. Revisit every quarter, because this line is moving fast.
The parts nobody budgets for
Telephony transport. Your caller is on a phone network you don't control, adding 50–200ms before your stack sees a single audio frame. If you're on WebRTC, an SFU routing Opus frames adds sub-50ms; a badly placed media server adds 150ms. Deploy inference in the same region as your telephony provider — we've seen 200ms round-trips die of nothing but geography.
Tail latency, not average. A 500ms P50 with a 2.5s P95 is a bad agent, because the P95 turns are exactly the complex, tool-heavy moments when the caller is already stressed. Budget the P95, alert on it per pipeline stage, and design explicit "still checking that for you" behavior for turns that will blow the budget — announced slowness reads as diligence; silent slowness reads as broken.
Interruptions. Callers barge in constantly. The pipeline must kill TTS playback within ~100ms of detected speech, flush queued audio, and — the part everyone forgets — track what was actually heard versus generated, so the conversation state doesn't contain claims the caller never received.
Testing voice like you mean it
One more thing that separates voice demos from voice products: latency evals. Text agents get judged on correctness; voice agents get judged on correctness and timing, so your eval suite needs both dimensions. In practice that means synthetic call campaigns — scripted callers with realistic audio conditions (background noise, accents, mid-sentence pauses, interruptions) run against the full production stack, not just the LLM — with per-stage latency recorded on every turn. The 2026 voice-eval tooling ecosystem (Coval, Hamming, Cekura and peers) has made this dramatically easier than a year ago; there's no longer an excuse for discovering your P95 from angry callers.
Two metrics we insist on tracking beyond raw latency: barge-in recovery rate (when the caller interrupts, does the agent stop cleanly and respond to the new input?) and dead-air incidents per 100 turns (any silence over 2 seconds without an acknowledgment). Both correlate with caller hang-up rates far more tightly than average latency does. A voice agent with a 600ms P50 that mishandles interruptions will lose to a 750ms agent that yields gracefully every time — callers forgive slightly slow; they don't forgive being talked over.
The budget is the spec
Every voice project we take starts with a one-page latency budget: each stage, its allocation, its measured P50/P95. It's the first artifact we produce and the one we test against weekly, because every future decision — bigger model? extra retrieval step? nicer TTS voice? — has to buy its way in with milliseconds.
You can't prompt your way to fast. You have to architect it.
Building phone-grade voice and stuck above the one-second line? We've done this fight before — book a call.
