How Fast Should Your AI Voice Agent Respond?

A production AI voice agent should start speaking within 800ms-1.2 seconds of the caller finishing a sentence. Past 2 seconds, callers assume the line dropped, start repeating themselves, or hang up. If you're evaluating vendors or building this in-house, latency isn't a nice-to-have metric, it's the difference between an agent people trust and one they route around.

Why latency decides whether people trust the agent

Human conversation has a rhythm. Research on turn-taking across languages found the average gap between one person finishing and the next starting is around 200 milliseconds, often even before the first speaker finishes (Stivers et al., PNAS). Nobody expects an AI to hit 200ms today, but the baseline it's competing against is that fast, not the 3-5 seconds a customer might tolerate from a slow web page.

On the interface side, Nielsen Norman Group's response-time thresholds are the classic reference: under 0.1s feels instant, up to 1s keeps the user's flow of thought uninterrupted, and past 10s they've mentally checked out. Voice compresses that scale, because there's no spinner, no progress bar, just silence, so a 2-second gap on a call feels much longer than the same gap on a webpage.

This matters most for anything transactional: booking changes, order status, support triage, outbound sales. If you're weighing this against a text-based channel, the latency bar for an AI WhatsApp sales agent is far more forgiving, a few seconds reads as normal typing time. Voice has no such cover.

Where the milliseconds actually go

A typical voice agent pipeline has three stages, and most implementations run them sequentially:

1. Speech-to-text (STT), transcribing the caller's audio. 150-500ms depending on whether it's streaming or batch.
2. LLM generation, deciding what to say and generating the full response. 500ms-2s+ depending on model size, prompt length, and whether it's calling tools (looking up an order, checking a calendar) before it can answer.
3. Text-to-speech (TTS), converting the full text reply into audio. 200ms-1s.

Add those up sequentially and you're easily at 1.5-3.5 seconds before the caller hears anything, even before network round trips. That's the gap that makes an agent feel broken.

The fix isn't a faster model, it's not waiting for the full response before you start talking. Streaming the LLM's output token-by-token into TTS, and speaking as soon as the first sentence is ready instead of the whole answer, is what actually collapses the perceived delay, the same idea captured in the "streaming sentences, not audio" approach some teams have landed on. It's a pipeline decision, not a model upgrade, and it's usually the highest-leverage fix available.

Tool calls make this harder. If the agent needs to check a database before it can answer ("let me check your order"), that lookup has to happen inside the gap, and it needs its own filler strategy, a natural acknowledgment ("one sec, pulling that up") rather than dead air. This is one of the reasons agentic voice systems need more careful orchestration than a simple scripted IVR replacement, the agent is doing real work mid-conversation, not just reciting text.

How to evaluate a vendor on latency

Don't take a demo video at face value. Ask for:

• A live, unscripted call on your own network, not a pre-recorded demo run on idle infrastructure.
• Time-to-first-audio-byte, measured, not "response time," which vendors sometimes quote as time-to-first-token from the LLM, ignoring TTS.
• Latency under concurrent load. A single test call tells you nothing about what happens at 50 simultaneous calls, when GPU queuing and rate limits kick in.
• What happens on tool calls. Ask them to trigger a real lookup (CRM, calendar, order system) mid-call and time the gap.
• Regional routing. If your callers are in the Gulf or elsewhere outside the vendor's default region, ask where inference actually runs, a model hosted three continents away adds real, physics-limited round-trip time no amount of engineering fixes.

This last point compounds if you're also dealing with accented speech or a non-English language, Arabic ASR in particular has real accuracy-versus-latency tradeoffs, since higher-accuracy models for Gulf dialects are often slower and pricier than generic English STT.

If you're putting any of this in a contract, treat latency the same way you'd treat an accuracy number, define it, and hold the vendor to it. The same discipline used for evals in AI vendor contracts applies directly: without a measurable SLA, "fast" is whatever the vendor felt like on demo day.

Red flags

• Vendor can't tell you their p50/p95 time-to-first-audio-byte off the top of their head.
• Demo only works well on scripted questions, ask something off-script and watch the gap widen.
• "Fast enough" is the only answer you get when you ask for a number.
• No mention of streaming anywhere in their architecture, if they wait for a complete LLM response before TTS starts, you're locked into the slow path no matter which model they swap in.

The bottom line

Latency is an architecture decision, not a model-selection decision. A well-engineered pipeline on a mid-tier model will consistently beat a poorly-streamed pipeline on the best model available. Before you sign with a vendor or greenlight a build, get a real number, not a vibe, and if you're integrating AI into an existing product, budget engineering time for the streaming plumbing up front, because retrofitting it later is a rewrite, not a patch.

If you're scoping a voice agent and want a second opinion on whether a vendor's latency claims hold up, let's talk.