Cutting latency in a real-time voice bot: VAD, speculative generation, and a 300 ms rescue

When you talk to a voice bot, the thing you judge it on isn't how smart it is. It's whether it answers like a person would — fast enough that the pause doesn't feel weird. Cross about a second of silence and people start saying "hello?" into the phone.

The problem is that a voice turn is a chain, and every link adds time:

1. Speech-to-text (ASR) turns what the caller said into words.
2. The LLM decides what to say back.
3. Text-to-speech (TTS) turns that into audio.

Run those strictly one after another and you stack three latencies end to end. The whole game is making them overlap — and not waiting for things you don't need to wait for.

Endpointing is half the fight

Before the model says anything, something has to decide the caller is done talking. Wait too long and the bot feels slow. Cut in too early and you talk over them mid-sentence.

Two things matter here:

• Interim vs final transcripts. A good streaming ASR gives you partial results as the caller speaks: "I want to…", "I want to pay…", "I want to pay on Friday." You don't have to wait for the final transcript to start thinking.
• Per-speaker endpointing. How long a pause means "I'm finished" isn't the same for both sides of a call. A tighter endpoint timeout on the agent and a more patient one on the human cuts both false interruptions and dead air. Something like ~300 ms on one side and ~800 ms on the other has worked for me; the right numbers are empirical and you tune them on real calls.

Getting endpointing right buys you more perceived speed than any model swap.

Speculative generation: start before you're sure

Here's the trick that helped most. You don't have to wait for the final transcript to start generating. When the partial transcript stops changing — when "I want to pay on Fri" becomes "I want to pay on Friday" and holds — you can start the LLM on that partial, speculatively.

If the final transcript matches what you already started on (say, a three-word overlap with what you predicted), you flush the response you've been preparing. If it doesn't match, you throw the speculative work away and regenerate. You eat some wasted tokens on the misses, but on the hits you've hidden the model's thinking time inside the caller's own last half-second of speech.

That's the trade: a little extra token spend in exchange for turns that begin before the human has fully stopped talking.

A rescue timer, so you're never just… silent

Even with all that, the model is sometimes slow. Dead air is the worst failure mode in voice — worse than a slightly clumsy answer — because the caller can't tell whether the line dropped.

A short rescue path fixes the feel. If nothing's ready within a few hundred milliseconds of when a response was due, play a tiny filler ("Okay—", "Let me check that…") while the real answer finishes. It's the verbal equivalent of "mm-hmm." Humans do it constantly. It buys time, and the call never goes uncomfortably quiet.

Stream the speech out, and let people interrupt

Two more that matter:

• Stream the TTS. Don't wait for the full answer text before you start speaking. As the model streams tokens, feed finished sentences into a streaming TTS so audio starts while the rest is still being written.
• Barge-in. Let the caller talk over the bot. The instant your VAD hears speech while the bot is talking, stop the audio and start listening. A bot that keeps talking over someone feels broken; one that yields immediately feels alive.

Measure the thing you actually care about

Per turn, log the end-to-end latency from "caller stopped talking" to "bot started talking," plus the piece times (ASR final, LLM first token, TTS first audio) and the WebSocket round-trip. The piece times tell you which link to attack next; the round-trip tells you when the network — not your code — is the problem. Without per-exchange numbers you're optimizing by vibes.

The shape of it

Fast voice isn't one big win. It's overlapping the chain, starting the model before the human is fully done, never going silent, and letting people cut in. Get those right and a stack of three slow steps starts to feel like a conversation.

---

Neville James Achieng builds LLM and voice systems in Nairobi. github.com/Neville777.