Why voice agents feel slow, and how to make them feel fast
A second of silence on a phone call feels much longer than it is. Where the delay comes from and the settings that bring it down.
The Vaakyo team2 min read
- Performance
- Engineering
In a conversation between people, the gap between one person finishing and the other replying is very short. When a voice agent takes noticeably longer, callers start saying "hello?", repeat themselves or talk over the reply. Latency is the most common reason a technically correct agent still feels robotic.
Where the time goes
Each reply passes through four stages:
- End of turn. The agent has to decide the caller has finished speaking. Wait too little and it interrupts; wait too long and every reply starts late.
- Transcription. The last words are turned into text.
- The language model. It reads the conversation and starts writing a reply. What matters is the time to the first words, since the voice can start speaking before the whole reply is written.
- The voice. The first words are turned into audio and played.
Tools add a fifth stage when the agent has to look something up mid-call.
The call timeline in Vaakyo shows the latency of each stage for every turn, so you can see which one to work on.
What to change
- Use a fast model. For most calls, a fast model gives a better experience than a smarter, slower one.
- Keep the prompt focused. Very long prompts and large knowledge pasted into the prompt slow the first words down. Move reference material into a knowledge base.
- Shorter replies. Ask for one or two sentences per turn. Short replies start sooner and leave room for the caller.
- Fast tools. A tool that takes two seconds adds two seconds of silence. Cache what you can, and have the agent say a short filler ("let me check") before slow lookups.
- Tune end-of-turn detection for your callers. Callers who pause mid-sentence need a slightly longer wait; quick yes/no flows can use a shorter one.
- Allow interruptions, so callers can cut in when they already know the answer, while short acknowledgements like "haan" or "ok" don't stop the agent.
Measure on real calls
Measure the delay as the caller hears it: from the moment they stop speaking to the moment the agent's voice starts. Look at the slow turns rather than the average; one five-second gap is what the caller remembers.
