Play someone an isolated clip of a modern TTS voice and they often cannot tell it's synthetic. Put that same voice into a live phone call and within one exchange, most people know they are talking to a machine.
The voice is not the tell. The rhythm is.
What actually gives it away
Human conversation has a turn-taking rhythm most people have never consciously noticed but instantly detect the absence of. Real conversational gaps between turns average around 200 milliseconds - sometimes negative, because people start responding before the other person finishes. A pause longer than about half a second reads as awkward. A full second reads as robotic.
A typical voice agent pipeline - transcribe the caller's speech, wait for a pause to confirm they're done, send text to a language model, wait for a response, convert to speech, start playing it - stacks up latency at every step. Even a well-optimized pipeline often lands in the 700ms-1200ms range before the agent starts speaking. That gap is exactly the signal humans use to detect they're not talking to a person.
The voice model was never the problem. The silence around it is.
The three things that actually fix this
Reduce the gap directly. Every stage in the pipeline - speech-to-text, the model's time-to-first-token, text-to-speech - has a latency cost, and they're additive. Streaming architectures that start generating a response before the caller has fully finished speaking, and start playing audio before the full response is generated, cut the perceived gap dramatically. This is an architecture decision, not a voice model decision.
Backchannel during long responses. Humans don't go silent while listening - they say "mm-hmm," "right," "yeah" while the other person talks, signaling they're still present. An agent that is silent for the full duration of the caller's turn and then produces a long, uninterrupted response sounds like it wasn't listening at all, then dumps a monologue. Small acknowledgment sounds during the caller's turn close that gap.
Handle interruption gracefully. Real conversation involves people talking over each other constantly - a caller correcting themselves mid-sentence, jumping in before the agent finishes a sentence they already understand. An agent that keeps talking through an interruption, or takes a full second to notice it stopped being the only voice on the line, breaks the illusion immediately. Fast, accurate barge-in detection - noticing the caller started speaking and yielding the turn within a couple hundred milliseconds - matters more to perceived naturalness than almost anything else in the pipeline.
Why this is a systems problem, not a model problem
Teams debugging a robotic-sounding agent often start by trying a different voice model, a different LLM, a different prompt. Those changes rarely move the needle, because the actual defect is in the plumbing between the components, not in any one component's quality.
The latency budget has to be treated as a first-class constraint from the start of the design, not a performance optimization pass at the end. Every added step in the pipeline - a moderation check, a tool call, a lookup to a database before responding - adds to the gap the caller experiences as "is this a machine." A pipeline that is architecturally correct but budget-blind will still feel robotic even with the best possible voice model sitting inside it.
What to actually measure
If your agent feels robotic, measure the actual gap between the caller finishing a sentence and the agent's audio starting, across real calls, not test calls with pre-warmed connections. Measure how often the agent talks over an interruption instead of yielding. Measure whether backchannel sounds exist at all during longer responses.
Those three numbers explain almost every "why does this feel fake" complaint teams get about voice agents, and none of them are about which voice model you picked.