A voice agent that sounds good in a demo and a voice agent that survives real calls are two different systems. Here's what we've learned building phone agents.
1. Latency is a budget, not a number. Every turn spends it across STT, the model, TTS, and the network hop to the carrier. Stream everything: partial transcripts in, first tokens out, audio synthesized sentence by sentence. Speech-to-speech models remove some hops, but tool calls still cost time.
2. Endpointing decides whether the agent comes across as rude or slow. Silence thresholds alone either cut people off mid-thought or leave dead air. Pair voice activity detection with semantic cues: did the caller finish the sentence, or did they stop to look up an account number?
3. Barge-in has to be real. When the caller talks over the agent, stop the audio right away, trim the agent's turn in context to what the caller actually heard, and answer the interruption instead of the script.
4. Tool calls mid-call need cover. If a lookup takes a beat, say so ("pulling that up now") and run it async so the line never goes quiet.
5. Handoff is a feature. Define the triggers (an explicit request, repeated failure, sensitive topics), give the human a structured summary, and never make the caller repeat themselves.
6. Compliance is part of the architecture. In 2024 the FCC confirmed that AI-generated voices count as "artificial" under the TCPA, so outbound calls need proper consent. Tell callers they're talking to an AI, log consent, and keep the recording policy explicit.
The model is the easy part to swap. The rest is timing, state, and knowing when to step aside.