https://x.com/i/article/2117741140438872064
3 layers that take a voice agent from demo to production
Voice interfaces feel natural, fluid, and forgiving because they align with how we were built to interact. For any product aiming to be frictionless, verbal communication isn't just a feature request. It's an essential interface.
However, creating a pleasant conversational experience and delivering business-grade accuracy are vastly different engineering challenges. What truly separates a simple demo from a production-ready product comes down to three underlying architectural layers: raw API constraints, transcript processing logic, and telephony requirements.
There is a second way to build this. Gemini 3.8 Live and 3.8 Live Extended Thinking, launched in September, are speech-to-speech models that listen, reason, and speak in one hop. Gemini 3.8 Live Extended Thinking ranked first at launch on Artificial Analysis' Speech to Speech Quality Index at 82.6. They remove the hand-off between transcription and reasoning, which is where much of a pipeline's latency lives.
This piece covers the other path, the one you pick when you need a text transcript you can inspect, audit, and route to your own orchestrator. Most of what follows, especially the telephony layer, applies to both.
By Sergio Villani and Pedro Melendez, Technical Solutions, Google Cloud AI.
Three layers, in the order a voice agent actually runs into them:
- The two endpoints don't share a feature set, and the API enforces the gaps, not just the docs.
- An interim transcript and a finalized one are different objects with different jobs, and mixing them up breaks your agent's decisions.
- The hardest part of a voice agent is the phone call and the audit trail, not the model call.
The two endpoints don't share a feature set, and the API enforces it
Read only the launch post and "Gemini 3.5 Transcribe" sounds like one model. But there are two: gemini-3.5-transcribe for pre-recorded audio and gemini-3.5-transcribe-live for bidirectional WebSocket streaming. Batch supports word-level timestamps and speaker diarization for up to eight speakers, with attribution for three or more still marked experimental. Live drops both of those in exchange for low-latency, real-time results, and caps you at 10 minutes per session against batch's 1 hour, or 30 minutes once diarization or timestamps are switched on.
The part worth knowing before you architect anything: these aren't soft guidelines. They're constraints the API enforces by rejecting the request. Custom vocabulary accepts up to 1,000 phrases, with best results under 100, and it can't be combined with diarization or timestamps. Smart mode strips filler words and resolves spoken self-corrections. Say "let's meet Tuesday, actually no, Wednesday" and it comes back as "let's meet Wednesday." Smart mode can't be combined with diarization or timestamps either. You pick verbatim-with-metadata or smart-and-clean. Not both, and not by accident.
An interim transcript and a finalized one are different objects, and your agent should only see one
The Live API streams two distinct fields inside server_content: interim_input_transcription, a speculative partial that updates continuously while someone is mid-sentence, and input_transcription, the authoritative segment emitted once the turn is done. The first is for your UI. The second is the only one your agent orchestrator should ever act on.
That distinction sounds obvious until you're the one wiring a voice bot and the temptation is to feed every interim update into your reasoning model to shave off latency. Resist it. An interim hypothesis can still change mid-sentence, and an agent that calls tools against a transcript that's about to be revised is an agent that occasionally acts on words that nobody said. The documented pattern decouples this cleanly: the live model handles speech-to-text and hands a finalized utterance to a separate orchestrator model, which is the only thing that decides what happens next.
There's a second decoupling worth building in from day one. If you're streaming audio from a browser or mobile client, mint a single-use ephemeral token server-side instead of shipping your API key to the device. The SDK scopes it precisely: a uses count, an expire_time, and a live_connect_constraints block that locks the token to one model and one config. A leaked client-side token is a ticket to one short session, not to your whole account.
The hardest part of a voice agent is the phone call, not the model
The model tier gets the attention because it's measurable. The tier that decides whether a voice agent survives contact with a real user is telephony, and it's the least glamorous part of the stack. The audio format is just as unglamorous: raw 16-bit PCM at 16 kHz, sent in roughly 100 ms chunks. That's exactly why frameworks like LiveKit, Pipecat, Fishjam, and Agora exist as named Live API partners. They sit between real-world audio sources and that PCM format so you don't build the bridge yourself.
The other production-only detail is turn-taking under real silence. Server-side voice activity detection, or VAD, alone waits out a fixed timeout before deciding someone stopped talking, which reads as dead air on a call. Hybrid VAD fixes it by pairing that server-side detection, which still catches the start of speech accurately with prefix padding, with a client-side detector that fires an immediate audio_stream_end signal the moment local silence is heard. The server treats that as an instant finalization request instead of waiting out its own timer, and falls back to its own detection if the client-side signal never arrives.
Once the call ends, batch transcription with diarization and timestamps turns a recording into an auditable record: speaker-labeled, word-timed text. This text can sit in a data warehouse for dispute resolution instead of a raw audio file nobody wants to listen to again. The same model family is available through Gemini Enterprise Agent Platform's Agent Studio, bindable as
gemini-3.5-transcribe-live-preview, behind the platform's existing access controls: IAM and private endpoints, not new plumbing you build yourself.
These three layers aren't sequential, they're three different reliability bars
It's tempting to treat this as a roadmap: ship the API call, then the agent wiring, then worry about telephony later. In practice a voice agent needs all three bars cleared at once. An incompatible mode configuration fails the same afternoon a missing VAD fallback does. Accuracy, latency-safe decision-making, and telephony resilience aren't stages of maturity. They're three separate ways a voice product breaks, and a launch post measuring word error rate only tells you about one of them. Choose 3.8 Live instead and the first two layers change shape, but the phone call is exactly as hard as it was.
Try the live model in Google AI Studio, then read the transcription guide end to end before your first integration, specifically the limitations section. Get runnable versions of all three layers, with offline tests, on GitHub at voice-agent-three-layers.