
What This Page Helps You Do
This page helps you decide whether Speech to Speech is the right agent type, choose between available OpenAI realtime, Gemini, and Ultravox model paths, configure the agent, and verify the first test call before routing live traffic.Before You Begin
You need:- Access to a DialNexa workspace where Speech to Speech is enabled
- Permission to create or edit agents
- A test phone route or web call route
- A short caller script for comparing OpenAI realtime and Gemini models fairly
- Any functions or integrations already configured if the realtime model must take actions during the call
When To Use Speech To Speech Agents
Use Speech to Speech Agents when fast spoken turns are central to the call experience. Good candidates include web calls, interruption-heavy sales conversations, short support triage, and demos where first audio timing is easy for users to notice. Use a cascaded Single Prompt Agent or Conversational Flow Agent instead when you need to tune the transcriber, pick a separate voice provider, use fallback STT, rely on Audio Cache, or audit each branch in a visual flow.How Speech To Speech Differs From Cascaded Agents
OpenAI Realtime And Gemini Model Paths
Speech to Speech model availability depends on the workspace. The dashboard model selector is the source of truth for which realtime models are currently enabled.
Gemini 3.8 Live replaces Gemini 3.1 Live. GPT Live appears in the same Speech to Speech selector for compatible single-language configurations and has no Voice Speed control. Confirm the rates and compatible voices in your workspace before planning production cost or quality.
Speech To Speech Language Choices
Choose the languages before choosing the voice. The language dialog supports a primary language and up to three secondary languages. The voice selector then shows voices compatible with the selected languages.
A Speech to Speech agent with secondary languages runs on Gemini Live. Its model and voice selectors show Gemini options for that selection; a non-Gemini realtime model cannot be saved with secondary languages. A single-language agent can use compatible OpenAI realtime, Gemini, or Ultravox options.
When English or Hinglish is selected, the language dialog offers English accent beside that language: Indian (default), British, or American. The choice applies to English speech throughout the call. It is not shown when none of the selected languages includes English.
Gemini can follow the caller between the configured languages. A caller can also ask for one of those languages explicitly. For speech in an unconfigured language, the agent is instructed to begin in its primary language with the configured unsupported-language message or a short explanation of its supported languages, then answer and continue the task in its primary language. Keep prompt instructions consistent with the selected languages, then test switches, short replies, names, dates, and amounts before routing production traffic.
Choose Explicit Speech To Speech Languages
Speech to Speech agents cannot use Multilingual auto-detection as their language selection. Choose an explicit primary language and supported secondary languages instead. An older draft saved with Multilingual must select a real primary language when reopened. Cancelling the voice picker discards the unsaved language selection. For cascaded agents that support Multilingual, it must be the only selected language; it cannot be combined with secondary languages.Gemini Speech To Speech Details
Gemini Speech to Speech uses a Gemini realtime model and compatible Gemini voices. It listens to caller audio, produces spoken audio directly, and does not require a separate transcriber or text to speech provider.Ultravox Speech To Speech Agents
Ultravox is available in every workspace for compatible single-language agents. Choose Ultravox v0.7 in the Speech to Speech model selector, then choose a compatible Ultravox voice for the primary language. Current voice choices cover English, Hindi, Hindi-English, and Tamil. The voice picker shows compatible voices with audio samples. An agent already using a voice that has since left the picker can keep it for calls. If you change the voice or model, select a currently compatible pair. Ultravox listens and speaks directly, so no separate transcriber, fallback STT, or voice model is configured. Ultravox voices have a Voice Speed control from 0.7 to 1.2; switching to an Ultravox voice sets 1.1. Gemini Live voices have no speed control. Speech Settings such as response eagerness and denoising do not apply to either. Use the draft’s test call to compare pronunciation, interruptions, opening timing, and configured tools before routing live traffic. Selecting secondary languages still requires Gemini Live. Use a single explicit primary language for an Ultravox agent and keep prompt language instructions consistent with that selection. Confirm the model and voice pricing shown in your workspace rather than assuming another provider’s rates.Set Up A Speech To Speech Agent
1
Create a new agent
Open the Agents tab, click Create agent, and select Speech to Speech where it is available.
2
Choose the languages
Select a primary language and any secondary languages. Choose the English accent when English or Hinglish is included.
3
Choose the realtime model
Use Gemini Live for an agent with secondary languages. For a single-language agent, select a compatible OpenAI realtime or Gemini option. Check the visible pricing preview before continuing.
4
Choose a compatible voice
Pick from the voices available for the selected languages and realtime model, and listen to samples before saving.
5
Write a concise prompt
Keep the role, goal, boundaries, tool rules, and closing behavior explicit. Realtime speech quality still depends on clear instructions.
6
Configure tools only when needed
Add functions or dashboard integrations only when the live call needs them. Then test the tool path with real caller phrasing.
7
Publish and route calls to the agent
Publish the version. Phone numbers and workflows linked to the agent use it right away; choose it explicitly for a batch call or web call.
Verify The Result
After the first test call, review both the subjective call feel and the call evidence.Troubleshooting
The transcriber selector is missing
The transcriber selector is missing
This is expected for Speech to Speech Agents. The realtime model listens directly, so separate STT settings are not used.
Audio Cache is missing
Audio Cache is missing
This is expected. Speech to Speech does not send text through a separate TTS provider, so there is no TTS cache to configure.
The model I expected is not visible
The model I expected is not visible
Model availability depends on workspace configuration. Check the model selector in the dashboard or contact DialNexa support if a required realtime model is missing.
The call is fast but less controllable
The call is fast but less controllable
Compare against a cascaded Single Prompt Agent using the same script. If separate STT, TTS, Audio Cache, or fallback STT controls matter more than latency, use the cascaded stack.
Function calls behave differently by model path
Function calls behave differently by model path
Keep the prompt, function schema, route, and caller script identical when comparing OpenAI realtime and Gemini models. Review function arguments in Call History before publishing.
A long maximum duration will not save
A long maximum duration will not save
Speech to Speech agents are capped at
3,600 seconds. Use a cascaded agent for routes that need a longer maximum duration.Recap
Speech to Speech Agents are for realtime voice behavior. OpenAI realtime and Gemini models can both be valid choices where enabled, but the winning path should be proven with the same prompt, route, caller script, tool setup, and Call History review.Speech To Speech Prompt Size
Speech to Speech agents use the prompt-size meter and save limits. The displayed budget reserves system instructions and counts expanded default variables. Shorten an over-limit prompt before saving or publishing.Gemini Session Recovery
If Gemini closes an established session during a call, DialNexa attempts to resume it with its conversation context. If resumption is unavailable, it can create a replacement session using the transcript so far. If recovery fails, the call ends withsystem_failure; review Call History before retrying a call.
Related Pages
Types Of Agents
Choose between Single Prompt, Conversational Flow, and Speech to Speech.
Provider Selection Guide
Compare speech, model, voice, and telephony layers.
LLMs And Conversation Behavior
Understand model behavior and fallback settings.
Languages, Voices, Models, And Transcribers
See how the Speech to Speech stack differs from the cascaded stack.
Testing Agents
Test before publishing a realtime model path.