# Speech-to-Text Models for Voice Agents Ask anyone who has tested a voice agent on Indian phone lines, and they'll describe the same moment. The caller says "haan, kal shaam ko", but the transcript reads something else. So the agent cheerfully answers a question nobody asked. When that happens, the speech-to-text layer is usually the first place to check. It decides what the agent hears, and it has very little time. Time really is short. People leave [around 200 milliseconds](https://www.frontiersin.org/articles/10.3389/fpsyg.2015.00731/full) between turns, according to Levinson and Torreira. So the best speech-to-text models must stay accurate on noisy phone audio and keep up with the caller. This guide compares the main options for Indian calls. ## **TL;DR** - Voice agents need streaming speech recognition that can cope with 8kHz phone audio and Hinglish. - Batch models such as Whisper are great for call transcripts and QA, but live calls need real-time speech-to-text. - Accuracy in Indian languages varies a lot. In one national benchmark, every model found Tamil, Telugu, Malayalam and Kannada harder than Hindi. - Deepgram, Sarvam AI, OpenAI, Gladia, ElevenLabs and AI4Bharat each suit a different kind of team. - Whichever speech-to-text API you shortlist, test it on your own recorded calls before you commit. ## **Why Does Speech-to-Text Decide How a Voice Agent Performs?** It's tempting to judge a voice agent by its voice or its script. Yet everything the agent says depends on what it heard first, and hearing is the speech-to-text layer's job. ### **Every Reply Starts With a Transcript** On a call, the agent listens, turns speech into text and passes it to a language model. Then it speaks the reply. So if the transcript is wrong, even a brilliant language model ends up answering the wrong thing. Names, amounts and dates hurt the most. One misheard order number or PIN code can undo an otherwise smooth call. ### **Phone Calls Are Harder Than Demos** Most demos use clean audio, recorded right next to the microphone. Real customers, however, call from busy streets, crowded shops and moving autos, often on patchy networks. For that reason, the speech-to-text API you pick has to cope with messy audio, not only studio samples. If you're building this into your own product, a [voice call API](https://dialnexa.com/blogs/voice-call-api/) can handle the telephony around it. ## **What Makes the Best ASR for Voice Agents?** So what should you actually look for? Four qualities matter far more than leaderboard rank, and each deserves its own test. ### **Accuracy on Real Calls** Most comparisons start with word error rate, or WER. It simply counts the words a model gets wrong, adds or drops, compared with a careful human transcript. The catch is that a low WER on a public dataset tells you little about your own calls. Accents, product names and background noise all nudge the numbers around. Picture a collections call where the customer says "fifty thousand", but the transcript reads "fifteen thousand". The agent confidently quotes the wrong amount, and from that moment the customer stops trusting it. ### **Speed and Streaming** Live agents can't wait for the caller to finish. Instead, they need partial transcripts while the person is still talking, so the language model gets a head start. The model also has to sense when the caller has actually finished. Cut in too early, and the agent talks over people. Wait too long, and the line goes awkwardly quiet. ### **Handling 8kHz Telephone Audio** Phone audio is narrowband, which is easy to forget when you test on laptop recordings. [Twilio's media streams](https://www.twilio.com/docs/voice/media-streams/websocket-messages), for example, deliver mono μ-law audio sampled at 8,000 Hz, well below podcast quality. Many models are built and tested on higher-quality audio, so their accuracy can slip on phone lines. Unfortunately, upsampling won't bring back the missing detail. The only reliable check is a test on genuine 8kHz recordings. ### **Language and Code-Mixing** Then there's language, and Indian callers rarely stick to just one. A single sentence might start in Hindi, borrow English for an amount and drift back into Hindi before it ends. A good model follows that switch without tearing the sentence apart. If it can't, even well-built [multilingual conversational AI](https://dialnexa.com/blogs/multilingual-conversational-ai-how-businesses-automate-calls-across-languages/) ends up working from a broken transcript. ## **Real-Time vs Batch Speech Recognition: Which Do Voice Agents Need?** Once you know what to look for, the next decision is how the model handles audio. There are two modes, and you'll often want both. ### **Real-Time Speech-to-Text for Live Calls** Streaming models transcribe audio as it arrives, usually over a WebSocket connection. Any live agent depends on this mode, simply because the reply can't wait for the call to end. It does tend to cost more, though, as real-time speech-to-text is often pricier per hour than batch. If you're new to [AI phone calling](https://dialnexa.com/blogs/ai-phone-calling-how-ai-agents-automate-business-calls/), budget for streaming from day one. ### **Batch Transcription After the Call** Batch models, by contrast, work on a finished recording. They can look at the whole call for context, and they often cost less. That makes them ideal for summaries, QA reviews and compliance archives. Gladia's pricing shows the gap nicely. Its real-time tier starts at €0.68 an hour, whereas its Solaria-3 batch model starts at just €0.16 an hour. So the best ASR for voice agents is often really a pair. One streaming model handles the live call, while a batch model takes care of the record afterwards. ## **How Well Do Models Handle Hindi and Indian Languages?** Language is where many Indian deployments come unstuck. Thankfully, there's now a public benchmark built on real call audio. ### **What the Voice of India Benchmark Found** Voice of India was built by Josh Talks with AI4Bharat at IIT Madras. It draws on unscripted telephone conversations in 15 languages, which makes it close to real call traffic. [As CXOToday reported](https://cxotoday.com/media-coverage/global-speech-ai-struggles-to-understand-india-new-national-benchmark-voice-of-india-reveals/), every model did far better on Hindi and Bengali than on the Dravidian languages. Hindi and Bengali error rates sat around 5 to 6%. Tamil, Telugu, Malayalam and Kannada sat around 15 to 20%. Dialects were tougher still. On varieties such as Bhojpuri, even the best models slipped to 20 to 30%. According to the report, Sarvam ranked first or second in almost every language and dialect tested. Google's Gemini stayed competitive. OpenAI's GPT-4o mini transcribe, on the other hand, went above 55% WER. ### **Hinglish Needs Its Own Test** Indic speech recognition gets harder again once callers start mixing languages. In fact, one [**voice AI report**](https://dialnexa.com/blogs/state-of-ai-voice-calling-india/) on Indian calls found some pipelines struggled with transcription accuracy in mixed-language conversations. So it's worth testing Hinglish separately from pure Hindi. Good Indic speech recognition should keep words like "EMI" or "booking" intact, instead of forcing them into awkward Devanagari guesses. ## **Which Are the Best Speech-to-Text Models for Indian Voice Agents?** With accuracy, speed and language in mind, it's time to look at the actual options. These six are worth testing for [multilingual voice agents](https://dialnexa.com/languages/multilingual), and each one suits a different situation. ### **1. Deepgram Nova-3 and Flux** Deepgram's Nova-3 covers Hindi, Indian English, Tamil, Telugu, Bengali, Marathi and Kannada, in batch or streaming. Its newer Flux model is built for streaming, with turn detection baked in, and its multilingual version covers Hindi. Nova-3 monolingual costs $0.0077 a minute on pay-as-you-go. New accounts also get $200 in free credit for a proper trial. **Suits:** Teams that want a telephony-friendly streaming API with Hindi support. **Check first:** Flux's multilingual model doesn't list Indian English, so test which model suits your call mix. ### **2. Sarvam AI Saaras** Saaras v4 is Sarvam's default speech model, and it covers 22 Indian languages plus English. It's trained on Indian accents and code-mixed speech, and it even offers a dedicated codemix output mode. Billing works per second. On pay-as-you-go, that comes to just under ₹30 an hour, whether you use real-time, streaming or batch. **Suits:** Agents serving regional-language callers across India. **Check first:** Pay-as-you-go allows 20 concurrent streaming connections, so higher volumes need a paid plan. ### **3. OpenAI Whisper and GPT-4o Transcribe** Whisper large-v3 is open source under Apache-2.0 and covers 99 languages. However, its own model card says it can't be used for real-time transcription out of the box. If you'd rather not host anything, OpenAI's gpt-4o-transcribe is estimated at $0.006 a minute. Its live transcription models, meanwhile, come in at $0.017 a minute. **Suits:** Post-call transcripts, analytics and teams happy to self-host. **Check first:** Given the Voice of India results for OpenAI's models, test Indian-language accuracy carefully. ### **4. Gladia** Gladia offers real-time transcription in up to 100+ languages, plus an async Solaria-1 model with similar reach. New accounts get €50 in free credit, valid for a year. If you're weighing Gladia AI alternatives for India, two things deserve the closest look: Indic accuracy and rupee billing. **Suits:** Multilingual products where Indian languages are one market among many. **Check first:** Its newest Solaria-3 model covers only six languages, so confirm which model handles your calls. ### **5. ElevenLabs Scribe v2 Realtime** Scribe v2 Realtime is ElevenLabs' streaming model, covering 90+ languages at $0.39 an hour. It suits teams already using ElevenLabs voices. **Suits:** Teams that want speech recognition and voice output from one vendor. **Check first:** Language counts don't show accuracy, so benchmark it on Hindi and regional calls. ### **6. AI4Bharat IndicConformer** IndicConformer is an open-source, 600-million-parameter model from AI4Bharat, released under the MIT licence. It covers 22 Indian languages and reports a Hindi WER of 13.2 on the Vaani benchmark. **Suits:** Teams that need full control and can host their own models. **Check first:** Its model card doesn't describe streaming, and AI4Bharat now recommends its newer Indic-Transcribe model instead. ### **Whisper vs Deepgram for Phone Calls** When it comes to Whisper vs Deepgram, the answer depends on the job. For live phone agents, Deepgram's streaming models and telephony encodings make it the more natural fit. Whisper, on the other hand, works well after the call, when speed matters less. Nothing stops you from running both, one for live calls and one for the record. ### **How These Options Compare** The table below lines up all six on what each one publishes: | **Model** | **Mode** | **Indian languages** | **Licence** | **Published price** | |---|---|---|---|---| | Deepgram Nova-3 | Batch and streaming | Hindi, Indian English and 5 more | Hosted API | $0.0077 a minute | | Sarvam Saaras v4 | Batch and streaming | 22 Indian plus English | Hosted API | Under ₹30 an hour | | OpenAI Whisper | Batch | Hindi and others in 99 | Open source | Free licence, self-hosted | | Gladia | Batch and real-time | Up to 100+ languages | Hosted API | €0.68 an hour real-time | | ElevenLabs Scribe v2 | Real-time | 90+ languages | Hosted API | $0.39 an hour | | AI4Bharat IndicConformer | Not stated | 22 Indian languages | Open source (MIT) | Free licence, self-hosted | If most of your callers speak regional languages, start with the Indic-first options. If Hindi and English cover most of your traffic, though, a global streaming API may be enough. ## **How Should You Benchmark Speech-to-Text Models Before Choosing?** Published numbers are a useful starting point, but they only take you so far. Even the best speech-to-text models on paper can stumble on your callers. So test each one the same way. ### **Build a Test Set From Real Calls** Start by pulling 200 to 300 recorded calls that look like your real traffic, across languages, regions and noise levels. Then ask people to transcribe them carefully, so you have a reliable reference. Include the awkward cases on purpose, such as Hinglish, long numbers, brand names and weak signal. After all, those are exactly the moments that trip up a live agent. That shared set also lets you compare Gladia AI alternatives, or any shortlist, fairly. ### **Measure More Than WER** Word error rate is a good start, but it isn't the whole story. Also track how often key fields come out right, such as amounts, dates and names. Next, time each model's first partial and final transcript. Then add cost per hour at your expected volume. These measures help again when you compare full [voice AI companies](https://dialnexa.com/blogs/voice-ai-companies-what-to-look-for-when-choosing-a-voice-ai-provider/). ### **Run a Short Live Pilot** Once two models stand out, put each one on live calls for about a week. Real traffic surfaces problems recordings miss, such as network drops and sudden background noise. And do listen to a handful of calls yourself. A transcript can look fine while the agent still sounds slow or confused to the caller. ## **Conclusion** A voice agent can only be as good as what it hears. On Indian phone lines, that means narrowband audio, background chatter and callers who happily switch languages mid-sentence. So judge the best speech-to-text models on your own calls rather than on leaderboards alone. Look at accuracy on key fields, streaming speed, language coverage and cost per hour together. Of course, not every team wants to wire up each provider itself. [DialNexa](https://dialnexa.com) runs its voice agents on speech providers such as Deepgram and Soniox, so that layer comes ready-built. Either way, test before you commit. ## **Frequently Asked Questions** ### **1. What Is the Best Speech-to-Text Model for Indian-Language Phone Calls?** There isn't a single winner. Among the best speech-to-text models, Indic-first options like Sarvam led the Voice of India benchmark. Global APIs such as Deepgram Nova-3 also support Hindi and Indian English. Ultimately, your own call recordings should make the final decision. ### **2. How Accurate Is Speech Recognition on Noisy 8kHz Telephone Audio?** Accuracy usually drops on phone lines because 8kHz audio carries less detail than wideband recordings. Noise and accents then push error rates up further. So never judge a model on clean samples alone. Instead, test it on genuine telephone recordings that include the noise and dialects your callers bring. ### **3. What Latency Is Acceptable for Real-Time Speech Recognition in a Voice Agent?** People usually reply within about 200 milliseconds, so every part of the pipeline needs to move quickly. Speech recognition should stream partial transcripts while the caller is still talking. It should also spot the end of a turn fast, leaving time for the language model and the spoken reply. ### **4. Can One Speech Model Handle Hindi-English Mixed (Hinglish) Speech?** Yes, several can. Sarvam's Saaras models are trained on code-mixed speech and offer a codemix output mode. Deepgram's multilingual models also handle Hindi alongside English. Quality still varies with accent and topic, though. So test Hinglish as its own category rather than assuming Hindi results will carry over. ### **5. Which Languages Does Speech Recognition Support for Indian Regional Languages?** It depends on the model. Sarvam's Saaras v4 and AI4Bharat's IndicConformer each cover 22 Indian languages. Deepgram Nova-3 covers Hindi, Tamil, Telugu, Bengali, Marathi and Kannada. Global models list many more. However, in the Voice of India benchmark, accuracy on Dravidian languages was lower than on Hindi. ### **6. How Should I Benchmark Speech-to-Text Models Before Choosing One for Calling?** Start with 200 to 300 real calls and careful human transcripts. Then run every model on the same audio. Compare word error rate, accuracy on key fields, streaming latency and cost per hour. Finally, weight the results by the languages your callers actually use.