The 30-Millisecond Judge: Training a Small Model to Decide When Our Voice Agent Is Allowed to Cheat
Three matchers in, we learned the real lessons the hard way: cosine similarity that couldn’t tell “yeah” from “nah”, a base model that ranked beautifully but couldn’t decide, a quantization step that resurrected our worst bugs — and why the data pipeline, not the model, turned out to be the asset.
The bug that made this a story
Our voice agents make outbound phone calls. Latency is the whole game: if the agent takes two seconds to respond, humans talk over it, repeat themselves, or hang up. One of our best latency tricks is a predictive cache: while the agent is speaking, a background LLM predicts the user’s most likely next utterances (“Yes”, “Who is this?”, “I’m busy right now”) and pre-generates full responses for each. When the user actually speaks, if their utterance matches a prediction, we serve the cached response in ~80 ms instead of paying ~1.5 s for LLM + TTS.
The catch is the word matches. Serving a cached response to the wrong utterance isn’t a latency win — it’s the agent confidently answering a question nobody asked.
One day a user said “Sure.” and our agent responded as if they’d said “नहीं, अभी busy हूँ” (“no, I’m busy right now”) — because our match gate scored that pair at 0.999. A user said “हां जी बोलिए” (“yes, go ahead”) and got the response for “नहीं, अभी समय नहीं है” (“no, I don’t have time”). Same polarity flip, same absurd confidence.
That incident is the middle of the story, not the start. The gate that failed there was already our second matching system — and understanding why it failed the way it did requires the whole journey: cosine similarity → a fine-tuned English cross-encoder (v3) → a multilingual one (v4.1). Each stage earned the next.
Act I — Cosine similarity, and where it tops out
The first version of the matcher was the obvious one: embed the incoming utterance, embed the predicted utterances, serve the best candidate if cosine similarity clears a threshold. One small sentence-embedding model, a few milliseconds, no training required.
It worked — for a narrow definition of “worked.” Two problems surfaced almost immediately, and they’re both fundamental to how phone conversations look as text:
Phone replies are short, and short replies all embed alike. The most common user turns are things like “Yes.”, “Yeah.”, “Nah.”, “Who is this?”, “Not now.” Sentence embeddings place these in a tight little cluster: “Yeah” vs “Nah” — opposite meanings — score ~0.7. “Yeah” vs “Yes, go ahead” — same meaning — also ~0.7. The true-match and false-match score distributions overlap almost completely on short text. There is no threshold that separates them, because the information that distinguishes them (one negation token) is a rounding error in a pooled embedding.
So the threshold gets cranked, and recall starves. The only safe operating point was ~0.9 with top-1 matching — which in practice means near-exact wording. “Not now” wouldn’t match “I’m busy right now.” “Go on” wouldn’t match “Please continue.” The cache hit on the turns where the LLM predicted the user’s exact words and missed the far larger set where it predicted their meaning. We were leaving most of the latency win on the table to stay safe — and still occasionally serving a wrong short-reply match that snuck over 0.9.
The conclusion wasn’t “embeddings are bad.” It was a division of labor that survived every later version: embeddings are a fine shortlist; they cannot be the judge. Retrieval and judgment are different jobs. We kept cosine to fetch candidates and went looking for a judge.
(Why not an LLM judge? 300–800 ms even on a fast hosted model — you’d spend the latency budget deciding whether you’re allowed to save it. The judge had to run in tens of milliseconds on CPU, next to the audio pipeline.)
Act II — v3: a small English cross-encoder earns the job
The right architecture for “are these two utterances interchangeable?” is a cross-encoder: feed the pair (actual_utterance, predicted_utterance) through one small transformer that attends across both texts, output a single probability. Unlike bi-encoder embeddings, the model sees both sentences at once — a negation token in one can attend to its counterpart in the other. With int8 quantization, a 6-layer model scores a 20-candidate shortlist in a few dozen milliseconds on two CPU threads.
We started from cross-encoder/stsb-distilroberta-base — a model already fine-tuned for sentence-similarity judgment — and taught it our domain in two data passes:
- ~2,900 real pairs mined from production logs: actual (utterance → prediction) decisions, labeled. Real pairs anchor the distribution.
- ~1,700 synthetic pairs in equivalence clusters: paraphrase sets for the intents that matter on calls (agreement, refusal, deferral, identity questions, repeat requests…), plus a hand-picked set of danger negatives — the “yeah”↔”nah” class that cosine could never see.
Two deployment choices from this era proved more valuable than the model itself:
- Shadow mode first. The matcher scored every lookup while cosine still made the serving decision, logging what it would have done. A week of shadow traffic calibrated the threshold on reality instead of the test set, for free.
- Fail-open, always. If the model artifact is missing or scoring fails, the cache degrades to conservative cosine behavior rather than blocking calls. (This choice ages interestingly — see the ops lesson at the end.)
The error asymmetry defined the operating point, and it’s worth stating because it shaped every version since: a false positive speaks a wrong response to a live human — user-visible harm. A false negative just falls back to the LLM — the latency win is lost, nothing bad is said. So we run at extreme thresholds (0.998 in probability space for v3) and optimize recall under an FPR ceiling, never accuracy.
v3’s results, against the cosine baseline it replaced: recall 0.42 at 1.7% FPR on held-out real pairs — versus cosine’s effectively-exact-match recall at any safe threshold. Every known hole flipped: “go on” → “Please continue” went from 0.001 to 0.9996; “who’s calling” matched “Who is this?”; “not now” matched “I’m busy”; and all 745 held-out negatives stayed below threshold. In production, the hit rate on candidate-bearing turns roughly tripled against safe-cosine, at a serve precision around 86%.
Life was good. In English.
Act III — The multilingual reckoning
Then our traffic did what traffic does: it stopped being English. Hindi and Hinglish campaigns ramped, and the v3 matcher — never told anything about Hindi — didn’t fail humbly. It failed at 0.999 confidence, in the worst possible class: polarity flips. “हां जी बोलिए” (yes, go ahead) served the cached response for “नहीं, अभी समय नहीं है” (no time right now). A user saying “3 लोग हैं हम” (we’re three people) matched “मैं अकेला हूँ” (I’m alone).
The root cause took twenty minutes to find and one sentence to state: the training data contained zero Devanagari rows. The model wasn’t bad at Hindi; it had never been asked to learn it, and cross-lingual pairs landed in a region of its space where the decision boundary was noise. A model’s confidence is only meaningful inside its training distribution — outside it, the sigmoid still happily prints 0.999.
We shipped a stopgap the same week — a cross-script serving guard (never serve a prediction whose Unicode script profile doesn’t match the utterance’s) — which blocked the visible cross-script disasters but couldn’t touch same-script errors. The real fix was a multilingual retrain.
Act IV — The base-model bake-off, or: pre-trained ≠ pre-aligned
For the multilingual base we short-listed two open-source candidates and fine-tuned both on identical data (~4.6k pairs at that point), evaluated on a held-out set split by call so no conversation leaks across splits:
- MuRIL (Google) — BERT pre-trained on 17 Indian languages. The “obvious” choice.
- mmarco-mMiniLMv2-L12-H384-v1 — a multilingual MiniLM already fine-tuned as a relevance cross-encoder on mMARCO (machine-translated MS MARCO, 14 languages).
| base | test AUC | recall @ FPR ≤ 2% |
|---|---|---|
| MuRIL (raw masked-LM) | 0.90 | 0.00 |
| mMiniLMv2 (mMARCO cross-encoder) | 0.94 | 0.43 |
MuRIL’s row is the one worth staring at. AUC 0.90 means it ranks pairs decently — but at any threshold tight enough for our FPR ceiling, it recalled nothing. A raw masked-LM head must learn the entire concept of “these two utterances are interchangeable” from your fine-tuning data, and a few thousand pairs cannot carve a boundary sharp enough to operate at sub-1% FPR. The mMARCO model had already spent its pre-training learning a calibrated relevance boundary — our fine-tune only had to move it, not create it. It even fixed all five of the real production Hindi FPs with zero Hindi fine-tuning rows, purely from multilingual relevance pre-training.
Lesson: for small-data fine-tunes, the head you inherit matters more than the languages you inherit. It’s the same lesson that made v3 work (stsb-distilroberta was already a similarity judge) — we just had to relearn it against a tempting domain-match.
The bonus surprise: the mMiniLM has ~118M parameters against distilroberta’s 66M, yet runs 2.8× faster (59 ms vs 165 ms for a 20-candidate batch on two pinned CPU threads). Nearly all its parameters sit in a 250k-token embedding table — a lookup, not compute. Its transformer stack is 12 layers × 384 wide vs 6 × 768, and attention/FFN cost scales roughly quadratically with width. Parameter count is a storage metric, not a speed metric.
Act V — v4 → v4.1: the data does the work
Architecture settled, the rest was data. The final v4.1 training set (~6,900 pairs) is three deliberate layers:
Layer 1: mined production pairs — including our own false positives. We reconstructed real (utterance → prediction) decisions from logs: served hits, near-miss rejections, and the full inventory of real Hindi FPs from the incident. Mined pairs flow into validation and test; synthetic rows go to train only. You measure on reality, always.
Layer 2: intent clusters with severity-weighted negative relations. Sixteen clusters of Hindi/Hinglish utterance surfaces (affirm, decline, busy-defer, repeat-request, can’t-hear, identity-question, hedge…), each mixing Devanagari and romanized spellings — transliteration equivalence comes free when “हाँ जी” and “haan ji” share a cluster. Negatives pair across clusters, and each negative relation is weighted by how badly its confusion hurt us in production: affirm↔decline gets 60 pairs (the polarity flip), agreement↔deferral 35 (a real incident), identity-question↔affirm 30. Eight relation pairs are deliberately excluded — greeting↔affirm, defer↔callback-time — because their equivalence is context-dependent, and teaching them as negatives would be teaching lies.
Layer 3: concept grids — the v4.1 breakthrough. v4 trained on layers 1–2 still failed 23 of a 100-pair adversarial suite, and the failures had a pattern: the model had memorized our surfaces, not the concepts. It knew “हाँ”≠”नहीं” but matched “चलेगा” to “नहीं चलेगा”. It knew one time-slot trap but matched “at 6 AM” to “at 6 PM”.
So we stopped writing examples and started writing generators:
- Negation frames — 27 frames × (affirmative, negated, paraphrase) × three surface languages:
("I can make it", "I can't make it", "I'll be there"),("चलेगा", "नहीं चलेगा", "चल जाएगा"). - Value slots — same frame, different values ⇒ miss; alias values (“at 6 PM” ~ “शाम 6 बजे”) ⇒ hit. Times, dates, amounts, tools, cities.
- Token swaps — “senior developer” ≠ “junior developer”, “marketing head” ≠ “marketing intern”.
- Question↔answer — “can I call you back?” ≠ “please call me back”. A question is not its answer.
410 generated rows. The adversarial suite went 23 FPs → 3 → 0 (the last step is the quantization fix below). Same architecture, same recipe.
Lesson: when a model fails a category, don’t add examples — add the axis. A grid that varies exactly one concept across many surfaces teaches the concept; a pile of one-off examples teaches the pile.
Act VI — Quantization tried to quietly undo our training
We ship int8 (dynamic quantization, ONNX Runtime, CPU). The first int8 export passed the aggregate metrics — AUC barely moved — and failed the adversarial gate: three negatives we had explicitly trained to reject came back above threshold, pairs like “रात को 8 बजे” ~ “सुबह 8 बजे” (8 PM vs 8 AM).
The mechanism generalizes. Fine-tuning against hard negatives places them just on the safe side of the boundary — that’s what “hard negative” means. Per-tensor quantization applies one scale to a whole weight matrix; outlier channels compress everyone else’s resolution, adding just enough logit noise to push borderline negatives back across. Your hardest-won training signal lives exactly where quantization noise does the most damage. Per-channel quantization (a scale per output channel) fixed all three at ~6% size cost.
Two rules came out of this: quantize per-channel, and re-run the adversarial gate on every exported artifact — not the checkpoint, the artifact. The bytes you validate must be the bytes you deploy; the threshold is recalibrated per artifact too, because every export shifts the score distribution.
The model can’t do it alone
Three non-model layers do real work in production:
- The cross-script guard stayed — demoted from stopgap to belt-and-suspenders. Even a semantically perfect cross-language match can be the wrong serve: the cached response was generated for a Hindi turn and would answer an English speaker in Hindi.
- A language allowlist. Weeks after v4.1 shipped, Urdu traffic appeared — zero training data, and same-script on both sides, so the guard was blind. It produced confident FPs (“کیا؟” → “I don’t know” at 0.996). Now any utterance carrying a non-Latin script outside the trained set bypasses the cache in microseconds, before embedding or model. On its first full day this absorbed 31% of all checked turns — a multilingual campaign had ramped overnight.
- Thresholds as economics. We launched v4.1 at 0.995 knowing the sweep (recall 0.65 @ 4.1% FPR, vs 0.59 @ 2.5% at 0.998) and reasoning about costs: an FP speaks a wrong sentence to a human; a miss costs ~1.4 s on one turn. The threshold is an env var; the decision reverses in one restart.
Results: the three-era scoreboard
| cosine-only | v3 (English SLM) | v4.1 (multilingual SLM) | |
|---|---|---|---|
| judge on short replies (“yeah”/”nah”) | inseparable (~0.7 both) | separated | separated, 3 languages of surfaces |
| recall at safe FPR | near-exact matches only | 0.42 @ 1.7% (EN) | 0.65 @ 4.1% (EN+HI+Hinglish) |
| the polarity-flip class | present | suppressed in EN, 0.999-confident in HI | zero observed in prod |
| hard FPs (share of serves, prod) | — | ~10% | ~4.5%, trending ~3% |
| 20-candidate batch (2 CPU threads) | ~1 ms (but can’t judge) | 165 ms | 26 ms p50 / 59 ms p95 |
| conversion on candidate-bearing turns | — | ~33% | 43–45% |
The first Tamil serve through v4.1 was correct (“ம்… கேளுங்க.” → “ஆம், பேசலாம்”) with zero Tamil training rows — multilingual relevance pre-training carrying recall to languages we never taught, while the concept grids kept the danger classes suppressed. Not every language gets that for free (see: Urdu), which is why the allowlist exists.
The lesson we didn’t want: how good models silently vanish
Remember the fail-open design from Act II? It bit us twice. Both times, the model artifact’s download credentials had expired; a host rebuild later, the fleet was quietly serving on cosine-only fallback — the Act I system — and the only symptom was one WARNING line at boot. Days of “the matcher seems off” reports later, the cause was a credential timestamp.
Every fail-open path needs a fail-loud alert. If your system degrades gracefully by design, an alert on the degradation is part of the feature. It costs one log-matching rule, and it converts “discovered by a confused test call” into “paged at deploy time.”
What’s next
The production FP audit is itself the next training set: every judged serve becomes a labeled row, every FP class becomes a grid. The v4.2 backlog, each item traceable to a specific production example: numeral↔word aliases (“10th standard” ~ “tenth grade” — currently over-suppressed), compound day+time slots (“today at 2” vs “tomorrow at 2” — one slipped through at 1.0000), fragment suppression (“Actually,” must never match a content answer), voicemail-greeting rejection, and an English hedge↔commit relation to mirror the Hindi one.
Which is the meta-lesson of the three versions: the model is a snapshot; the data pipeline is the asset. Mine production, judge it, turn judgments into grids, retrain, re-gate, ship, repeat. The second lap around that loop took a tenth the time of the first — and the third is mostly waiting for a GPU.
Current model: fine-tuned cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, int8 per-channel, ONNX Runtime CPU. Training: single T4, ~155 s per run on ~6.9k pairs. Serving: ~30 ms per 20-candidate batch on 2 CPU threads at a 0.995 threshold, behind a cosine shortlist, a cross-script guard, and a language allowlist.
Leave a Reply