# The 30-Millisecond Judge: Training a Small Model to Decide When Our Voice Agent Is Allowed to Cheat

##### *Three matchers in, we learned the real lessons the hard way: cosine similarity that couldn't tell "yeah" from "nah", a base model that ranked beautifully but couldn't decide, a quantization step that resurrected our worst bugs — and why the data pipeline, not the model, turned out to be the asset.*

## The bug that made this a story

Our voice agents make outbound phone calls. Latency is the whole game: if the agent takes two seconds to respond, humans talk over it, repeat themselves, or hang up. One of our best latency tricks is a **predictive cache**: while the agent is speaking, a background LLM predicts the user's most likely next utterances ("Yes", "Who is this?", "I'm busy right now") and pre-generates full responses for each. When the user actually speaks, if their utterance matches a prediction, we serve the cached response in ~80 ms instead of paying ~1.5 s for LLM + TTS.

The catch is the word *matches*. Serving a cached response to the wrong utterance isn't a latency win — it's the agent confidently answering a question nobody asked.

One day a user said **"Sure."** and our agent responded as if they'd said **"नहीं, अभी busy हूँ"** *("no, I'm busy right now")* — because our match gate scored that pair at 0.999. A user said **"हां जी बोलिए"** *("yes, go ahead")* and got the response for **"नहीं, अभी समय नहीं है"** *("no, I don't have time")*. Same polarity flip, same absurd confidence.

That incident is the middle of the story, not the start. The gate that failed there was already our second matching system — and understanding why it failed the way it did requires the whole journey: **cosine similarity → a fine-tuned English cross-encoder (v3) → a multilingual one (v4.1)**. Each stage earned the next.

## Act I — Cosine similarity, and where it tops out

The first version of the matcher was the obvious one: embed the incoming utterance, embed the predicted utterances, serve the best candidate if cosine similarity clears a threshold. One small sentence-embedding model, a few milliseconds, no training required.

It worked — for a narrow definition of "worked." Two problems surfaced almost immediately, and they're both fundamental to how phone conversations look as text:

**Phone replies are short, and short replies all embed alike.** The most common user turns are things like "Yes.", "Yeah.", "Nah.", "Who is this?", "Not now." Sentence embeddings place these in a tight little cluster: **"Yeah" vs "Nah" — opposite meanings — score ~0.7. "Yeah" vs "Yes, go ahead" — same meaning — also ~0.7.** The true-match and false-match score distributions overlap almost completely on short text. There is no threshold that separates them, because the information that distinguishes them (one negation token) is a rounding error in a pooled embedding.

**So the threshold gets cranked, and recall starves.** The only safe operating point was ~0.9 with top-1 matching — which in practice means near-exact wording. "Not now" wouldn't match "I'm busy right now." "Go on" wouldn't match "Please continue." The cache hit on the turns where the LLM predicted the user's *exact words* and missed the far larger set where it predicted their *meaning*. We were leaving most of the latency win on the table to stay safe — and still occasionally serving a wrong short-reply match that snuck over 0.9.

The conclusion wasn't "embeddings are bad." It was a division of labor that survived every later version: **embeddings are a fine shortlist; they cannot be the judge.** Retrieval and judgment are different jobs. We kept cosine to fetch candidates and went looking for a judge.

(Why not an LLM judge? 300–800 ms even on a fast hosted model — you'd spend the latency budget deciding whether you're allowed to save it. The judge had to run in tens of milliseconds on CPU, next to the audio pipeline.)

## Act II — v3: a small English cross-encoder earns the job

The right architecture for "are these two utterances interchangeable?" is a **cross-encoder**: feed the pair `(actual_utterance, predicted_utterance)` through one small transformer that attends *across* both texts, output a single probability. Unlike bi-encoder embeddings, the model sees both sentences at once — a negation token in one can attend to its counterpart in the other. With int8 quantization, a 6-layer model scores a 20-candidate shortlist in a few dozen milliseconds on two CPU threads.

We started from `cross-encoder/stsb-distilroberta-base` — a model already fine-tuned for sentence-similarity judgment — and taught it our domain in two data passes:

- **~2,900 real pairs mined from production logs**: actual (utterance → prediction) decisions, labeled. Real pairs anchor the distribution.
- **~1,700 synthetic pairs in equivalence clusters**: paraphrase sets for the intents that matter on calls (agreement, refusal, deferral, identity questions, repeat requests…), plus a hand-picked set of **danger negatives** — the "yeah"↔"nah" class that cosine could never see.

Two deployment choices from this era proved more valuable than the model itself:

1. **Shadow mode first.** The matcher scored every lookup while cosine still made the serving decision, logging what it *would* have done. A week of shadow traffic calibrated the threshold on reality instead of the test set, for free.
2. **Fail-open, always.** If the model artifact is missing or scoring fails, the cache degrades to conservative cosine behavior rather than blocking calls. (This choice ages interestingly — see the ops lesson at the end.)

The error asymmetry defined the operating point, and it's worth stating because it shaped every version since: a **false positive speaks a wrong response to a live human** — user-visible harm. A **false negative just falls back to the LLM** — the latency win is lost, nothing bad is said. So we run at extreme thresholds (0.998 in probability space for v3) and optimize recall *under* an FPR ceiling, never accuracy.

v3's results, against the cosine baseline it replaced: **recall 0.42 at 1.7% FPR** on held-out real pairs — versus cosine's effectively-exact-match recall at any safe threshold. Every known hole flipped: "go on" → "Please continue" went from 0.001 to 0.9996; "who's calling" matched "Who is this?"; "not now" matched "I'm busy"; and all 745 held-out negatives stayed below threshold. In production, the hit rate on candidate-bearing turns roughly tripled against safe-cosine, at a serve precision around 86%.

Life was good. In English.

## Act III — The multilingual reckoning

Then our traffic did what traffic does: it stopped being English. Hindi and Hinglish campaigns ramped, and the v3 matcher — never told anything about Hindi — didn't fail humbly. It failed at **0.999 confidence**, in the worst possible class: polarity flips. "हां जी बोलिए" (*yes, go ahead*) served the cached response for "नहीं, अभी समय नहीं है" (*no time right now*). A user saying "3 लोग हैं हम" (*we're three people*) matched "मैं अकेला हूँ" (*I'm alone*).

The root cause took twenty minutes to find and one sentence to state: **the training data contained zero Devanagari rows.** The model wasn't bad at Hindi; it had never been asked to learn it, and cross-lingual pairs landed in a region of its space where the decision boundary was noise. A model's confidence is only meaningful inside its training distribution — outside it, the sigmoid still happily prints 0.999.

We shipped a stopgap the same week — a **cross-script serving guard** (never serve a prediction whose Unicode script profile doesn't match the utterance's) — which blocked the visible cross-script disasters but couldn't touch same-script errors. The real fix was a multilingual retrain.

## Act IV — The base-model bake-off, or: pre-trained ≠ pre-aligned

For the multilingual base we short-listed two open-source candidates and fine-tuned both on identical data (~4.6k pairs at that point), evaluated on a held-out set split by call so no conversation leaks across splits:

- **MuRIL** (Google) — BERT pre-trained on 17 Indian languages. The "obvious" choice.
- **mmarco-mMiniLMv2-L12-H384-v1** — a multilingual MiniLM *already fine-tuned as a relevance cross-encoder* on mMARCO (machine-translated MS MARCO, 14 languages).

| base | test AUC | recall @ FPR ≤ 2% |
|---|---|---|
| MuRIL (raw masked-LM) | 0.90 | **0.00** |
| mMiniLMv2 (mMARCO cross-encoder) | 0.94 | 0.43 |

MuRIL's row is the one worth staring at. AUC 0.90 means it *ranks* pairs decently — but at any threshold tight enough for our FPR ceiling, it recalled **nothing**. A raw masked-LM head must learn the entire concept of "these two utterances are interchangeable" from your fine-tuning data, and a few thousand pairs cannot carve a boundary sharp enough to operate at sub-1% FPR. The mMARCO model had already spent its pre-training learning a calibrated relevance boundary — our fine-tune only had to *move* it, not *create* it. It even fixed all five of the real production Hindi FPs *with zero Hindi fine-tuning rows*, purely from multilingual relevance pre-training.

**Lesson: for small-data fine-tunes, the head you inherit matters more than the languages you inherit.** It's the same lesson that made v3 work (stsb-distilroberta was already a similarity judge) — we just had to relearn it against a tempting domain-match.

The bonus surprise: the mMiniLM has ~118M parameters against distilroberta's 66M, yet runs **2.8× faster** (59 ms vs 165 ms for a 20-candidate batch on two pinned CPU threads). Nearly all its parameters sit in a 250k-token embedding table — a lookup, not compute. Its transformer stack is 12 layers × 384 wide vs 6 × 768, and attention/FFN cost scales roughly quadratically with width. **Parameter count is a storage metric, not a speed metric.**

## Act V — v4 → v4.1: the data does the work

Architecture settled, the rest was data. The final v4.1 training set (~6,900 pairs) is three deliberate layers:

**Layer 1: mined production pairs — including our own false positives.** We reconstructed real (utterance → prediction) decisions from logs: served hits, near-miss rejections, and the full inventory of real Hindi FPs from the incident. Mined pairs flow into validation and test; synthetic rows go to train only. **You measure on reality, always.**

**Layer 2: intent clusters with severity-weighted negative relations.** Sixteen clusters of Hindi/Hinglish utterance surfaces (affirm, decline, busy-defer, repeat-request, can't-hear, identity-question, hedge…), each mixing Devanagari and romanized spellings — transliteration equivalence comes free when "हाँ जी" and "haan ji" share a cluster. Negatives pair *across* clusters, and **each negative relation is weighted by how badly its confusion hurt us in production**: affirm↔decline gets 60 pairs (the polarity flip), agreement↔deferral 35 (a real incident), identity-question↔affirm 30. Eight relation pairs are deliberately *excluded* — greeting↔affirm, defer↔callback-time — because their equivalence is context-dependent, and teaching them as negatives would be teaching lies.

**Layer 3: concept grids — the v4.1 breakthrough.** v4 trained on layers 1–2 still failed 23 of a 100-pair adversarial suite, and the failures had a pattern: the model had memorized our *surfaces*, not the *concepts*. It knew "हाँ"≠"नहीं" but matched "चलेगा" to "नहीं चलेगा". It knew one time-slot trap but matched "at 6 AM" to "at 6 PM".

So we stopped writing examples and started writing *generators*:

- **Negation frames** — 27 frames × (affirmative, negated, paraphrase) × three surface languages: `("I can make it", "I can't make it", "I'll be there")`, `("चलेगा", "नहीं चलेगा", "चल जाएगा")`.
- **Value slots** — same frame, different values ⇒ miss; alias values ("at 6 PM" ~ "शाम 6 बजे") ⇒ hit. Times, dates, amounts, tools, cities.
- **Token swaps** — "senior developer" ≠ "junior developer", "marketing head" ≠ "marketing intern".
- **Question↔answer** — "can I call you back?" ≠ "please call me back". A question is not its answer.

410 generated rows. The adversarial suite went **23 FPs → 3 → 0** (the last step is the quantization fix below). Same architecture, same recipe.

**Lesson: when a model fails a category, don't add examples — add the axis.** A grid that varies exactly one concept across many surfaces teaches the concept; a pile of one-off examples teaches the pile.

## Act VI — Quantization tried to quietly undo our training

We ship int8 (dynamic quantization, ONNX Runtime, CPU). The first int8 export passed the aggregate metrics — AUC barely moved — and failed the adversarial gate: **three negatives we had explicitly trained to reject came back above threshold**, pairs like "रात को 8 बजे" ~ "सुबह 8 बजे" (8 PM vs 8 AM).

The mechanism generalizes. Fine-tuning against hard negatives places them *just* on the safe side of the boundary — that's what "hard negative" means. Per-tensor quantization applies one scale to a whole weight matrix; outlier channels compress everyone else's resolution, adding just enough logit noise to push borderline negatives back across. **Your hardest-won training signal lives exactly where quantization noise does the most damage.** Per-channel quantization (a scale per output channel) fixed all three at ~6% size cost.

Two rules came out of this: **quantize per-channel, and re-run the adversarial gate on every exported artifact — not the checkpoint, the artifact.** The bytes you validate must be the bytes you deploy; the threshold is recalibrated per artifact too, because every export shifts the score distribution.

## The model can't do it alone

Three non-model layers do real work in production:

1. **The cross-script guard stayed** — demoted from stopgap to belt-and-suspenders. Even a semantically perfect cross-language match can be the wrong *serve*: the cached response was generated for a Hindi turn and would answer an English speaker in Hindi.
2. **A language allowlist.** Weeks after v4.1 shipped, Urdu traffic appeared — zero training data, and same-script on both sides, so the guard was blind. It produced confident FPs ("کیا؟" → *"I don't know"* at 0.996). Now any utterance carrying a non-Latin script outside the trained set bypasses the cache in microseconds, before embedding or model. On its first full day this absorbed **31% of all checked turns** — a multilingual campaign had ramped overnight.
3. **Thresholds as economics.** We launched v4.1 at 0.995 knowing the sweep (recall 0.65 @ 4.1% FPR, vs 0.59 @ 2.5% at 0.998) and reasoning about costs: an FP speaks a wrong sentence to a human; a miss costs ~1.4 s on one turn. The threshold is an env var; the decision reverses in one restart.

## Results: the three-era scoreboard

|  | cosine-only | v3 (English SLM) | v4.1 (multilingual SLM) |
|---|---|---|---|
| judge on short replies ("yeah"/"nah") | inseparable (~0.7 both) | separated | separated, 3 languages of surfaces |
| recall at safe FPR | near-exact matches only | 0.42 @ 1.7% (EN) | 0.65 @ 4.1% (EN+HI+Hinglish) |
| the polarity-flip class | present | suppressed in EN, **0.999-confident in HI** | **zero observed in prod** |
| hard FPs (share of serves, prod) | — | ~10% | ~4.5%, trending ~3% |
| 20-candidate batch (2 CPU threads) | ~1 ms (but can't judge) | 165 ms | **26 ms p50 / 59 ms p95** |
| conversion on candidate-bearing turns | — | ~33% | 43–45% |

The first Tamil serve through v4.1 was correct ("ம்... கேளுங்க." → "ஆம், பேசலாம்") with zero Tamil training rows — multilingual relevance pre-training carrying recall to languages we never taught, while the concept grids kept the danger classes suppressed. Not every language gets that for free (see: Urdu), which is why the allowlist exists.

## The lesson we didn't want: how good models silently vanish

Remember the fail-open design from Act II? It bit us twice. Both times, the model artifact's download credentials had expired; a host rebuild later, the fleet was quietly serving on cosine-only fallback — the Act I system — and the only symptom was one WARNING line at boot. Days of "the matcher seems off" reports later, the cause was a credential timestamp.

**Every fail-open path needs a fail-loud alert.** If your system degrades gracefully by design, an alert on the degradation is part of the feature. It costs one log-matching rule, and it converts "discovered by a confused test call" into "paged at deploy time."

## What's next

The production FP audit is itself the next training set: every judged serve becomes a labeled row, every FP class becomes a grid. The v4.2 backlog, each item traceable to a specific production example: numeral↔word aliases ("10th standard" ~ "tenth grade" — currently over-suppressed), compound day+time slots ("today at 2" vs "tomorrow at 2" — one slipped through at 1.0000), fragment suppression ("Actually," must never match a content answer), voicemail-greeting rejection, and an English hedge↔commit relation to mirror the Hindi one.

Which is the meta-lesson of the three versions: **the model is a snapshot; the data pipeline is the asset.** Mine production, judge it, turn judgments into grids, retrain, re-gate, ship, repeat. The second lap around that loop took a tenth the time of the first — and the third is mostly waiting for a GPU.

---

*Current model: fine-tuned `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1`, int8 per-channel, ONNX Runtime CPU. Training: single T4, ~155 s per run on ~6.9k pairs. Serving: ~30 ms per 20-candidate batch on 2 CPU threads at a 0.995 threshold, behind a cosine shortlist, a cross-script guard, and a language allowlist.*

---