Your Classifier Shouldn’t Write Prose: Decisions APIs for Voice Agents
We moved the decision “should this call end now?” from a general-purpose LLM (gpt-4o-mini) to Jev, a model built only to choose between options. The decision got about 13 times faster, 3.7 times cheaper and more careful about cutting people off. The bigger change was in how we reason about the decision. Instead of a yes/no from a text generator, we now get a probability for each option, and we can see exactly where the model is unsure.
Ending a call is a decision, not a button
Our voice agents run as a pipeline: speech-to-text, an LLM that writes the reply, then text-to-speech. The reply LLM has no end-call button. It only writes the next thing to say.
So after every agent turn, a separate check reads the conversation and answers one question: is this call over? That check runs several times on every call, inside a live phone conversation. It has two ways to be wrong, and they are not equally bad:
- Too early: the agent hangs up on a live customer, sometimes right after asking them a question. This is the failure customers remember and complain about.
- Too late: the agent lingers, says goodbye twice, or waits for the customer to hang up. Awkward, but nobody loses anything.
Any tool we use for this has to be fast, cheap per call, and let us lean towards “keep talking” when it is unsure.

Why a general LLM was the wrong tool
We used to send each check to gpt-4o-mini: about 3,400 tokens of conversation, call state and rules, and it wrote back a short JSON verdict. It mostly worked. But we were asking a text generator to act as a switch, and that showed up in four ways.
- It was slow for a phone line. A check took about 1.4 seconds at the median and nearly 3 seconds at p95. Most of that is generating text we only reduce to one bit.
- It cut people off. In an audit of 170 calls it ended, 15 (8.8%) were serious premature hangups. The agent had just asked a question, said “one minute”, or left the customer’s question unanswered.
- It was not stable. Asked the same 800 decisions three times, it changed its answer on 25 of them.
- It gave us nothing to tune. The output was just
hangup: trueorfalse. A narrowly-yes and a clearly-yes looked the same. To make it more careful, all we could do was rewrite the prompt and hope.
The last point mattered most. We had no way to say “only hang up when you are really sure”. That is the one setting this decision needs.
Why a decisions API fits this kind of task
A decisions API turns the problem around. You do not ask a model to write an answer and then parse it. You give it a state, a question and a fixed set of options, and it scores each option. Jev from TypeSafe works this way. This is our hangup check, trimmed:
{
"model": "jev-1.14.0",
"state": "<the conversation and call state as JSON>",
"questions": {
"hangup": {
"type": "choice",
"instructions": "<your question>",
"criteria": {
"HANGUP": "<what 'the call is over' means>",
"CONTINUE": "<what 'still going' means>"
}
}
}
}
The answer comes back already structured:
{ "choice": "CONTINUE", "probabilities": { "HANGUP": 0.12, "CONTINUE": 0.88 }, "confidence": 0.76 }
For a decision like this, that shape helps in five ways:
- No parsing, no bad output. The answer is always one of the options you defined. There is no malformed JSON and no extra field to handle.
- Fast. Nothing is generated, so there is nothing to wait for. Our checks dropped from about 1.4 s to about 0.1 s at the median.
- Cheap. You pay for input tokens only; output is free.
- The options carry meaning. Each option has its own description, so you say concretely what HANGUP and what CONTINUE mean. When we described CONTINUE only as “otherwise”, turns that sounded finished but were not got pulled towards HANGUP.
- You get a number, not a verdict. Every option comes with a probability. That is what makes the decision tunable.
The same shape fits most yes/no or pick-one decisions inside a live system: is this a voicemail, which team should take this ticket, did the customer agree, which language is this. If your code will branch on the answer, a decisions API is usually a better fit than a chat model.
What the probability in Jev actually is
For a choice question, Jev returns a probability for every option, and they add up to 1. With two options, one number tells you everything: if p(HANGUP) is 0.12, then p(CONTINUE) is 0.88. The choice field is just the option with the highest probability.
What the number means: Jev is trained so that its probabilities are calibrated. Across many answers, things given 0.2 should happen about 20% of the time. That is a claim about groups of answers, not a promise about any one call. It is why you can set a threshold on it and predict roughly how often you will be wrong.
Jev also returns a confidence field. It looks like a second signal, but it is not. For a choice question it is defined from the top probability alone:
confidence = (p_max − 1/n) / (1 − 1/n)
With two options this is just |2p − 1|. We checked it on 800 of our own answers, and it matched every time. A confidence of 0.8 is the same as a top probability of 0.9. Checking both adds nothing, so we only look at the probability of the option we are about to act on.
A two-option probability reads naturally as three zones. Near 0 or 1, the model is sure. Near 0.5, it is telling you the case is genuinely unclear. The drawing below shows what we found in each zone when we replayed 800 calls gpt-4o-mini had ended.
The low band is where Jev catches the old model’s mistakes. The middle band is where it hesitates, and that is where to look.

How to navigate the probabilities
A probability is only useful if you know what to do with it. These are the rules we ended up with.
1. Decide which mistake is worse, then set the bar. Gate on the probability of the action you will take. If hanging up wrongly is expensive, require a high p(HANGUP). If a wrong “continue” is cheap, let it win near the middle. The threshold is a cost setting, not a model setting.
2. Find the threshold on your own data. Replay real cases, group them by probability, and read the disagreements in each group. On our final turns, the low band held the real cut-offs and the middle band held goodbyes Jev was hesitant about. That told us more than any overall accuracy number.
3. Test the turns where nothing should happen. Final turns are the easy part. Most checks happen mid-conversation, where the right answer is “keep going”. When we replayed earlier turns, Jev wanted to hang up on live conversations in 166 of 800 calls at p ≥ 0.2. A threshold tuned only on final turns would have hidden that.
4. If the errors share a shape, fix the question, not the threshold. Moving the threshold only trades one error for the other. Our mid-call errors all looked alike: the agent confirmed something or said what happens next, with no farewell. We rewrote the question around one idea, has the agent taken leave of the person? Substantive early cuts fell from 59 calls to 31. A blind judge then found 5 premature hangups from Jev against 27 from gpt-4o-mini on the same 800 calls.
5. A high probability answers the question you asked, not the one you meant. TypeSafe’s own docs show 0.99 for “approve transfer” on “Um, the transfer thing… yeah, go ahead I guess.” The intent is clear; the confirmation is not. Write the question to match the action that follows.
6. Treat the middle band as your review queue. We log p(HANGUP) on every check. Calls with values near 0.5 are where we look first when something feels off.
7. Pin the model version. A threshold is tuned against one model. An alias like jev-latest can move to a new release and shift your numbers, so we send jev-1.14.0.
8. Ask one question, not five. We tried splitting the decision into several smaller questions. Each one paid for the state and rules again, so the cost saving disappeared with no gain in accuracy.
What it solved
Jev now makes every hangup decision in production. In our first full measurement it decided 3,449 checks on 686 calls, while gpt-4o-mini ran alongside it for comparison.
| gpt-4o-mini | Jev | |
|---|---|---|
| Median latency per check | 1,413 ms | 108 ms |
| p95 latency per check | 2,971 ms | 171 ms |
| Premature hangups (blind-judged, same 800 calls) | 27 | 5 |
| Errors | — | 0 of 3,449 checks |
| Cost per check | 1× | about 0.27× |
| What you get back | yes / no | a probability per option |
The two models agreed on 97.4% of checks. Of the 316 calls Jev ended, we found none that it ended wrongly. Jev kept 13 calls (1.9%) going that gpt-4o-mini would have cut off with the customer still talking, 5 of them with a question still open. It stayed on too long in 4 calls (0.6%), which is the cheap kind of mistake.
What surprised us
The model was the easy part. Both surprises came from the system around it.
The old model hadn’t fully left. Another feature still read a field from gpt-4o-mini’s answer, so for a while we paid for both models. A side-by-side test proves the new model’s answers. It does not prove the old model’s side effects are gone.
Speed exposed a timing bug. Part of our call flow had quietly relied on the old model being slow. Once decisions arrived in 0.1 s instead of 1.4 s, a small race that had always been there started showing up more often. We fixed the flow, not the model. When one step gets much faster, look for the steps that relied on it being slow.
Takeaways
- If your code branches on the answer, use a model that chooses. A chat model asked for yes or no is slower, costs more and gives you nothing to tune.
- A probability per option is the real upgrade. It lets you decide how sure the model must be before you act, and it shows you where the hard cases are.
- Confidence is the probability in another form. Gate on the probability of the option you will act on.
- Set thresholds by the cost of each mistake, and test the turns where nothing should happen.
- When the errors look alike, change the question before you move the threshold.
- A faster decision changes the timing of everything after it. Check what was relying on the old delay.

Leave a Reply