What happens in the six seconds after your DialNexa AI agent says “let me get you a person”

This post has two halves. The Experience is what our call transfer actually does, and why the obvious approach fails. The Engineering is the deep dive: the races, the audio pipeline, the four separate fights we had with latency. Read one, or both.

The Experience

There’s a moment in every AI phone call where the right answer is “let me get you a person.”

Everything up to that moment is the part people demo: the speech recognition, the model, the voice. What happens after it is where products quietly die. The customer has just been promised a human. From that second on, every bit of silence is a withdrawal from an account you spent the whole call filling.

Most systems handle this with one line of configuration: dial a number, hope someone picks up. Here’s what we built instead.

Five phones ring at once

The naive transfer dials one number. If that person is busy, in a meeting, or in a tunnel, your customer listens to ringing for forty seconds and then gets an apology. You’ve converted a warm lead into a bad memory.

So we don’t dial a number. We run a hunt.

Parallel hunt rings every number on the list simultaneously. Five phones light up at once. Whoever picks up first gets the customer, and the instant they do, every other phone stops ringing. Nobody on your team answers to dead air.

Priority hunt works down the list in order, with a gap you set. Ring the account owner. Wait five seconds. Add the next person. Then the next. It escalates without ever skipping the person who should have taken the call.

Either way, the customer is never listening to a phone ring. They’re still talking to the AI.

The rep gets briefed. The customer never hears it.

A cold handoff is nearly as bad as no handoff. Your rep picks up to a live stranger, knows nothing about them, and burns the first thirty seconds asking questions the AI already has the answers to.

The obvious fix, announcing the context to the rep, has an obvious problem. In a normal conference call, the customer hears you briefing the rep. Which is worse than saying nothing.

So we brief the rep on their own line, before they’re ever connected to the customer. One sentence, in their ear, in total privacy:

"Priya Sharma is calling about the delayed order on invoice 4471 and has already been offered a partial refund."

And it isn’t a form letter. It’s written fresh for that call, from what the customer actually said, while the phones are still ringing. By the time someone picks up, the sentence is already waiting for them.

Meanwhile the customer is mid-sentence with the AI, which has no idea any of this is happening. No hold music. No “one moment please.” No dead air at all.

Nobody waits, and nobody talks over anybody

Once the two are connected, we’re sitting in the middle of two live phone calls, which means we own every millisecond between them.

Audio lag never shows up as a complaint. It shows up as two people talking over each other, both of them mildly irritated, and neither of them knowing why the call felt awkward.

We capped it. There is a hard ceiling on how far behind the audio can ever fall, enforced by design rather than by hope: when audio starts to back up, we drop the stale piece rather than delay the fresh one. A missing fraction of a second is inaudible. Accumulated delay ruins a conversation.

We also alarm on gaps as short as a tenth of a second, not the half-second that would be comfortable to engineer against, but the point where a human actually notices something is off.

We can tell you what actually happened

Most transfer systems report two things: it worked, or it didn’t. That’s not enough to run a team on.

For every transfer, you get a record of every number we dialed, and for each one:

how long it rang
whether it was answered, busy, rejected, or never picked up
the actual reason, when the network gave us one
who won, and how long they talked

And the outcome isn’t “success” or “failure.” It’s one of eleven specific results, because “all three of your reps’ phones were busy” is a completely different problem from “all three declined”, which is different again from “nobody answered.”

One of those tells you to retry in five minutes. One tells you to have a conversation with your team. One tells you to hire someone. Collapsing them into “failed” throws away the only information that was useful.

We’re also strict about the wording. You’ll never see a raw error code in a report. You’ll see “Customer call ended before bridge could complete.”

“Someone answered” is not a result

Here’s the one we’re most opinionated about.

A rep answering the phone tells you almost nothing. They can answer while muted. They can answer in a lift. They can answer, put the phone down on a desk, and walk away. Every one of those is a lost customer that a normal system will proudly log as a successful transfer.

So we don’t trust the answer. We listen for a human voice on the rep’s line, and the transfer isn’t marked connected until we hear one. Not one loud noise, but a sustained third of a second of actual speech, because the beginning of any phone connection is full of clicks and artifacts that would fool a lazier check.

And if it goes wrong, the AI catches the call

This is the part that only works because of a decision we explain under The Engineering: we never actually leave the call.

If the rep is silent, or drops before speaking, the customer doesn’t get dumped into a dead line or dial tone. We hang up the rep’s leg, the AI wakes back up, and it picks the conversation up again. And it knows specifically what went wrong, so it can say something true about it instead of a generic apology.

The mirror case is handled the opposite way on purpose. If the rep did speak and then hangs up, we end the whole call. The conversation with the human is genuinely over, and dropping someone back to a bot after a real human conversation is worse than a clean goodbye.

What it looks like from the outside

The customer asks for a person. About six seconds later, there’s a person, who already knows their name and why they called.

They don’t hear the five phones that rang. They don’t hear the sentence that got written while those phones were ringing. They don’t hear the fraction of a second we threw away on purpose to make sure a human voice was the first thing they heard.

That’s the whole point. The best handoff is the one nobody notices.

The Engineering

Everything above, with the lid off.

The race nobody thinks about

“Whoever picks up first wins” is a distributed-systems problem in a trench coat.

Two reps can answer inside the same handful of milliseconds. Both of their phone legs race to claim the customer, and if both succeed you have two humans and one caller in a very awkward conference.

Winner election is a Redis lock with a check-twice pattern. First, before touching the lock at all, we take a cheap snapshot of the transfer and ask whether the outcome is already decided. Then we acquire the lock with a single atomic set-if-not-exists carrying a thirty second expiry, keyed on the conference name. Then, critically, we re-read the state from Redis inside the lock and ask the same question again, because two legs can both sail past that first check before either of them acquires anything.

The pre-lock check isn’t an optimization, it’s a correctness requirement. Because we hold the lock all the way from winner-claim through briefing, a legitimate winner re-confirming itself later in the flow would fail to re-acquire its own lock and get rejected as an impostor. The snapshot lets the true winner recognize itself without touching the lock at all.

There’s a subtler decision buried in when we claim. Plivo posts to an answer URL the instant a rep picks up, but doesn’t open the media stream until any greeting has finished playing. If you wait for the stream to claim your winner, as we did originally, and the rep hangs up during the greeting, you never learn they answered at all. That transfer gets filed as “nobody picked up,” which is a lie your dashboard tells you forever.

So we claim eagerly, at the answer URL, and arm a watchdog for the stream that should follow.

The briefing is a topology problem, not a mixing problem

The whole implementation is two instructions of telephony markup on the rep’s own leg: speak the brief, then open a bidirectional media stream back to us.

They execute in order, and that ordering is the feature. The speak instruction runs to completion before the stream opens. So the briefing isn’t inaudible to the customer because we muted anything. It’s inaudible because at that moment no audio path between the two legs exists yet. The customer is on an entirely separate connection, still talking to the AI.

The brief itself is rendered from a per-node template against the live conversation history, through a small model, constrained hard: one sentence only, no bullet points, no explanations, under twenty-five spoken words.

And it’s generated concurrently with the hunt, not after it. Both are dispatched as independent background tasks the moment the transfer begins, so the two slowest things in the flow overlap instead of queueing. The failure path degrades in stages: generated sentence, then generic-with-customer-name, then generic one-liner. The rep always hears something useful.

One consequence we had to design around: because the speak instruction delays the stream, “the stream hasn’t connected yet” is ambiguous. Slow handshake, or a rep who hung up mid-brief? So the watchdog sizes itself from the sentence. We estimate playback at roughly twelve characters per second, add fifteen seconds of headroom for the handshake and jitter, then clamp the result between twenty-five and ninety seconds.

The timeout for a network handshake is derived from the length of a sentence a language model wrote thirty seconds earlier. We’re quite fond of that.

Never leave the room

Our first working version did what every tutorial does: move the customer into a provider-hosted conference, then shut down our own media stream. It worked. We deleted it.

The commit that replaced it is titled “Rework warm transfer to keep speech-ai in the call,” and it’s the most consequential thing in this system.

When you hand the call to a provider conference, you hand away everything: your telemetry, your ability to tell whether the rep is actually talking, your ability to take the customer back when it goes wrong, and your ability to continue the conversation flow afterward. You get a working transfer and a blind spot.

So now we never leave. The customer’s connection stays open for the entire life of the call, the rep’s leg opens a second connection into the same session, and our service becomes the switchboard between two live humans. One session, three modes:

Mode	Customer hears	Rep hears	AI pipeline
`OFF`	the AI	nothing yet	running
`BRIEFING`	the AI, still talking	their private brief	running
`BRIDGED`	the rep	the customer	suspended

In bridged mode the routing is symmetric and boring, which is the goal. Customer audio goes to the rep. Rep audio goes to the customer. Transcription, the language model and speech synthesis are all suppressed for the duration.

That last suppression matters more than it looks. While bridged we hold off not just the AI’s voice but every AI-side watchdog: silence detection, voicemail detection, maximum call duration. From the pipeline’s point of view a bridged call looks exactly like a customer who has gone quiet, and it would helpfully hang up on your best rep mid-sentence.

The war on latency

We fought this four separate times.

Fix 1: A queue is not a buffer, it’s a debt

Our first bridge used unbounded queues. Perfect in testing, terrible on real calls: audio drifting further behind as the conversation went on, never recovering.

The realization is that for real-time audio, a queue never catches up. It only grows. Any hiccup, a slow socket write or a garbage-collection pause, becomes permanent latency, because there is no idle moment later in which to replay a backlog faster than real time.

Live audio sitting in a backlog has no value. So we cap the queue and evict the oldest frame. The cap is ten frames, and because the provider delivers one twenty millisecond frame per message, that works out to a two hundred millisecond hard ceiling. When the queue is full, the enqueue helper discards the stalest frame to make room for the newest.

It feels wrong to deliberately delete a human’s voice. It is completely correct. A dropped twenty millisecond frame is inaudible; eight hundred milliseconds of accumulated delay ruins the call. Latency is capped by construction rather than by hope, and a counter tells us how often we pay that price.

Fix 2: Clearing the stage before the rep walks on

At the instant of handoff, the customer’s playback buffer still holds queued AI speech. Without intervention, the first thing the customer hears after “connecting you now” is a fragment of the AI’s previous sentence, or worse, the rep’s opening words arrive queued politely behind it.

So the merge isn’t a switch, it’s a barrier. In order: freeze the sequence identifiers so no in-flight AI turn can enqueue anything more, drain all three output queues, interrupt and flush the synthesizer, clear the mark-event bookkeeping and output sequence state, reset the playback flag, kill ambient noise, and send an explicit clear-audio instruction to the customer’s stream.

And then the part that took a real production call to discover: we throw away the rep’s first quarter of a second. Frames already in flight can land in the playback buffer ahead of the clear taking effect, so rather than risk the rep’s first syllable being sandwiched into stale AI audio, we gate rep-to-customer audio for two hundred and fifty milliseconds and discard whatever arrives in that window. Deliberately dropping data to protect an experience is the whole job.

Fix 3: Sockets don’t know what time it is

A socket will cheerfully accept fifty frames in a millisecond. The phone on the other end plays them at twenty milliseconds each. So a burst doesn’t arrive early. It arrives as a full second of latency, permanently.

So we put a wall clock in front of every write. The pacer infers each frame’s true duration from the payload itself, reading its encoded length, its sample rate, and whether the format uses one byte per sample or two, then sleeps as needed to stay on schedule. And when it falls more than two hundred and fifty milliseconds behind, it drops the frame outright and resets its own clock to now rather than replaying the backlog.

Same philosophy as the queue cap, one layer down: prefer a hole in the audio over a delay in the audio. Every threshold is environment-tunable, because the right answer differs between a fibre-connected office phone and a mobile on a rural tower.

Fix 4: Telephony forwarders, or, geography is real

Our conversational brain wants to live where the models and the GPUs are. Carrier media wants to terminate close to the carrier. Those are rarely the same building, and a network connection doesn’t care about your architecture diagram. It charges you the physical distance, twice, on every frame.

So we split the roles. A deliberately thin telephony forwarder runs at the network edge, terminates the provider’s connection, and pairs it with one to the orchestrator. Two loops, nothing else: one reads from the provider and writes to the orchestrator, the other reads from the orchestrator and writes back to the provider. There is no parsing on the hot path at all. The forwarder inspects only those messages whose raw text already contains a media or play-audio marker, and only to count them.

The part that matters for transfers is that both legs must take the same road. Every warm-transfer callback address resolves through a single function, which substitutes the public forwarder’s hostname whenever the process is running in the orchestrator role and a forwarder is configured. So the rep’s answer URL and the rep’s audio connection both point at the same public forwarder the customer’s leg is already using, and both halves of the conversation traverse identical network geography.

Asymmetric paths mean one person is consistently two hundred milliseconds behind the other, which humans don’t perceive as lag. They perceive it as rudeness.

The forwarder is also where we watch for trouble, and one comment in that file tells you how seriously we take this. The gap alarm was originally set at five hundred milliseconds. We lowered it to one hundred, because gaps of a tenth of a second are audible on a phone call even though they never show up in a five-hundred-millisecond log. Half a second is a comfortable engineering threshold. A tenth of a second is what a person notices. We alarm on the second one.

Bonus round: a bridge across two processes

We run multiple workers per host, and the operating system hands an incoming connection to whichever worker accepts it first, with no affinity whatsoever. The rep’s connection routinely lands on a worker that has never heard of this call.

So every session publishes its own address. A worker identity of hostname plus process id goes into Redis under a per-call key, and into a local file, because warm transfer surviving a brief Redis flap is worth one small write per call.

When a rep’s connection lands on the wrong worker, it looks up the resident worker and becomes a proxy over three publish-subscribe channels:

Channel	Carries
customer-to-rep	raw audio frames, one direction
rep-to-customer	raw audio frames, the other direction
control	lifecycle messages: connect, acknowledge, decline, disconnect, teardown

Audio moves on the binary Redis client, deliberately: decoding raw telephony audio as text corrupts every frame it touches, silently. And because a two-hop audio path deserves scrutiny, each side reports per-interval deltas every two seconds covering frames published, frames received, write errors and pacing yields. Deltas rather than running totals, so a human can read the cadence at a glance instead of doing mental subtraction during an incident.

Proving a human actually spoke

We claimed above that we verify a real voice. Here’s the detection, and the boring part is the hard part.

The audio codec used across most of the phone network encodes a zero-amplitude sample as one of two possible bytes, depending on a sign-bit convention that varies by carrier. Both our provider and most mobile networks send a mix of the two during silence and comfort noise. So we treat any byte within a narrow band of the nearer of those two centres as silent, and call a frame speech-like only when at least forty percent of its bytes fall outside that band. Two silence centres, because different carriers disagree about which byte means nothing. Miss that and half your network reports permanent speech.

The second lesson: one loud frame is not a person. The first frames after a greeting ends carry codec onset artifacts that sail straight past any per-frame energy test. So we require sustained voice. Half a second of startup grace, then fifteen consecutive speech-like frames, which at a twenty millisecond cadence is roughly three hundred milliseconds of continuous speech. Any gap resets the counter to zero, because we want continuous voice and not scattered bursts of noise.

It costs three hundred milliseconds of confirmation latency, it’s the difference between a metric and a guess, and critically it stays conservative enough that a genuinely muted rep still trips the silence branch.

A monitor polls this every two hundred and fifty milliseconds for fifteen seconds after the merge, with exactly three ways out:

Speech detected. Real handoff, marked connected. Done.
The rep’s leg closed first. Recorded as dropped during handoff.
The window elapsed in silence. Recorded as silent after merge. Hang up the rep’s leg only. The customer stays with the AI.

That last line is the entire reason we refused to leave the call.

Two status vocabularies, on purpose

Every transfer emits a session record with one leg per number dialed, and each leg carries its own timeline: when we started dialing, when it was answered, when the briefing began and ended, when the bridge opened and closed, and when we first heard speech. Alongside that sits the leg’s outcome, which moves through dialing, ringing and answered before settling on winner, cancelled, busy, rejected, no-answer or failed, plus the provider’s actual reason when something broke.

Ring time per number is a subtraction. So is time-to-human, briefing delay, handoff duration and the winner’s talk time. None of it is inferred; all of it is timestamped where it happened.

The design decision we’d defend hardest is keeping two separate status vocabularies.

One is the internal state machine, running from idle through hunting, agent-joined, briefing-sent, handoff-in-progress, bridged and monitoring to completed or failed. It answers “where is this right now” and it exists for code.

The other is eleven canonical outcomes, and it exists for people:

Outcome	What it tells you
Connected	A human spoke to a human. The only success.
No rep answered	Everyone was reachable, nobody picked up. Scheduling problem.
All reps busy	Every line engaged. Retry shortly, or add capacity.
All reps rejected	Someone actively declined on every line. Management problem.
Hunt timeout	The window closed before anyone answered.
Briefing failed	We couldn’t prepare the rep, so we didn’t hand over blind.
Bridge failed	The audio path could not be established.
Rep dropped during handoff	They answered, then vanished before speaking.
Rep silent after merge	Connected, but never spoke. Muted, or walked away.
Customer disconnected	The caller hung up mid-transfer.
System error	Ours. Named as ours.

When every leg has finished, the canonical outcome is derived from the legs rather than guessed at. If every hunt leg came back busy, the session resolves to all-reps-busy, and the same holds for rejected and no-answer.

We’re equally strict about never leaking plumbing into a customer-facing field. The dashboard renders the failure reason verbatim, so a raw provider error, the sort of string that names an HTTP status and an internal API URL, is translated before it ever leaves the system. What you read is “Customer call ended before bridge could complete.” Nobody should have to decode a status code to find out their customer hung up.

The AI takes back the wheel

When a transfer fails after the merge, the bridge tears down, the mode flips back to off, the AI pipeline wakes up, and the conversation flow branches on the specific canonical outcome.

Getting that seamless took one more fix that only real traffic could have taught us. Comfort noise from the collapsing leg leaks into speech recognition for a few hundred milliseconds after teardown, and those phantom fragments register as the customer interrupting the AI’s very first sentence back. So we discard transcription output for three full seconds after a bridge drops. Three seconds is comfortably wider than our transcription provider’s sub-second finalization latency, and the constant carries a comment naming the exact production call that taught us the number: call 401837.

That constant has a call ID in its comment. That’s what this kind of work actually looks like.

There’s even a timing bug we had to fix in our own reporting. A call’s outbound payload is assembled the moment the customer’s connection closes, but the canonical transfer outcome isn’t known until the post-merge window closes, several seconds later. Every webhook was honestly reporting an empty transfer list. So finalized sessions are recorded to a call-scoped registry and drained right before dispatch. Our telemetry needed telemetry.

The numbers we chose

Max queued bridge audio	200 ms (10 frames of 20 ms, oldest evicted)
Playback clear gate at merge	250 ms deliberately discarded
Max pacing lag before dropping a frame	250 ms
Sustained voice to confirm a connection	300 ms (15 consecutive frames)
Stream startup grace	500 ms
Audible-gap alarm threshold	100 ms
Transcription cooldown after teardown	3 s
Post-merge verification window	15 s
Default hunt timeout	60 s
Canonical outcomes	11

These are design targets as they appear in the source, not measured production percentiles.

Six things we’d tell you over coffee

For live audio, drop the stale frame, never delay the fresh one. A queue in a real-time path is not a buffer, it’s a debt that never gets repaid.
Put the private thing on a private leg. The briefing looked like a mixing problem. It was a topology problem, and topology problems have two-line solutions.
Keep two status vocabularies. One for your state machine, one for the human reading the dashboard. Collapsing them costs you either correctness or clarity, and you’ll notice which one too late.
“Answered” is not “connected.” If the outcome you care about is a human talking, verify a human talking, in the medium itself.
Never let the customer fall through the floor. Handing your call to someone else’s conference is handing away every recovery path you’ll wish you had.
Alarm on what’s audible, not on what’s convenient. We moved a threshold from 500 ms to 100 ms because that’s where people start to notice. Instrument for the human, not for the log.

Built on Plivo media streams, telephony audio at 8 kHz, Python, Redis, and an unreasonable number of production call recordings.

Written by Swapnil Rajawat | Published Aug 19, 2026