Bring Your Own Carrier: Building SIP Trunking In a Live Voice AI Product

For about a year, our AI voice agents made and received phone calls the way most products do at that stage: we rented them. A communications platform sold us phone numbers, connected our calls, and charged us by the minute. It worked. It was the right decision on day one, because connecting phone calls is a genuinely hard problem and paying someone else to have already solved it is how you ship.

Then customers started asking for something we couldn’t give them.

Over two months we replaced that layer with telephony infrastructure we ran ourselves — so that any customer could point their own phone carrier at our servers and have our AI agents answer on it. The AI half of the system, where almost a year of tuning lived, never changed at all.

This is how that was built.

Why leave a platform that works

Four reasons, roughly in the order they became urgent.

Customers already have carriers. An enterprise with a telecom contract, a negotiated per-minute rate, and a phone number their own customers recognise does not want to abandon all three to use your product. When you ask them to port their numbers to your vendor, you are often asking them to renegotiate a contract. That is not a request you can satisfy with better onboarding. Either you accept their carrier, or you lose the deal.

Geography. Coverage and economics vary enormously between countries. We had live deployments in India and Zambia, and the calculus in each was different. No single global platform is the best route into every market you sell to.

Margin. A per-minute markup on every call is a permanent tax on gross margin — one that scales precisely with your success.

Visibility. This one mattered more than we expected. When a call failed through the platform, we could relay their verdict and little else. Owning the signalling layer means seeing the actual cause code the carrier returned, which turns out to be the difference between “something went wrong” and “your password is wrong.”

The requirement, in one line: any customer’s carrier, provisioned through our API, with our AI agents on it — and no changes to the AI.

That last clause did more to shape the design than anything else.

Five minutes of telephony

Four ideas. That’s all this post needs.

A SIP trunk is a phone line made of internet. Instead of copper running to a telephone exchange, it’s a network connection to a carrier who accepts your calls and puts them onto the real phone network. “Bring your own carrier” means the customer points their carrier at your servers.

Setting up a call and carrying the audio are two different things. SIP is the negotiation — dial, ring, answer, hang up. RTP is the audio itself: small UDP packets, fifty per second in each direction. They travel separately and, as we’ll see, they fail separately.

Asterisk is an open-source phone system, and its REST Interface (ARI) lets your own code drive it: a REST API for acting on calls, and a WebSocket that streams call events as they happen. Your application becomes the dialplan.

A bridge is a room. Put two channels in the same bridge and they hear each other. That’s the model that makes everything below straightforward rather than mysterious.

The constraint that shaped everything

Here was the problem.

Our AI pipeline — speech recognition, the language model turn loop, text-to-speech, interruption handling — already spoke a particular WebSocket protocol, because that’s what the communications platform sent it. JSON messages carrying base64 audio in both directions, plus control messages: play this audio, discard what’s queued, tell me when this checkpoint has played. Months of latency tuning and edge-case handling lived behind that interface.

Asterisk speaks none of that. Asterisk speaks raw RTP: bare UDP packets, twenty milliseconds of 8 kHz μ-law audio each, twelve bytes of header, fifty packets a second, forever.

Two options. Teach the AI pipeline to speak RTP — which means touching transcription, playback, barge-in, and every test that covers them. Or build a translator.

We built the translator, and we made it speak the protocol the pipeline already knew. Same message names, same shapes, same semantics. A new media server sits between Asterisk and the orchestrator, converting RTP frames into the JSON messages our AI stack had always received, and back again.

The result is the thing I’d most want another team to take from this post: we replaced the entire telephony backend of a live product and the AI half needed no changes whatsoever. Not “small changes.” None. Transcription, the turn loop, TTS, barge-in — all untouched, all still covered by the same tests.

The general form: when you replace one side of a system, port the new side to the old interface, not the old side to the new one. Both directions technically work. Only one of them preserves your existing behaviour, your existing tests, and — critically — your ability to run both paths side by side. Because the new media server was interface-compatible, the old platform path and the new SIP path could coexist in a single deployment, chosen per call. That is what turned a two-month replacement of our phone system into an incremental migration instead of a cutover with a rollback plan.

What actually happens on a call

Concretely, for an outbound call:

An API request asks for a call to a number, on behalf of a particular AI agent.
The media server asks Asterisk, over ARI, to originate a channel toward that customer’s trunk.
The person answers. Asterisk hands the channel to our application as a Stasis event on the ARI WebSocket.
Our code creates a mixing bridge and puts the human’s channel into it.
It creates a second channel — Asterisk’s external media channel — whose far end is a UDP socket the media server owns. Audio in the bridge is sent there as RTP; RTP we send back is heard in the bridge.
It opens a WebSocket to the orchestrator for that agent.
From there it’s a translation loop, for as long as the call lasts.

The part worth pausing on: the AI is just another participant in the room. Not a special case bolted onto the phone system — a channel like any other. Which is why inbound and outbound calls take exactly the same path after the first step, and why recording and bridging compose naturally instead of needing their own plumbing.

Three details that make it real work rather than a diagram:

Audio arrives as 8 kHz μ-law and the AI stack wants linear PCM, so there’s format conversion on every packet in both directions.

Outbound audio has to be paced. You cannot hand two seconds of synthesised speech to the network at once; it has to leave at wall-clock speed, one 20 ms frame every 20 ms, from a dedicated sender thread with a bounded queue.

And barge-in means throwing away audio you’ve already queued but not yet sent — which is precisely why that queue belongs to us and not to the operating system.

Turning infrastructure into a product

Owning a phone system is only useful if customers can configure it without us in the loop. A customer submits their carrier’s details — where to send calls, and how to authenticate — and a provisioning API renders a SIP configuration file for that trunk, writes it into a per-trunk directory, and reloads the SIP stack. Deleting the trunk reverses it. No engineer involved.

Three things we learned building it.

Carriers authenticate in two incompatible ways. Some want a username and password on every call. Others authorise by source IP address and reject credentials outright. Getting this wrong is invisible until a real call fails, so the API models it as an explicit mode and refuses combinations that cannot work — supplying credentials alongside IP authentication is a rejected request, not a silently ignored field. Errors at configuration time are enormously cheaper than errors at call time.

Inbound needs strictly more configuration than outbound. To place a call you only need to know where to send it. To receive one, you have to recognise the carrier’s IP addresses, or their calls arrive as anonymous traffic and fall through to whatever your default handling does. A trunk can therefore be perfectly functional outbound and completely broken inbound, which is a genuinely nasty way to find out you’ve under-specified something.

Input validation here is a security boundary, not politeness. Customer-supplied values are written verbatim into a configuration file, and the trunk identifier becomes part of a filesystem path. Config injection and path traversal are both real. Every field is therefore constrained by an explicit pattern and rejected if it doesn’t match — not escaped. Constraining the input space is far easier to audit than trying to escape your way out of it.

Making failures legible

This is the part I’d defend hardest, and it’s about error messages.

A customer types their SIP password wrong. Every call fails. What do they see?

“System error” is useless: it routes a problem only they can fix into our support queue. But guessing confidently and guessing wrong is worse. Tell someone their trunk is misconfigured when the number they dialled was simply dead, and you’ve sent them to audit correct settings for an afternoon.

The raw material is a Q.850 cause code — a number from the telephony standards that Asterisk reports when a call ends. That gets mapped to an internal hangup cause, which the dashboard renders as text a customer actually reads. Three systems, three translations, and no single owner of the final wording. That, on its own, explains most bad error messages in most products.

We added two verdicts. Authentication failed — the carrier refused the call outright; wrong credentials, or this number isn’t authorised on the trunk. Trunk unreachable — the address is wrong, unresolvable, or nothing is answering.

Distinguishing them needed a trick. Asterisk reports the Q.850 cause, not the SIP response code, and one cause conflates “your credentials were rejected” with “the person declined the call.” The discriminator is call progress: a human declines only after their phone rings. A carrier rejecting our authentication answers immediately and never alerts. So the rule is “this cause, on a leg that never rang” — and because it’s a pure function of a cause code and a boolean, the whole truth table is directly testable.

Nothing is measured before a call

The design principle underneath all of it: we never probe. No qualify, no OPTIONS polling, no reachability check when a trunk is configured. Every verdict is derived from the call that just failed.

This is deliberate, and the reasoning is worth stating because the alternative looks so attractive. Health-check polling means continuously generating traffic to a customer’s carrier that the customer never asked for, may be billed for, and may rate-limit. We are guests on their infrastructure. And a probe that succeeded thirty seconds ago is not evidence about the call that failed just now anyway.

The blind spot we chose not to fix

There’s a case this cannot catch, and we wrote it down rather than papering over it.

A termination address that is wrong but happens to point at a live SIP host doesn’t look unreachable at all. The host answers, politely, “no such number.” Asterisk collapses that into the same cause code as a genuinely unallocated number. So one of the most common ways to misconfigure a trunk gets reported to the customer as “invalid phone number.”

We know. We left it.

The cause is genuinely ambiguous — a carrier returns the same response for a number that really is dead, and outbound campaigns dial plenty of those. Reclassifying it would trade a rare wrong “did not pick up” for a frequent wrong “your trunk is broken.” Telling a hundred customers their working configuration is broken, to correctly diagnose one that isn’t, is a bad trade.

Separating the two honestly needs a signal that a single failed call doesn’t carry: either the literal SIP response text and its source, or a per-trunk failure streak — because a trunk that has never carried a connected call and fails identically every time is a configuration problem, whatever any individual cause code says. Neither is built yet. Both are written down, with the specific test calls that demonstrated the behaviour.

A documented, reasoned limitation is an engineering asset. An undocumented one is a liability. The tempting move here was a clever heuristic, and it would have generated a support burden nobody would ever have traced back to it.

Stereo recordings, and a decision about truth

Recording seemed like the easy feature. It wasn’t.

Asterisk will happily record a bridge — but a bridge is a mixing room, so you get one mono track with both voices layered on top of each other. For an AI product that’s a real loss. You can’t cleanly transcribe who said what, can’t measure interruptions, can’t reliably review whether the agent talked over someone.

The insight was that both voices are already separate inside our own code. Every packet in either direction passes through the translation loop. The human’s audio on the way in, the agent’s audio on the way out. Two taps, no phone-system configuration, no post-processing.

Then the decision that makes the section worth writing: where exactly do you tap the agent’s audio — where it’s queued to be sent, or where it actually leaves on the wire?

The wire. Because barge-in discards queued audio the instant the human starts speaking. Tap at the queue and your recording contains a sentence the caller never heard. Tap at the wire and the recording is what actually happened — cut off mid-word, exactly as they experienced it.

Record at the boundary, not at the intention. Anywhere a system can discard work between deciding and doing, those are two different stories, and only one of them is true.

The rest was reuse. Our web-call path already built stereo files in exactly this format, so the SIP path used the same builder, the same storage location, and the same downstream notification. Only the file became stereo — so nothing downstream had to change, and channel-split transcription worked immediately. The same principle as the protocol translator, applied again: hold the interface still and the change stays local.

The half nobody demos

Between a working demo and a product sits a large amount of work that is impossible to make exciting. Briefly:

Who hung up? Agent, human, or system failure are three different business outcomes derived from one event, and they drive analytics, billing and retry logic.

A call can end twice. Two different events both mean “this call is over,” and they arrive seconds apart. Handle both and you send duplicate hangups; handle one and you drop calls. The fix is a deduplication window — with a time bound, because an unbounded set of seen call IDs is a memory leak that takes weeks to surface.

Orphans. Channels that die before a session exists still have to be reported, or calls vanish silently from a customer’s records.

Every registry needs an expiry. State keyed by call ID with no eviction is a slow leak in a process that runs for months. This one bit us in more than one subsystem.

Capacity is finite. A media server carries only so many concurrent calls, so the system must refuse rather than degrade. Accepting a call you can’t carry is worse than declining it.

What production taught us

One lesson reshaped how we monitor everything.

A phone call can succeed completely and carry no audio at all. Signalling and media travel separately and fail independently, so a call can set up perfectly, be answered, run for sixteen seconds, hang up cleanly — and be totally silent to the human on the phone. Every metric we had watched call setup. Call setup was green.

We now count audio packets per call in each direction, and the media server warns about its own silence when outbound audio can’t leave. The system reports the problem instead of waiting for a customer to.

The incidents that taught us this — advertised media addresses that the far end couldn’t route to, a cloud firewall rule scoped to a carrier a customer had since left — deserve their own write-up, and will get one.

Three things worth stealing

When replacing one side of a system, port the new side to the old interface. It keeps your behaviour, your tests, and your ability to run both in parallel.
Derive verdicts from real events, not from probes fired speculatively at someone else’s infrastructure. And when a verdict is genuinely ambiguous, document the ambiguity instead of guessing.
Record and measure at the boundary where things actually happen, not where you intended them to.

None of those are really about telephony.

Written by Yatn B | Published Aug 29, 2026