For about a year, our AI voice agents made and received phone calls the way most products do at that stage: we rented them. A communications platform sold us phone numbers, connected our calls, and charged us by the minute. It worked. It was the right decision on day one, because connecting phone calls is a genuinely hard problem and paying someone else to have already solved it is how you ship.<\/p>\n\n\n\n
Then customers started asking for something we couldn’t give them.<\/p>\n\n\n\n
Over two months we replaced that layer with telephony infrastructure we ran ourselves \u2014 so that any customer could point their own phone carrier at our servers and have our AI agents answer on it. The AI half of the system, where almost a year of tuning lived, never changed at all.<\/p>\n\n\n\n
This is how that was built.<\/p>\n\n\n\n
Four reasons, roughly in the order they became urgent.<\/p>\n\n\n\n
Customers already have carriers.<\/strong> An enterprise with a telecom contract, a negotiated per-minute rate, and a phone number their own customers recognise does not want to abandon all three to use your product. When you ask them to port their numbers to your vendor, you are often asking them to renegotiate a contract. That is not a request you can satisfy with better onboarding. Either you accept their carrier, or you lose the deal.<\/p>\n\n\n\n Geography.<\/strong> Coverage and economics vary enormously between countries. We had live deployments in India and Zambia, and the calculus in each was different. No single global platform is the best route into every market you sell to.<\/p>\n\n\n\n Margin.<\/strong> A per-minute markup on every call is a permanent tax on gross margin \u2014 one that scales precisely with your success.<\/p>\n\n\n\n Visibility.<\/strong> This one mattered more than we expected. When a call failed through the platform, we could relay their verdict and little else. Owning the signalling layer means seeing the actual cause code the carrier returned, which turns out to be the difference between “something went wrong” and “your password is wrong.”<\/p>\n\n\n\n The requirement, in one line: any customer’s carrier, provisioned through our API, with our AI agents on it \u2014 and no changes to the AI.<\/strong><\/p>\n\n\n\n That last clause did more to shape the design than anything else.<\/p>\n\n\n\n Four ideas. That’s all this post needs.<\/p>\n\n\n\n A SIP trunk is a phone line made of internet.<\/strong> Instead of copper running to a telephone exchange, it’s a network connection to a carrier who accepts your calls and puts them onto the real phone network. “Bring your own carrier” means the customer points their<\/em> carrier at your<\/em> servers.<\/p>\n\n\n\n Setting up a call and carrying the audio are two different things.<\/strong> SIP is the negotiation \u2014 dial, ring, answer, hang up. RTP is the audio itself: small UDP packets, fifty per second in each direction. They travel separately and, as we’ll see, they fail separately.<\/p>\n\n\n\n Asterisk is an open-source phone system<\/strong>, and its REST Interface (ARI) lets your own code drive it: a REST API for acting on calls, and a WebSocket that streams call events as they happen. Your application becomes the dialplan.<\/p>\n\n\n\n A bridge is a room.<\/strong> Put two channels in the same bridge and they hear each other. That’s the model that makes everything below straightforward rather than mysterious.<\/p>\n\n\n\n Here was the problem.<\/p>\n\n\n\n Our AI pipeline \u2014 speech recognition, the language model turn loop, text-to-speech, interruption handling \u2014 already spoke a particular WebSocket protocol, because that’s what the communications platform sent it. JSON messages carrying base64 audio in both directions, plus control messages: play this audio, discard what’s queued, tell me when this checkpoint has played. Months of latency tuning and edge-case handling lived behind that interface.<\/p>\n\n\n\n Asterisk speaks none of that. Asterisk speaks raw RTP: bare UDP packets, twenty milliseconds of 8 kHz \u03bc-law audio each, twelve bytes of header, fifty packets a second, forever.<\/p>\n\n\n\n Two options. Teach the AI pipeline to speak RTP \u2014 which means touching transcription, playback, barge-in, and every test that covers them. Or build a translator.<\/p>\n\n\n\n We built the translator, and we made it speak the protocol the pipeline already knew. Same message names, same shapes, same semantics. A new media server sits between Asterisk and the orchestrator, converting RTP frames into the JSON messages our AI stack had always received, and back again.<\/p>\n\n\n\n The result is the thing I’d most want another team to take from this post: we replaced the entire telephony backend of a live product and the AI half needed no changes whatsoever.<\/strong> Not “small changes.” None. Transcription, the turn loop, TTS, barge-in \u2014 all untouched, all still covered by the same tests.<\/p>\n\n\n\n The general form: when you replace one side of a system, port the new side to the old interface, not the old side to the new one.<\/em> Both directions technically work. Only one of them preserves your existing behaviour, your existing tests, and \u2014 critically \u2014 your ability to run both paths side by side. Because the new media server was interface-compatible, the old platform path and the new SIP path could coexist in a single deployment, chosen per call. That is what turned a two-month replacement of our phone system into an incremental migration instead of a cutover with a rollback plan.<\/p>\n\n\n\n Concretely, for an outbound call:<\/p>\n\n\n\n The part worth pausing on: the AI is just another participant in the room.<\/strong> Not a special case bolted onto the phone system \u2014 a channel like any other. Which is why inbound and outbound calls take exactly the same path after the first step, and why recording and bridging compose naturally instead of needing their own plumbing.<\/p>\n\n\n\n Three details that make it real work rather than a diagram:<\/p>\n\n\n\n Audio arrives as 8 kHz \u03bc-law and the AI stack wants linear PCM, so there’s format conversion on every packet in both directions.<\/p>\n\n\n\n Outbound audio has to be paced<\/strong>. You cannot hand two seconds of synthesised speech to the network at once; it has to leave at wall-clock speed, one 20 ms frame every 20 ms, from a dedicated sender thread with a bounded queue.<\/p>\n\n\n\n And barge-in means throwing away audio you’ve already queued but not yet sent<\/strong> \u2014 which is precisely why that queue belongs to us and not to the operating system.<\/p>\n\n\n\n Owning a phone system is only useful if customers can configure it without us in the loop. A customer submits their carrier’s details \u2014 where to send calls, and how to authenticate \u2014 and a provisioning API renders a SIP configuration file for that trunk, writes it into a per-trunk directory, and reloads the SIP stack. Deleting the trunk reverses it. No engineer involved.<\/p>\n\n\n\n Three things we learned building it.<\/p>\n\n\n\n Carriers authenticate in two incompatible ways.<\/strong> Some want a username and password on every call. Others authorise by source IP address and reject credentials outright. Getting this wrong is invisible until a real call fails, so the API models it as an explicit mode and refuses<\/em> combinations that cannot work \u2014 supplying credentials alongside IP authentication is a rejected request, not a silently ignored field. Errors at configuration time are enormously cheaper than errors at call time.<\/p>\n\n\n\n Inbound needs strictly more configuration than outbound.<\/strong> To place a call you only need to know where to send it. To receive one, you have to recognise the carrier’s IP addresses, or their calls arrive as anonymous traffic and fall through to whatever your default handling does. A trunk can therefore be perfectly functional outbound and completely broken inbound, which is a genuinely nasty way to find out you’ve under-specified something.<\/p>\n\n\n\n Input validation here is a security boundary, not politeness.<\/strong> Customer-supplied values are written verbatim into a configuration file, and the trunk identifier becomes part of a filesystem path. Config injection and path traversal are both real. Every field is therefore constrained by an explicit pattern and rejected if it doesn’t match \u2014 not escaped. Constraining the input space is far easier to audit than trying to escape your way out of it.<\/p>\n\n\n\n This is the part I’d defend hardest, and it’s about error messages.<\/p>\n\n\n\n A customer types their SIP password wrong. Every call fails. What do they see?<\/p>\n\n\n\n “System error” is useless: it routes a problem only they can fix into our support queue. But guessing confidently and guessing wrong is worse. Tell someone their trunk is misconfigured when the number they dialled was simply dead, and you’ve sent them to audit correct settings for an afternoon.<\/p>\n\n\n\n The raw material is a Q.850 cause code \u2014 a number from the telephony standards that Asterisk reports when a call ends. That gets mapped to an internal hangup cause, which the dashboard renders as text a customer actually reads. Three systems, three translations, and no single owner of the final wording. That, on its own, explains most bad error messages in most products.<\/p>\n\n\n\n We added two verdicts. Authentication failed<\/strong> \u2014 the carrier refused the call outright; wrong credentials, or this number isn’t authorised on the trunk. Trunk unreachable<\/strong> \u2014 the address is wrong, unresolvable, or nothing is answering.<\/p>\n\n\n\n Distinguishing them needed a trick. Asterisk reports the Q.850 cause, not the SIP response code, and one cause conflates “your credentials were rejected” with “the person declined the call.” The discriminator is call progress: a human declines only after their phone rings.<\/strong> A carrier rejecting our authentication answers immediately and never alerts. So the rule is “this cause, on a leg that never rang” \u2014 and because it’s a pure function of a cause code and a boolean, the whole truth table is directly testable.<\/p>\n\n\n\n The design principle underneath all of it: we never probe. No This is deliberate, and the reasoning is worth stating because the alternative looks so attractive. Health-check polling means continuously generating traffic to a customer’s carrier that the customer never asked for, may be billed for, and may rate-limit. We are guests on their infrastructure. And a probe that succeeded thirty seconds ago is not evidence about the call that failed just now anyway.<\/p>\n\n\n\n There’s a case this cannot catch, and we wrote it down rather than papering over it.<\/p>\n\n\n\n A termination address that is wrong but happens to point at a live<\/em> SIP host doesn’t look unreachable at all. The host answers, politely, “no such number.” Asterisk collapses that into the same cause code as a genuinely unallocated number. So one of the most common ways to misconfigure a trunk gets reported to the customer as “invalid phone number.”<\/p>\n\n\n\n We know. We left it.<\/p>\n\n\n\n The cause is genuinely ambiguous \u2014 a carrier returns the same response for a number that really is dead, and outbound campaigns dial plenty of those. Reclassifying it would trade a rare wrong “did not pick up” for a frequent wrong “your trunk is broken.” Telling a hundred customers their working configuration is broken, to correctly diagnose one that isn’t, is a bad trade.<\/p>\n\n\n\n Separating the two honestly needs a signal that a single failed call doesn’t carry: either the literal SIP response text and its source, or a per-trunk failure streak \u2014 because a trunk that has never carried a connected call and fails identically every time is a configuration problem, whatever any individual cause code says. Neither is built yet. Both are written down, with the specific test calls that demonstrated the behaviour.<\/p>\n\n\n\n A documented, reasoned limitation is an engineering asset. An undocumented one is a liability.<\/strong> The tempting move here was a clever heuristic, and it would have generated a support burden nobody would ever have traced back to it.<\/p>\n\n\n\n Recording seemed like the easy feature. It wasn’t.<\/p>\n\n\n\n Asterisk will happily record a bridge \u2014 but a bridge is a mixing room, so you get one mono track with both voices layered on top of each other. For an AI product that’s a real loss. You can’t cleanly transcribe who said what, can’t measure interruptions, can’t reliably review whether the agent talked over someone.<\/p>\n\n\n\n The insight was that both voices are already separate inside our own code<\/em>. Every packet in either direction passes through the translation loop. The human’s audio on the way in, the agent’s audio on the way out. Two taps, no phone-system configuration, no post-processing.<\/p>\n\n\n\n Then the decision that makes the section worth writing: where exactly do you tap the agent’s audio \u2014 where it’s queued to be sent, or where it actually leaves on the wire?<\/p>\n\n\n\n The wire. Because barge-in discards queued audio the instant the human starts speaking. Tap at the queue and your recording contains a sentence the caller never heard. Tap at the wire and the recording is what actually happened \u2014 cut off mid-word, exactly as they experienced it.<\/p>\n\n\n\n Record at the boundary, not at the intention.<\/em> Anywhere a system can discard work between deciding and doing, those are two different stories, and only one of them is true.<\/p>\n\n\n\n The rest was reuse. Our web-call path already built stereo files in exactly this format, so the SIP path used the same builder, the same storage location, and the same downstream notification. Only the file became stereo \u2014 so nothing downstream had to change, and channel-split transcription worked immediately. The same principle as the protocol translator, applied again: hold the interface still and the change stays local.<\/strong><\/p>\n\n\n\n Between a working demo and a product sits a large amount of work that is impossible to make exciting. Briefly:<\/p>\n\n\n\n Who hung up?<\/strong> Agent, human, or system failure are three different business outcomes derived from one event, and they drive analytics, billing and retry logic.<\/p>\n\n\n\n A call can end twice.<\/strong> Two different events both mean “this call is over,” and they arrive seconds apart. Handle both and you send duplicate hangups; handle one and you drop calls. The fix is a deduplication window \u2014 with a time bound, because an unbounded set of seen call IDs is a memory leak that takes weeks to surface.<\/p>\n\n\n\n Orphans.<\/strong> Channels that die before a session exists still have to be reported, or calls vanish silently from a customer’s records.<\/p>\n\n\n\n Every registry needs an expiry.<\/strong> State keyed by call ID with no eviction is a slow leak in a process that runs for months. This one bit us in more than one subsystem.<\/p>\n\n\n\n Capacity is finite.<\/strong> A media server carries only so many concurrent calls, so the system must refuse rather than degrade. Accepting a call you can’t carry is worse than declining it.<\/p>\n\n\n\n One lesson reshaped how we monitor everything.<\/p>\n\n\n\n A phone call can succeed completely and carry no audio at all.<\/strong> Signalling and media travel separately and fail independently, so a call can set up perfectly, be answered, run for sixteen seconds, hang up cleanly \u2014 and be totally silent to the human on the phone. Every metric we had watched call setup. Call setup was green.<\/p>\n\n\n\n We now count audio packets per call in each direction, and the media server warns about its own silence when outbound audio can’t leave. The system reports the problem instead of waiting for a customer to.<\/p>\n\n\n\n The incidents that taught us this \u2014 advertised media addresses that the far end couldn’t route to, a cloud firewall rule scoped to a carrier a customer had since left \u2014 deserve their own write-up, and will get one.<\/p>\n\n\n\n
\n\n\n\nFive minutes of telephony<\/h2>\n\n\n\n
\n\n\n\nThe constraint that shaped everything<\/h2>\n\n\n\n
\n\n\n\nWhat actually happens on a call<\/h2>\n\n\n\n
\n
\n\n\n\nTurning infrastructure into a product<\/h2>\n\n\n\n
\n\n\n\nMaking failures legible<\/h2>\n\n\n\n
Nothing is measured before a call<\/h3>\n\n\n\n
qualify<\/code>, no OPTIONS polling, no reachability check when a trunk is configured. Every verdict is derived from the call that just failed.<\/p>\n\n\n\nThe blind spot we chose not to fix<\/h3>\n\n\n\n
\n\n\n\nStereo recordings, and a decision about truth<\/h2>\n\n\n\n
\n\n\n\nThe half nobody demos<\/h2>\n\n\n\n
\n\n\n\nWhat production taught us<\/h2>\n\n\n\n
\n\n\n\nThree things worth stealing<\/h2>\n\n\n\n