Who gets the next call?
The prior scheduler state
Two things were true about our call scheduler at the same time, and they sound like opposites.
It dialled calls we had no capacity to carry. Nothing in the dispatch path consulted the providers that actually run out — the streaming connections, the model tokens, the carrier channels — so the first sign that one was full arrived as an error in the middle of a live conversation with a customer on the line.
It also refused calls we could easily have carried. A per-organisation ceiling, which had no relationship to any real resource, would hold a workspace’s campaign at a standstill while the platform around it sat half idle.
Both failures came out of the same place: the two numbers governing dispatch described nothing that existed. Fixing one of them without the other would have been straightforward and useless. Clamp down harder and we’d protect the providers by dialling even fewer calls than the hardware could carry. Loosen up and we’d fill the pipe with calls destined to fail mid-conversation.
The actual requirement, which took a while to state this plainly: place as many calls as the underlying resources can genuinely carry, and not one more. Both halves, at once, under live traffic, without a maintenance window.
No endpoint changed. The dispatch loop still ticks every five seconds the way it always has. But every call now passes a verdict that did not previously exist, and switching those verdicts on turned out to be a stranger exercise than building them.
Two numbers that described nothing
The old system admitted calls against a global cap and a per-organisation number.
The global cap was 30. It counted outbound calls only, so inbound traffic sat on top of it without appearing in it. On a busy afternoon the platform could be carrying 45 calls and reporting 28.
The per-organisation number was worse, and the reason is a design bug rather than a coding one. A single integer, default 1. One validator refused any change that would push the sum of everyone’s number past the global cap, which only makes sense if the number is a reservation — a promise that those slots are yours. A different piece of code refused to dispatch once an organisation reached that number, which only makes sense if it’s a ceiling.
Those are opposite things. A reservation is a floor you are guaranteed. A ceiling is a wall you cannot pass. Storing both meanings in one column means a customer can never burst into capacity that is sitting idle, and also cannot be guaranteed their floor when everything is busy. The sum check made it worse by excluding every organisation sitting at the default of 1, on the theory that the default didn’t count. With two hundred free workspaces on a platform cap of 30, the guarantee was arithmetically impossible, and nothing anywhere reported that.
That gap had been patched by the time this project started. What it left behind was the more interesting question. Now that the number was enforced, we were enforcing a number that had never described anything, and enforcing it correctly meant refusing calls the platform could have carried.
Admission now asks four questions in a single atomic step before a call is placed. Does every provider this call needs have headroom. Is the organisation inside its envelope. Is the platform under its cap. And if the answer to any of those is no, is this particular call entitled to a slot anyway.
The vocabulary, briefly
Five words, because the rest of the post leans on them.
In flight is a call holding provider sessions right now. Floor is what an organisation is guaranteed. Ceiling is the most it may ever run. Burst is everything above its floor, which competes with every other organisation’s burst for what’s left over. That leftover is the free pool: the platform cap, minus the sum of everybody’s floors.
The only genuinely counterintuitive one is the floor, and it trips almost everyone on first contact. A floor is not a slot held empty with your name on it. It is a position in a queue. An organisation below its floor goes to the front for the next slot that frees up; nothing sits idle waiting for it to call.
From caps to guarantees
The fix to the one-number problem was to split it into two columns, which took an afternoon. Deciding what to put in them took considerably longer.
We ran a migration that took every organisation carrying a real ceiling and turned that ceiling into a floor. Their old maximum became their new guarantee, and their ceiling was removed entirely. A workspace that had been capped at 20 was now promised 20 and free to use more whenever the platform had room. A second migration a few days later did the same to the free tier, so that today no organisation has a ceiling at all.
That last sentence sounds reckless and mostly isn’t, because the ceiling was never the thing protecting the platform. Provider limits and the global cap were. What the per-organisation ceiling actually did was stop a paying customer from using capacity that was sitting there empty, and then take the blame for it in a support ticket.
What did change was the shape of those support conversations. “You’re capped at 20” and “you’re guaranteed 20” are the same number and completely different products.
Counting turned out to be the whole job
Here is the part I’d most want another team to read, because it cost us an outage and the lesson generalises well past voice.
An inbound call, in our data model, is written with status created and then never advances. Not by design, particularly. The webhook handlers for the later stages each stamp their own timestamp and return before reaching the generic status write, so created is what a live inbound call looks like from the moment it’s answered until the caller hangs up. That had been true for a long time and nobody had reason to notice.
Three separate counters filtered on the two statuses that outbound calls pass through. All three therefore read zero inbound calls, permanently, no matter how many were running.
Everything downstream of those counters was consequently fiction. The per-organisation inbound limit never fired. The inbound reserve — slots held back so a business can always receive calls — was computed as permanently owed, so it was subtracted from outbound capacity forever and never released by the inbound traffic actually using it. And inbound calls were invisible to the platform cap while sitting squarely on top of it, which is where that 45-calls-reported-as-28 came from.
So we fixed it. One shared definition of “in flight”, used by every counter so they couldn’t drift apart again, covering both shapes.
Fairness is an ordering problem
Once you stop refusing people, the interesting question stops being “what is the limit” and becomes “who goes next”. Those need different machinery, and the second one is where the design actually lives. Between organisations, the order is: anyone below their floor first, largest shortfall leading, then everybody else competing for the free pool by fewest calls currently running, one slot each per pass.
Round-robin was the obvious alternative and it’s subtly bad. Picture two organisations holding twenty calls apiece and a third with a single call to make. Plain round-robin hands the next freed slot to whoever is next in the list, and the newcomer can lose that draw repeatedly while the platform stays full. Ordering by calls actually in flight puts an organisation at zero structurally at the front. Its wait is one call ending plus one tick, which is seconds, and nothing was held idle to achieve that. We use the raw count rather than a percentage of capacity, deliberately, so that a large customer can’t jump ahead of a small one on the strength of having a bigger allowance.
Within an organisation the order is by what a refusal costs. Inbound first, because a refused inbound call is a person who couldn’t reach a business and there is no retry that fixes that. Then calls triggered directly by a user through the API, where somebody is sitting there waiting. Then retries, which sound low-priority and aren’t: a retry has a window, and deferring one long enough destroys it. Then workflows. Then campaign traffic last, because campaigns are the only class that genuinely doesn’t care whether a call goes out now or in ninety seconds. They absorb the shock for everyone else.
Strict priority did need one correction. A workspace with a steady trickle of API calls could zero out everything behind it indefinitely, so a few slots per tick are withheld from the higher classes and kept for retries and campaigns. The part worth copying is that the withholding only binds while the lower class actually has work waiting. A hard split idles reserved capacity every time one side is empty; reserving against demonstrated demand costs nothing on a quiet platform, where the top class still gets everything.
A reserve is owed, not held
An organisation taking inbound calls can reserve part of its floor for them, so a campaign running flat out can’t leave its own customers hitting a busy signal.
The arithmetic is one line and it’s the line people get wrong: what’s held back is the reserve minus the inbound calls already running, floored at zero. A live inbound call occupies its reserved slot rather than holding a second one. Reserve three lines, have two calls up, and exactly one slot is withheld from everything else.
Now the failure this design permits, which we found on an internal workspace rather than in a customer’s account, though only by luck.
The constraints are that the reserve can’t exceed the floor, and the floor can’t exceed the ceiling. So floor, ceiling and reserve all set to 1 is a legal configuration. It is also a permanent deadlock. The queueable ceiling works out to zero, and the only thing that can reduce what’s owed is an inbound call already in progress, which an idle workspace by definition doesn’t have. Outbound calls are withheld every tick, forever, and no reaper covers a call parked in that state. They just never dial. Two calls sat like that for days, with the admission log patiently writing withheld: 1 every five seconds at zero calls in flight.
We haven’t fixed it. There are two candidate fixes written down, either sufficient on its own, and neither has been urgent since the migration that removed ceilings made the configuration nearly unreachable. What exists instead is a documented sizing rule — an organisation’s real outbound capacity is its ceiling minus its reserve, so leave at least one between them — and a note explaining why the softer version of the same effect shows up under load at any size.
A limitation you’ve reasoned about and written down is worth more than a clever heuristic that quietly does the wrong thing in a case you didn’t enumerate. That’s the second time that principle has earned its place in one of these posts.
What we actually built
You could summarise the project as “we started refusing calls.” That is accurate, and it misses the point.
Refusing is the cheap part — a comparison and an early return. What the platform could not do before was answer a question: why didn’t this call go out? A campaign would sit still and there was no answer anywhere in the system, not for the customer, not for support, not for whoever went digging three days later. The 742 stranded retries had been ageing out of that scan window for as long as the window had existed. The ghost reservations had been quietly eating inbound capacity for months. Neither was hidden by anything clever. Nothing in the system had ever been asked to record why a call it could have placed stayed where it was.
That is what a verdict is for, and it is why building the decision as a structured object rather than a boolean turned out to matter more than any individual limit in it. Every call now leaves behind a reason, a scope and an organisation, whether it dialled or not. The workspace that had configured itself into a permanent deadlock — floor, ceiling and reserve all set to one — was diagnosed entirely from those log lines, without anyone opening the database.
None of the numbers are right, incidentally, and that turns out to be survivable. The provider ceilings are contracts where we have them and educated guesses where we don’t. The token weights are an estimate multiplied by an assumed conversation rate. Some of the floors are a commercial decision that predates anyone measuring anything. What changed is not that we found the correct numbers. It is that a wrong one now surfaces as a named refusal in a report somebody reads, rather than as a campaign that mysteriously runs slow.
Saying no was never the hard part. Dialling right up to the edge of what the platform can genuinely carry, and being able to say precisely what stopped you at the point you stopped — that was the work.
Leave a Reply