Our AI bill had been creeping up for months, a little at a time, never enough in one month to trigger a proper investigation. Then I finally sat down with a full month of usage data and a pot of coffee, and found the money was not going where I thought it was. It was leaking in three places, and none of them was the obvious one.
The first leak was a retry loop I had forgotten writing. A classification endpoint — the cheap, boring one that sorts incoming messages — was failing intermittently on a flaky network hop and retrying every failed call three times. Each retry was a full-priced call. The feature itself was cheap; the retries were not. Together they were eating 40% of the monthly bill, and the dashboard never flagged it because the per-call cost looked tiny. I only found it because I sorted by total spend instead of unit price. The fix took an afternoon: one retry policy, a smaller model for that specific job, and the bill dropped by a third. The lesson is not that retries are bad. It is that a retry you cannot see is a subscription you did not sign up for.
The second leak was using the wrong model for the job. Our flagship model was answering questions that a model a tenth of its price could have handled — simple lookups, formatting, yes-or-no checks. Nobody had done it on purpose. Each feature had been built with the best model available at the time, and nobody went back to ask whether "best" was still necessary once the feature was stable. Downgrading those calls was a config change, not a rewrite, and nothing about the user experience changed. The expensive model stayed where it earned its keep: the genuinely hard questions. Everything else moved down a tier. The savings were immediate and, frankly, embarrassing — the kind of number that makes you wonder what else you have not looked at.
The third leak was the absence of a single place to look. Five models meant five dashboards, five bills, five sets of keys in an environment file I was afraid to touch. There was no one page that could tell me what a feature cost, so no one ever asked. That is how the first two leaks survived for months. They were not hidden. They were just spread across enough screens that nobody assembled the picture.
The fix for all three was the same structural change: put one gateway in front of everything. I routed the whole app through AzYun, a unified AI Models API that speaks one standard protocol no matter which model sits behind it. One endpoint, one key, one bill. Switching a feature to a cheaper model became changing one parameter. The retry policy became one setting instead of five scattered loops. And for the first time, one dashboard showed the real per-feature cost, which is what made the next audit take an evening instead of a weekend.
There is a fourth benefit I did not expect, and it arrived on a Tuesday afternoon when one provider had a partial outage. In the old setup, that would have meant our main feature was down until someone noticed and flipped to a backup by hand. With an AI API gateway in front, failover routed around the outage before I opened the status page. I found out from the status page, not from my users. That is the part of the bill nobody puts in a spreadsheet: the users who quietly leave during an outage you did not see.
None of this is magic, and the tradeoffs are real — you are adding one more dependency, and heavy usage still costs real money. But the audit paid for itself many times over, and the structure that came out of it keeps paying. If your AI bill has been creeping up and you cannot say exactly why, do the audit before you do anything else. Sort by total spend, not unit price. Ask whether each feature still needs the model it was built with. And if you are juggling more than one model, look at a unified AzYun setup — not because it is trendy, but because one dashboard, one key and one failover policy delete a whole category of leaks you should not have to hunt for one by one.
What I check every month now
The audit is not a one-off anymore; I run a shorter version of it on the first Monday of every month, and it takes about forty minutes. I pull the per-model spend, I sort by total dollars, and I look for anything that moved more than twenty percent since last month without a matching change in usage. Last month that check caught a summarisation job that had quietly doubled after a prompt change made its outputs longer, and I fixed the prompt before the bill doubled with it. I also check the retry rate on each endpoint, because a retry rate creeping up from one percent to four is a leak wearing a different coat.
The number I actually watch
If I had to keep one number from the whole audit, it would be cost per completed task, not cost per token. Tokens are cheap to quote and easy to compare, but they do not tell you whether the work got done. A cheap model that retries three times and still needs a human to fix the output is not cheap. I track cost per completed task for our three biggest features, and when that number moves, I know exactly where to look first. That is the whole point of the audit: not a smaller bill for its own sake, but a bill you can explain, line by line, to anyone who
Responses (0)