Token Spend Is an Architecture Problem, Not a Budget Problem

By the time April rolled around this year, Uber had already spent its entire 2026 AI budget, which is a genuinely wild sentence to type out, and the reaction across the industry was almost entirely financial. Microsoft revoked Claude Code licenses from big chunks of its own engineering org. Gartner started shopping around a projection that AI coding costs will overtake the average developer salary by 2028. Procurement teams everywhere started drawing up per-seat token caps and monthly ceilings and dashboards that light up red when someone runs one too many agentic loops. All of that is treating a fever with a blanket. The bill exploded because the systems generating those tokens were designed carelessly, and no amount of budget governance fixes a system that shovels 180k tokens of irrelevant context into every single request. We've built enough of these things now, document pipelines, internal copilots, customer-facing support agents, to say with confidence that token spend tracks almost perfectly with architectural laziness. If your costs are terrifying, that's diagnostic information about your design, not your finance department. The companies panicking about spend are usually the ones who wired an agent to dump the entire codebase into context on every turn because it was easier than building retrieval, and then acted surprised when the invoice came. You cannot budget your way out of that. You can only build your way out.

The single most common mistake we see, and it shows up in probably eight out of ten agent codebases we audit, is treating the context window like it's free scratch space you fill up to feel safe. Someone reads that a model supports 200k or a million tokens of context and decides the correct move is to stuff as much as physically fits, the full conversation history, every retrieved document, the entire schema, three example outputs, and a system prompt that's grown to 4,000 words because nobody ever deletes anything from it. Every token in that window gets billed on every turn, and in an agentic loop that turn happens twenty or thirty times to complete one task. A single support ticket resolution can rack up two million billed input tokens not because the work is hard but because the same bloated context rides along on every iteration. The fix is unglamorous. You aggressively trim what goes into the window, you summarize old conversation turns instead of carrying them verbatim, you retrieve the three most relevant chunks instead of the top fifty, and you delete the parts of the system prompt that were added to fix a bug that no longer exists. We had a client running a document classification agent that was passing the full source document plus every prior classification decision on each call. Splitting that into a retrieval step that pulled only the relevant sections cut their per-document cost by roughly seventy percent, and accuracy went up, because the model wasn't drowning in noise anymore. Bigger context is a capability, not a strategy.

Anthropic and OpenAI both offer prompt caching, and it is close to the highest-leverage thing you can do to a running agent, yet a shocking number of teams either don't know it exists or never bothered to wire it up correctly. The idea is simple. Your system prompt, your tool definitions, your few-shot examples, the large static context that's identical across requests, all of that can be cached server-side so you're billed a fraction of the normal input rate on cache hits instead of paying full freight every time. Anthropic gives you something like a ninety percent discount on cached reads. For an agent where the first 30k tokens of every request are byte-for-byte identical, which describes almost every agent with a fixed system prompt and toolset, you're leaving enormous savings on the table by not structuring your requests to hit the cache. The catch is that caching is prefix-based, so the cached content has to come first and stay stable. If you interleave dynamic content early in the prompt you blow the cache on every request and get nothing. So you restructure. Static stuff up top, marked for caching, dynamic stuff at the bottom. We rebuilt a customer portal's support agent this way and their monthly Anthropic bill dropped by a bit over half with zero change to output quality, purely from ordering the prompt correctly and setting cache breakpoints. That's not an optimization, that's picking up cash off the sidewalk.

The other structural failure is sending every request to your most expensive model because it's the one that gives the best answers, which is true and also a spectacular way to set money on fire. Most of what agents actually do is not hard. Classifying an intent, extracting a field from a form, deciding which tool to call next, reformatting some JSON, none of that needs a frontier model, and yet teams route the whole pipeline through the top-tier model because they built one path and never split it. Model routing means you look at the task in front of you and pick the cheapest model that can do it reliably. A small fast model handles the classification and routing decisions, a mid-tier model handles most of the generation, and you only escalate to the expensive model when the task genuinely needs the reasoning, and you can even build a fallback where a cheap model attempts the task first and a confidence check kicks it up to the bigger model only when needed. This does require you to actually measure which tasks need which model instead of guessing, and to build an eval set so you know when a downgrade hurts quality, which is real work that most teams skip. We usually build this as a thin router layer in Go sitting in front of the model providers, doing the classification, cache management, and escalation logic in one place, so the application code doesn't need to know or care which model answered. A well-routed pipeline commonly runs the majority of its calls on models that cost a small fraction of the frontier tier, and the user never notices, because the frontier model still handles the parts that are actually hard.

You cannot fix what you don't measure, and almost nobody instruments token cost at the granularity that would let them find the waste. The monthly invoice tells you that you spent a horrifying amount but not that ninety percent of it came from one poorly-designed retry loop in your ingestion pipeline that re-sends the full context on every failure. You want per-request, per-agent, per-task-type cost attribution flowing into whatever tracing setup you already run, tagged so you can slice it by feature and by model, and you want it visible to the engineers writing the agent code, not just to finance three weeks after the fact. When an engineer can see that the change they just shipped doubled the average cost per conversation, they fix it that afternoon. When they find out from a spreadsheet at the end of the quarter, it's already been bleeding money for months. This is the same discipline we've applied to latency and error rates for a decade, just pointed at a new axis. Tag your spans with input tokens, output tokens, cached tokens, the model used, and the dollar cost, and suddenly the expensive parts of your system become obvious instead of mysterious.

Uber's collapse and Microsoft yanking licenses are being read as cautionary tales about AI being too expensive, and that reading is lazy in exactly the same way the underlying architectures were lazy. The technology isn't overpriced. The systems built on top of it were built without any of the cost discipline we'd consider table stakes for database queries or API calls or memory allocation. Nobody would ship a service that runs a full table scan on every request and then blame the cloud provider when the bill arrives, but that's functionally what an agent stuffing its whole context window on every turn is doing. The teams that treat runaway token spend as a procurement problem will spend the next two years fighting their own engineers with rate limits and approval workflows, throttling the exact output the tools were bought to produce, while the teams that treat it as a design problem will trim context, cache prefixes, route by complexity, instrument their spend, and quietly run the same workloads for a fifth of the cost. At steezr we've been building these pipelines for clients long enough to know the savings are real and repeatable, and they come from engineering decisions, not spreadsheet decisions. The invoice is telling you something specific about your architecture. Read it as a design brief and go fix the system.