What Does Your AI Actually Cost?

Ask most finance teams what their AWS bill looks like and they can name the top three services without checking. Ask the same team what their AI API spend looks like and, at a lot of companies, the honest answer is “we get an email from the provider once a month and hope it’s not a surprise.” AI spend is one of the fastest-growing line items in a modern stack, and one of the least visible. Here’s why, and what actually moves the number.

The bill is priced in tokens, not dollars

Cloud providers eventually bill in dollars for compute-hours or GB-months — units you can reason about. Model providers bill per token, and a token isn’t a fixed thing: it’s roughly three-quarters of a word in English, more for other languages, and it’s counted separately (at different rates) for what you send in and what the model sends back out.

Output tokens typically cost several times what input tokens cost, which means the two biggest levers on your bill aren’t “use AI less” — they’re how much you send and how much you ask the model to generate. A prompt that pastes in an entire document to ask one question, or a response format that returns a full essay when three bullet points would do, multiplies cost in a way that’s easy to miss because each individual call looks cheap.

Caching is the highest-leverage lever most teams haven’t flipped

If your application sends the same large system prompt, tool definitions, or reference document on every request — which most agent and RAG architectures do — prompt caching is the single biggest cost lever available, often cutting the cost of the repeated portion of a prompt by most of an order of magnitude on cache hits. It requires no change to model quality and usually a small change to how a prompt is structured (stable content first, variable content last, so the cache boundary lines up). Teams that haven’t looked at this yet are often leaving the largest easy win on the table. Try the LLM cost calculator to see what a given cache-hit rate is worth at your own token volume.

Model selection is a cost decision, not just a quality one

Providers publish multiple model tiers for a reason: not every task needs the frontier model. Classification, extraction, and short transformations often run perfectly well — and far cheaper — on a smaller, faster tier, with the largest model reserved for tasks that genuinely need deep reasoning. The practical move is routing by task, not defaulting every call to whatever model is newest: a triage step that picks the cheapest model that meets a quality bar for that specific call, rather than one model for the entire application. This is the AI-cost equivalent of rightsizing an EC2 instance — most teams are running more workload than necessary on the most expensive tier, by default, because that’s what the quickstart used.

Batch processing, for anything that isn’t real-time

If a workload doesn’t need a synchronous response — nightly enrichment, bulk classification, backfills — batch APIs typically process at a meaningful discount off standard pricing in exchange for asynchronous turnaround. It’s one of the easiest wins in this list because it changes nothing about the actual task, only when the response arrives, and most teams default every workload to the real-time API even when nothing is waiting on the other end.

Why this is invisible in most FinOps tooling

Traditional cost management was built around cloud infrastructure — compute, storage, network — and it shows. Cost Explorer, Azure Cost Management, and most third-party FinOps platforms were designed years before token-based AI billing existed, and it shows in what they can ingest: they have no native concept of an input/output token split, no per-model or per-key breakdown, and no way to show AI spend next to cloud spend on the same axis. The result is that AI spend either doesn’t appear in the FinOps conversation at all, or it appears as one unexplained lump next to line items that are broken out in exhaustive detail.

That gap is exactly what CostMon targets: it pulls usage data straight from the Anthropic Admin cost report — the same source of truth the provider bills from — and normalizes it into the same unified view as your AWS spend, so “how much are we spending on cloud” and “how much are we spending on AI” are finally one dashboard, not two separate mental models. More model providers are landing as connectors continue to roll out, because this problem doesn’t stay AWS-and-one-model shaped for long at any company scaling AI usage.

The near-term trajectory only makes this more urgent

Token prices per unit have been falling as models get more efficient, but usage volume — more features calling models, more agentic workflows making multiple calls per task, more context per call — has been growing faster than price is falling. That’s the pattern behind almost every “our AI bill tripled and we don’t know why” conversation: it’s rarely a single expensive call, it’s the sum of dozens of moderate ones happening more often than anyone budgeted for, in a system that doesn’t natively show which team, key, or model is behind them, and which never got flagged until an invoice made the total impossible to ignore.

Get visibility into that breakdown early, and every other lever in this article — caching, model routing, batching — is something you can decide to pull, on your own schedule, instead of scrambling to explain after the fact.