What Does Your AI Actually Cost?

Ask a finance team about the AWS bill. They can name the top three services without checking. Ask the same team about AI API spend. At many companies the answer is an email from the provider once a month, and a hope that it is small. AI spend is one of the fastest-growing lines in a modern stack. It is also one of the least visible. This post explains why, and what moves the number.

The bill is priced in tokens, not dollars

Cloud providers bill in dollars for compute-hours or GB-months. You can reason about those units. Model providers bill per token. A token is not a fixed unit. In English it is about three-quarters of a word. Other languages use more. Providers also count what you send in and what the model sends back out separately, at different rates.

Output tokens typically cost several times what input tokens cost. So two levers matter most: how much you send and how much you ask the model to generate. Say a prompt pastes in a whole document to ask one question. That multiplies cost. So does a response format that returns an essay when three bullet points would do. Both are easy to miss, because each call looks cheap. Reasoning models also bill at the output rate for thinking you mostly never see. Our post on thinking tokens shows how large that part can get.

Caching is the highest-leverage lever most teams haven’t flipped

Most agent and RAG systems send the same large system prompt, tool definitions, or reference document with every request. If yours does, prompt caching is the biggest cost lever you have. On a cache hit, it often cuts the cost of the repeated part of a prompt by most of an order of magnitude. It does not change model quality. It usually needs a small change to the prompt structure: put stable content first and variable content last, so the cache boundary lines up. Many teams have not looked at this yet. Try the LLM cost calculator to see what a cache-hit rate is worth at your own token volume.

Model selection is a cost decision, not just a quality one

Providers publish several model tiers for a reason. Not every task needs the frontier model. Classification, extraction, and short transformations often run well on a smaller, faster tier, at far lower cost. Keep the largest model for tasks that need deep reasoning. Route by task. Do not send every call to the newest model by default. Add a triage step that picks the cheapest model that meets your quality bar for that call. This is the AI version of rightsizing an EC2 instance. Many teams run too much work on the most expensive tier, because the quickstart used it.

Batch processing, for anything that isn’t real-time

Some work needs no instant reply, such as nightly enrichment, bulk classification, and backfills. Batch APIs typically process this work at a discount to standard pricing. In return, the answer arrives later. This is one of the easiest wins here. The task stays the same. Only when the response arrives changes. Many teams still send every workload to the real-time API, even when nothing waits on the answer.

Caching, model routing, and batch do not simply add up. You stack them by multiplying, not adding. Prompt caching also has a break-even point. Below it, caching costs more than no caching. See how to cut your LLM bill without downgrading your product for the worked-out waterfall.

Why this is invisible in most FinOps tooling

Traditional cost management was built for cloud infrastructure: compute, storage, and network. Cost Explorer, Azure Cost Management, and most third-party FinOps platforms predate token-based AI billing. That shows in what they can ingest. They have no native input/output token split. They have no per-model or per-key breakdown. They cannot show AI spend next to cloud spend on one axis. So AI spend either stays out of the FinOps conversation, or it shows up as one unexplained lump beside line items broken out in detail.

CostMon closes that gap. It has connectors for AI providers: Anthropic, OpenAI, Amazon Bedrock, Vertex AI / Gemini, and Helicone. Each connector pulls cost data from the provider and normalizes it into the same unified view as your AWS spend. “How much do we spend on cloud” and “how much do we spend on AI” then live on one dashboard.

The near-term trajectory only makes this more urgent

Token prices per unit have fallen as models get more efficient. But usage has grown faster than prices have fallen. More features call models. More agent workflows make several calls per task. Each call carries more context. That pattern sits behind most “our AI bill tripled and we don’t know why” conversations. It is rarely one expensive call. It is dozens of moderate calls, made more often than anyone budgeted. Most systems do not show which team, key, or model is behind them. Nobody flags the pattern until an invoice makes the total hard to ignore.

Get that breakdown early. Caching, model routing, and batching are all choices you can make on your own schedule. You then avoid explaining the bill after the fact.