AI & LLM cost
Prompt caching
A pricing and performance feature that discounts (and speeds up) tokens repeated across requests, instead of billing the full input rate every time.
Last updated
Definition
Many AI workloads resend the same large chunk of context (a system prompt, a knowledge-base excerpt, tool definitions) on every single request, paying the full input-token rate each time for content that hasn't changed. Prompt caching lets the provider reuse that already-processed context across calls, billing repeated portions at a steep discount versus a fresh read.
It's most valuable for the workloads that feel expensive at scale: high-volume chat with a large fixed system prompt, or an agent loop that resends the same tool schema on every step. Structuring prompts so the static portion comes first and stays byte-for-byte identical is what makes the cache hit.
Where it shows up
Prompt caching shows up as a separate line in the API response's usage object, reporting cached tokens alongside the regular input and output token counts, and billed at the provider's own published cache-read rate rather than at the full input rate for those same tokens.
What makes it expensive
This cache breaks silently. Edit the static portion of a prompt so it is no longer byte-for-byte identical across calls, and nothing errors, nothing warns, and the full input rate quietly returns. The usage object reports cached tokens on every response. A workload that resends the same context and reports none has already lost the cache.
Related
Token spend moves every day.
Model prices and rate limits change between invoices. CostMon tracks your AI spend daily, so a change shows up before the bill does.