AI & LLM cost
Prompt caching
A pricing and speed feature that discounts and speeds up tokens repeated across requests. You do not pay the full input rate each time.
Last updated
Definition
Many AI workloads resend the same large block of context on every request. It may be a system prompt, a knowledge-base excerpt, or tool definitions. They pay the full input-token rate each time for content that has not changed. Prompt caching lets the provider reuse that already-processed context across calls. It bills the repeated parts at a steep discount against a fresh read.
Caching helps most with workloads that feel expensive at scale. One example is high-volume chat with a large fixed system prompt. Another is an agent loop that resends the same tool schema on every step. Put the static part of the prompt first and keep it byte-for-byte identical. That is what makes the cache hit.
Where it shows up
Prompt caching appears as a separate line in the usage object of the API response. It reports cached tokens next to the regular input and output token counts. The provider bills those tokens at its published cache-read rate, not at the full input rate.
What makes it expensive
This cache breaks silently. Edit the static part of a prompt so it is no longer byte-for-byte identical across calls. Nothing errors and nothing warns. The full input rate quietly returns. The usage object reports cached tokens on every response. A workload that resends the same context and reports none has already lost the cache.
Related
Token spend moves every day.
Model prices and rate limits change between invoices. CostMon tracks your AI spend daily, so a change shows up before the bill does.