AI & LLM cost
Context window
The maximum number of tokens (prompt, conversation history, and generated output combined) a model can process in a single request.
Last updated
Definition
Every model has a fixed context window measured in tokens; once a conversation or prompt exceeds it, older content has to be truncated or summarized before the model can accept the request. Larger context windows unlock use cases (long documents, extended conversations, large codebases) but every additional token of context sent is also an additional billed input token. A bigger window is a cost lever, not a free upgrade.
Workloads that stuff a large, mostly-static context into every request (a knowledge base, a system prompt, prior turns) are where prompt caching has the most cost impact.
Where it shows up
A model's context window limit is published on the provider's own model documentation page as a fixed token count, and a request that exceeds it returns an error from the API or gets silently truncated by whatever client code assembled the prompt beforehand, before the request is even sent.
What makes it expensive
A common expensive habit is stuffing a large, mostly static context into every request, such as a knowledge base or a long system prompt, paying the full input rate on it each time. An input token count that barely varies from call to call is describing a fixed payload. Fixed payloads are exactly what prompt caching exists for.
Related
Token spend moves every day.
Model prices and rate limits change between invoices. CostMon tracks your AI spend daily, so a change shows up before the bill does.