AI & LLM cost
Context window
The maximum number of tokens a model can process in one request. It counts the prompt, the conversation history, and the generated output together.
Last updated
Definition
Every model has a fixed context window, measured in tokens. When a prompt or conversation goes over it, older content must be cut or summarized before the model accepts the request. Larger windows allow new uses, such as long documents, long conversations, and large codebases. But every extra token of context is also an extra billed input token. A bigger window is a cost lever, not a free upgrade.
Prompt caching has the most effect on cost for workloads that put a large, mostly static context into every request. Examples are a knowledge base, a system prompt, and earlier turns.
Where it shows up
The provider publishes a model's context window limit on its model documentation page as a fixed token count. A request over the limit returns an API error. In other setups, the client code that built the prompt truncates it silently before the request is sent.
What makes it expensive
Stuffing a large, mostly static context into every request is a costly habit. A knowledge base or a long system prompt is a common example. You pay the full input rate on it each time. An input token count that barely varies between calls describes a fixed payload. Prompt caching exists for fixed payloads.
Related
Token spend moves every day.
Model prices and rate limits change between invoices. CostMon tracks your AI spend daily, so a change shows up before the bill does.