AI & LLM cost
Batch API
An asynchronous request mode that trades a longer turnaround (minutes to hours) for a significant discount versus real-time API calls.
Last updated
Definition
Batch processing submits a large set of requests together and returns results once they've all completed, rather than one response per call in real time. Because the provider can schedule that work more efficiently than an on-demand, latency-sensitive request, batch pricing is typically discounted 50% or more versus the standard API rate.
It's a strong fit for anything that doesn't need an immediate answer (bulk classification, offline evaluation, nightly summarization jobs) and a poor fit for anything user-facing where response time matters.
Where it shows up
A batch API request is submitted through its own dedicated endpoint, separate from the standard real-time completion endpoint, and the provider's usage dashboard reports batch token volume as its own category, billed at its own, discounted rate shown separately from the real-time total on the same invoice.
What makes it expensive
The costly mistake is routing a latency-sensitive, user-facing request through the batch endpoint by default rather than checking whether the workload can tolerate a delay of minutes to hours. The reverse error is quieter and far more common: nightly jobs and bulk classification left on the real-time endpoint, paying a premium for speed nobody needed.
Related
Token spend moves every day.
Model prices and rate limits change between invoices. CostMon tracks your AI spend daily, so a change shows up before the bill does.