The Monitoring Bill Nobody Budgets For

Every other line on a cloud bill grows because someone decided it should: more instances, more storage, more traffic. An observability bill grows for reasons nobody decided at all. A developer adds a customer_id tag to a metric for better debugging. An autoscaling group spikes for a few hours overnight. A logging library gets a little more verbose in a routine release. None of those are provisioning choices, which is exactly why the team that diligently rightsizes EC2 instances and cleans up orphaned volumes can still watch its Datadog or CloudWatch invoice quietly triple. The meter that’s growing lives inside the monitoring platform, not the infrastructure it’s watching.

Host billing isn’t an average

Datadog prices infrastructure hosts on a high-water mark, not a monthly average. Per Datadog’s own billing documentation, the billable host count is “the maximum count (high-water mark) of the lower 99 percent of usage” for the month’s hourly counts, meaning the very top outlier hours get excluded before the maximum is taken. A spike that lasts less than roughly 1% of the month (about seven hours) can fall inside that excluded slice and barely register. A spike that runs longer, or repeats often enough that it stops being an outlier, gets billed as if that peak host count ran for the entire month. The lever isn’t eliminating every spike. It’s capping how high and how long autoscaling is allowed to run before it stops looking like an outlier.

A tag can turn one metric into thousands

Custom metrics don’t bill per metric name. They bill per unique combination of metric name and tag values, including the host tag. This is what Datadog calls cardinality. Tag one metric by customer_id across a growing user base and that single metric.submit call in your code becomes thousands of billable series without a new metric name ever being written. Histogram and distribution metric types make this worse before a single high-cardinality tag even enters the picture: a HISTOGRAM generates roughly five billable custom metrics per unique tag combination by default (max, median, average, a percentile, and count), and turning on percentile aggregations for a DISTRIBUTION stacks another five on top. A metric that looks like one line of code can be ten billable series before cardinality even factors in.

Ingest and index are two different bills

Both Datadog and AWS CloudWatch charge for receiving a log and for keeping it searchable as two entirely separate meters. A log line triggers an ingestion charge the moment it arrives, whether or not it’s ever indexed. Sending everything to full-price indexing by default, rather than deciding case by case what actually needs to be searchable, is one of the most common ways a logging bill outgrows the traffic it describes. CloudWatch has a matching trap on the storage side: a log group’s default retention is “Never Expire.” Every byte ever ingested keeps billing storage indefinitely until someone sets an explicit limit, and because nothing fails when that setting is left alone, almost nothing forces anyone to notice.

There’s a real tradeoff hiding in the obvious fix, too. Moving a log group to a cheaper access tier, CloudWatch’s Infrequent Access class, or Datadog’s Flex Storage, cuts the ingestion or storage rate substantially, but it isn’t a free discount. AWS’s own launch announcement for Infrequent Access is direct that Standard remains the right choice for logs needing “real-time operational visibility and advanced analytical capabilities such as extracting metrics from logs, anomaly detection, sensitive data protection, or real-time logs analysis”, none of which the cheaper tier supports, even though both tiers still support Logs Insights queries. It’s a safe move for a log group nothing is deriving a live signal from, and a bad one for anything that is.

What actually catches this before the invoice does

None of this requires watching a dashboard all day. It requires the same thing catching any other silent-creep cost requires: a normalized, daily view of what’s actually driving the number, checked on a schedule rather than discovered on the invoice. Cost Alerts & Anomaly Detection walks through the mechanics of catching a spend spike, in observability tooling or anywhere else, while it’s still small enough to be routine instead of a line item finance is asking about after the fact.

The observability cost calculator has the exact rate card behind every number above: Datadog’s host, custom-metric, APM, and log pricing, plus CloudWatch’s own logs, metrics, and API meters. It prices both platforms meter by meter and ranks the config change behind each saving. If the AWS side of your bill is the more immediate mystery, the AWS Bill Decoder covers what the rest of a CloudWatch or EC2 line item actually means.

CostMon doesn’t rewrite a Datadog tag or set a CloudWatch retention policy for you. What it does is pull your real Datadog and AWS cost data on the same daily schedule as every other provider you run, so an observability line climbing for reasons that have nothing to do with your traffic shows up while it’s still a small number, not a large one you’re explaining after the fact.