The Monitoring Bill Nobody Budgets For

Most lines on a cloud bill grow because someone decided they should. You add instances, storage, or traffic. An observability bill grows for reasons nobody decided.

A developer adds a customer_id tag to a metric for better debugging. An autoscaling group spikes for a few hours overnight. A logging library gets a little more verbose in a routine release. None of these is a provisioning choice.

So a team can rightsize its EC2 instances and clean up orphaned volumes. That same team can still see its Datadog or CloudWatch invoice triple. The growing meter is inside the monitoring platform. It is not in the infrastructure that the platform watches.

Host billing is not an average

Datadog prices infrastructure hosts on a high-water mark, not a monthly average. Datadog’s billing documentation defines the billable host count. It is “the maximum count (high-water mark) of the lower 99 percent of usage” for the month’s hourly counts.

So Datadog drops the top outlier hours before it takes the maximum. A spike shorter than about 1% of the month (roughly seven hours) can fall in that dropped slice. It then barely registers.

A longer spike is different. So is a spike that repeats until it stops being an outlier. Datadog bills that peak host count as if it ran for the whole month.

You cannot remove every spike. You can cap how high autoscaling goes and how long it stays there. Then it stops looking like a normal level of use.

A tag can turn one metric into thousands

Custom metrics do not bill per metric name. They bill per unique combination of metric name and tag values, including the host tag. Datadog calls this cardinality.

Tag one metric by customer_id across a growing user base. That single metric.submit call in your code becomes thousands of billable series. Nobody wrote a new metric name.

Histogram and distribution metric types add cost before any high-cardinality tag appears. By default, a HISTOGRAM creates roughly five billable custom metrics for each unique tag combination. They are max, median, average, a percentile, and count. Turn on percentile aggregations for a DISTRIBUTION and you add another five.

A metric that looks like one line of code can be ten billable series before cardinality counts at all.

Ingest and index are two different bills

Datadog and AWS CloudWatch both charge for receiving a log. They charge separately for keeping it searchable. These are two meters.

A log line triggers an ingestion charge when it arrives. That happens whether or not anyone indexes it. Many teams send everything to full-price indexing by default. They do not decide case by case what needs to be searchable. This is a common reason a logging bill outgrows the traffic it describes.

CloudWatch has a matching trap on the storage side. The default retention for a log group is “Never Expire.” Every byte you ingest keeps billing for storage until someone sets a limit. Nothing fails when the setting stays at its default. So nothing forces anyone to notice.

The obvious fix has a tradeoff. You can move a log group to a cheaper access tier. CloudWatch has the Infrequent Access class. Datadog has Flex Storage. Both lower the ingestion or storage rate by a large amount.

The cheaper tier has limits. AWS’s launch announcement for Infrequent Access says Standard stays the right choice for logs that need:

“real-time operational visibility and advanced analytical capabilities such as extracting metrics from logs, anomaly detection, sensitive data protection, or real-time logs analysis”

The cheaper tier supports none of those. Both tiers still support Logs Insights queries. Moving a log group is a safe choice when nothing derives a live signal from it. It is a bad choice for any log group that feeds one.

What catches this before the invoice

You do not need to watch a dashboard all day. You need what any silent-creep cost needs: a normalized, daily view of what drives the number. Check it on a schedule. Do not discover it on the invoice.

Cost Alerts & Anomaly Detection explains how to catch a spend spike. It works for observability tools and for anything else. You catch the spike while it is still small enough to be routine. You do not wait until finance asks about a line item.

The observability cost calculator has the rate card behind every number above. It covers Datadog host, custom-metric, APM, and log pricing. It also covers CloudWatch logs, metrics, and API meters. It prices both platforms meter by meter. It ranks the config change behind each cost.

If the AWS side of your bill is the bigger mystery, read the AWS Bill Decoder. It explains what the rest of a CloudWatch or EC2 line item means.

CostMon does not rewrite a Datadog tag. It does not set a CloudWatch retention policy for you. It pulls your Datadog and AWS cost data on the same daily schedule as every other provider you run. An observability line that climbs for reasons unrelated to your traffic then shows up while it is still a small number.