Cloud cost
Bytes scanned & partition pruning
On-demand BigQuery bills by how much data a query reads, not by how long it runs. Partition pruning and column selection are the two levers that shrink that number.
Last updated
Definition
The bytes a query scans set its cost, full stop. A query that reads a whole table in one second costs more than one that reads a single day's partition in a minute. Two habits inflate the number for no benefit. The first is leaving out a filter on the partitioning column, which forces a scan of every partition. The second is selecting every column with SELECT * when you use only a few. BigQuery uses columnar storage, so columns you do not reference cost nothing to skip.
The require_partition_filter setting turns a missing partition filter into a query error, not a silent full scan. Of the two fixes, it is the more durable.
Where it shows up
Bytes scanned appears in the BigQuery console's query validator before a query runs. It appears again in the job details after the query completes. The require_partition_filter table setting is visible in the table's configuration. When you turn it on, a missing partition filter becomes a query error, not a silent scan.
What makes it expensive
SELECT * against a partitioned table, with no filter on the partitioning column, is the expensive default. It reads every partition and every column for a query that needed one day and three fields. The console shows bytes scanned before the query runs. Almost nobody looks. The habit spreads across a team quickly.
Related
More in Cloud cost
- Committed use discount (CUD)
- Reserved instance (RI)
- Savings plan (SP)
- On-demand vs. spot instances
- Data egress
- Amortized, blended, and unblended cost
- Billed vs. effective vs. list vs. contracted cost
- Rightsizing
- Idle and orphaned resources
- Storage class
- Lifecycle policy
- Incremental snapshot
- Allocatable capacity
- Requests vs. limits
- Cluster idle
- Workload idle
- Bin packing
- Cost allocation tags
- Tag coverage
- Unallocated spend
- Showback vs. chargeback
- Unit economics
- RI/SP coverage and utilization
- Observability tax
- Metric cardinality
- High-water-mark billing
- Log ingest vs. index
- Snowflake credit
- Auto-suspend / resume minimum
- BigQuery slot
- Time Travel & Fail-safe
Connect it. Get the first alert.
Connect a provider and CostMon builds a daily baseline for it. From there, a spike reaches your team as an alert.