Bill anatomy, order-of-magnitude cuts, engine decisions, and real-time OLAP — the corpus's deepest pool (90 companies).
From Foodpanda's −45% BigQuery bill to a 93% COUNT(DISTINCT) cut with HyperLogLog, from the Redshift→Snowflake→Databricks migration genre to Netflix's trillion-row Druid — warehouse cost and performance is the most universally interviewable material in data engineering. Includes the Data Modeling module (SCD2 mechanics, Picnic's data vault, warehouse-first architecture).
Three meters, an attribution layer, and the incentive gradient underneath — decompose the invoice, make idle a first-class cost, and know why governance is yours to build.
Monday morning, finance forwards the invoice with one line highlighted: compute credits, up 40% month over month. Nothing big shipped. Headcount is flat. You open the console: fourteen warehouses, all green, all "running fine." The invoice tells you *what* you owe, not which of the fourteen — or which hours, or whose queries — put the number there. Before you can cut anything, you have to answer a question the bill itself refuses to answer: where, exactly, does a warehouse dollar go?
A warehouse bill is the sum of three meters with different physics. Learn the meters and invoices stop being surprises.
The compute meter dominates — at Wise it runs ~90% of the Snowflake bill against ~10% storage, and that split is typical. Its mechanics: a virtual warehouse accrues credits per second *only while running*, with a 60-second minimum charged on every start, resume, or resize. The rate card doubles per size: an X-Small burns 1 credit/hour, and each tier up doubles it. Two consequences fall straight out. First, idle-but-running is pure waste — which is why auto-suspend settings are the cheapest optimization in the product. Second, the 60-second minimum quantizes rapid cycling: resume a warehouse, run a 5-second query, suspend — you paid for a minute, twelve times the work you did. A fleet that resumes hundreds of times a day pays that quantization tax constantly.
The storage meter is a flat monthly rate per compressed terabyte — but the metered footprint is bigger than your live data, because Time Travel and Fail-safe — the historical copies powering point-in-time recovery — bill too. A table with heavy churn and long retention can carry a history multiple times its live size.
The transfer meter is the quiet one: ingress is billed by the cloud provider (not the warehouse vendor — "free from Snowflake" is not free), and egress bills on cross-region and cross-cloud movement. Staying single-region zeroes the line; replication turns it into a standing cost.
And one trap meter: cloud services — parsing, metadata, optimization — is free *until* its daily usage exceeds 10% of that day's warehouse compute, and then only the excess bills. Metadata-heavy workloads (swarms of tiny queries, INFORMATION_SCHEMA crawlers) cross the threshold and materialize a line item that didn't exist last month.
INFORMATION_SCHEMA
🏭 In production at Wise: the levers follow the meters — a 2-minute auto-suspend on everything, a scheduled task that *downsizes* warehouses in known demand valleys, and a fleet deliberately concentrated on X-Small and Small sizes. Nothing clever; each lever maps to one meter's mechanics.
🤔 Check yourself: A team's dashboard fires 30 independent refresh queries spread across each hour, each ~4 seconds, against a dedicated Medium warehouse (4 credits/hr) with 60-second auto-suspend. Estimate what fraction of the paid time did useful work, and propose the cheapest fix.
Each firing pays resume + ~60s minimum + a 60s idle tail before suspend: call it ~2 minutes paid per 4 seconds of work — ~3% useful. Fixes, cheapest first: batch the 30 queries into one refresh window (one resume, one tail), and downsize — 30 four-second queries don't need a Medium. Consolidation beats tuning.
The bill's unit is the warehouse-hour. Your levers are queries. Between them sits an attribution problem, and how you solve it decides whether optimization ever targets the right thing.
The mechanism: within any metered hour a warehouse serves many concurrent queries *and* idle gaps. To attribute, slice every query and every idle window into per-hour slots, then split that hour's actual metered credits proportionally by active seconds — queries get their share, and idle becomes a first-class cost object with its own line. Built this way, the allocation reconciles to the metering history — the billing source of truth — by construction, and idle stops hiding inside "query costs."
Why not just use the vendor's native attribution view? A validation experiment answers it: an X-Small with a 600-second auto-suspend ran mostly sub-half-second queries for an hour — about 40 seconds of real execution. The native view attributed 0.43 of the hour's 0.88 credits — roughly half — to those 40 seconds of queries, because it *excludes idle time entirely* (and drops sub-100ms queries, and lags hours). The custom time-weighted method assigned the credits where they belonged: to the idle window the auto-suspend setting created. The lesson generalizes: validate any vendor cost tool by reconciling it against the metering table; if attribution doesn't sum to the bill, someone's costs are being hidden — usually idle, which is precisely the cost your configuration controls.
Attribution feeds the capacity decision. On-demand pricing (BigQuery's $5/TB scanned, Snowflake's pay-per-credit) has no fixed cost and no ceiling; committed capacity (slot reservations, capacity contracts) inverts the deal — fixed spend, fixed throughput. The break-even is arithmetic once you've measured: pull the real usage (slot-milliseconds, credit consumption) from the metadata tables, size the 24×7 baseline, and commit to that — then layer short-term flex capacity on scheduled peaks.
🏭 In production at Foodpanda: exactly that sequence — three months of measured INFORMATION_SCHEMA slot data, annual commitments for the baseline, Airflow jobs buying and cancelling flex slots around working-hours peaks, and the pool carved into reservations per workload group (ETL, ad-hoc, dev) so idle capacity shares but no group hoards. Result: BigQuery spend down 45% — with average query time up from 11 to 17 seconds, accepted explicitly.
⚖️ Tradeoff: commitment converts cost risk into performance risk. On-demand, an inefficient query silently inflates the bill; on a fixed pool, it steals slots from colleagues — queues form, dashboards slow. Foodpanda counted the culture shift a win: waste now visibly hurt peers, so query authors started owning efficiency. Your chargeback design picks which pressure you want.
🤔 Check yourself: Your BigQuery on-demand spend is $18k/month at $5/TB. A 2,000-slot annual commitment costs ~$12k/month and would serve the same workload. Name two measurements you'd take before switching, and the failure mode if you skip them.
Measure (1) slot-ms by hour-of-day from job metadata — does demand have a stable baseline commitments can cover, or is it spiky? (2) queueing/concurrency at peaks — will 2,000 slots throttle what on-demand absorbed elastically? Skip them and the failure mode is Foodpanda's tradeoff without consent: everything slows at peak, and you bought a year of it.
One more mechanism completes the anatomy, and it's economic, not technical. Consumption pricing means the vendor's revenue is your consumption: every wasted scan is their growth. The public numbers say it plainly — net revenue retention where a customer's $1 becomes ~$1.70 a year later, overwhelmingly from consumption expansion. This doesn't make vendors villains; it explains structural priorities. Features that *reduce* your spend — per-user budget caps, aggressive per-query attribution, optimizer fixes for cost-pathological-but-correct queries — arrive slowly. Features that grow consumption arrive fast.
Two field observations give the incentive teeth. Cost concentrates absurdly: on one analyzed account, roughly 5% of users drove 95% of spend — a handful of SELECT * dashboards and cross-joins nobody was watching. And the platform's own scheduling can upsell: a roughly-FIFO queue lets small interactive queries pile up behind big scans, and the vendor-recommended remedies — bigger warehouse, multi-cluster, acceleration services — all *raise* spend rather than reorder work.
SELECT *
So the governance layer is yours to build, and this week gives you its parts list: attribution that reconciles to the bill (with idle first-class), per-team budgets and alerts on the concentration curve, guardrails on the query shapes that defeat the optimizer (next week's subject), and a standing review of the capacity model against measured demand. Week 2 turns to the other half of the program: making the queries themselves an order of magnitude cheaper.
⚠️ Gotcha: "5% of users drive 95% of cost" is an opportunity *and* a trap. It means three fixes can move the bill double digits — and it means fleet-wide mandates ("everyone must add clustering keys!") burn goodwill on the 95% of users who were never the problem. Aim governance at the head of the distribution, not the tail.
🤔 Check yourself: Design the minimum viable cost-governance stack for a 200-person analytics org on a consumption-priced warehouse: name three components, the data source each reads, and which failure each one catches.
(1) Attribution reconciling to the metering table (source of truth), idle separated — catches configuration waste and misattributed blame. (2) Per-team spend dashboards with anomaly alerts on the top-N spenders — catches the 5%-user problem within days instead of at invoice time. (3) Query-shape guardrails (bytes-scanned budgets, required partition filters, review on new scheduled jobs) — catches pruning-hostile patterns before they become standing costs.
| Concept | One-line mechanism | Number to remember | Production proof |
|---|---|---|---|
| Compute meter | per-second credits while running; 60s minimum per start/resume/resize; rate doubles per size | ~90% of the bill is compute | Wise bill anatomy |
| Idle & quantization | resume + short query + suspend pays the minimum; idle bills like work | 40s of queries ≠ 49% of an hour's credits | GreyBeam attribution experiment |
| Attribution | slice queries + idle into hourly slots; weight by active seconds; reconcile to metering | must sum to the bill by construction | custom method vs native view gaps |
| Capacity models | measure slot/credit demand → commit baseline → flex the peaks → carve reservations | −45% cost, 11s → 17s queries | Foodpanda reservations |
| Incentive gradient | consumption pricing rewards your inefficiency; governance won't come from the vendor | $1 → ~$1.70 NRR; 5% of users ≈ 95% of cost | Sajin's analysis |
Bill-anatomy questions separate people who've *owned* a warehouse from people who've queried one. Expect: *"Our Snowflake bill jumped 40% — walk me through your first hour"* (meters first: compute vs storage vs services split, then attribution — which warehouses, which hours, how much is idle; only then query hunting). *"Design cost-per-query attribution when the platform bills warehouse-hours"* (hourly slotting, active-second weighting, idle as first-class, reconciliation to metering as the correctness test). *"When do slot commitments beat on-demand?"* (measured baseline vs spiky demand, break-even arithmetic, and the cost-risk-to-performance-risk conversion with its noisy-neighbor consequence). *"Why does the vendor not just fix this?"* (consumption-pricing incentives, stated without conspiracy — and what that implies you must build). Senior candidates volunteer the reconciliation test and the idle question unprompted; that's the tell that they've closed the loop between a bill and a fix.
