The Airbnb quality arc, contracts from discourse to production, validation frameworks, and SLOs for data.
The most editorially distinctive cohort — data contracts are the debate this newsletter helped shape. Production anchors: Airbnb's Wall→Quality-Score arc, Uber's operational excellence program, Meta's Table Compare, GoCardless and Whatnot shipping contracts for real.
The platform arcs: scoring, ownership, and what quality-at-scale programs actually institutionalize — Airbnb's rebuild, Uber's operating model, Twitter's automation, Criteo's DataDoc.
Your team took data quality seriously this year. Forty carefully written checks became four hundred. The alerts channel scrolls all day. And this morning a VP asked the question the whole effort was supposed to answer — "can I trust this table?" — and nobody could say yes with a straight face. You have more checks than ever. Why is trust *lower*?
That inversion is this week's subject: why checks are the easy part, and what the platform around them — ownership, consequence, visibility — actually does.
A check is just an assertion — a query that returns pass or fail. Trust is a property of the *system around* assertions, and that system has three load-bearing questions. Who owns a failure? An alert routed to a shared channel is owned by nobody, and unowned alerts decay into scroll-past noise on a schedule you can predict. What happens when it fires? A check with no consequence — nothing halts, nothing pages a specific person — is decoration. Can consumers see quality where they choose data? If the only people who see check results are the people who wrote the checks, quality is invisible exactly where it would change decisions.
Airbnb's rebuild is the canonical evidence that the platform starts organizationally, not technically. After hypergrowth eroded trust, the first moves were a dedicated data-engineering craft, domain-owned subject areas replacing one monolithic star schema, and Midas certification — a review process requiring an approved design spec (metrics, schemas, business logic) before any build, with the badge surfaced everywhere consumers meet the data. Then came the instructive surprise: an internal survey showed people distrusted *everything without the badge*. Certification is deliberately expensive and therefore sparse — which created the coverage problem the rest of this week solves.
🤔 Check yourself: Your 400 checks alert into one shared #data-quality channel. Predict the steady state in three months, and name the two structural fixes this section implies.
Steady state: alert fatigue — the channel becomes ambient noise, real incidents hide among false positives, and trust doesn't move because consumers never see check results anyway. Fixes: route every failure to a named owner (per-dataset ownership, not a channel), and surface quality state at the point of consumption (catalog, not channel) so the signal reaches the people making trust decisions.
Every check needs one more decision than "what to assert": what happens on failure. A blocking check is a circuit breaker — the pipeline halts, and downstream consumers see *stale but correct* data. A non-blocking check lets data flow and raises an alert — consumers see *fresh but suspect* data. Neither is safer in general; the choice is failure economics. For financial reporting, stale beats wrong by miles: yesterday's correct revenue is useful, today's wrong revenue is a incident. For an ML feature pipeline, wrong-ish rows may dilute into noise while missing data breaks serving — fresh can beat perfect.
Airbnb's Wall is the pattern industrialized: teams declare checks in YAML, the framework generates the Airflow tasks and wires dependencies, every check declares blocking or non-blocking, and configuration cascades hierarchically (team → file → table → check) so defaults live at the top and exceptions stay local. Migrating cut DAG code by more than 70% — the checks got *easier to own* — and thousands now run daily across every critical business and financial pipeline.
⚠️ Gotcha: blocking checks inherit the pipeline's blast radius. A too-eager blocking check on a shared upstream table halts every downstream consumer at once — the quality system becomes the outage. Blocking status deserves the same review rigor as the SQL, and Wall gates custom check types behind a paved-path team for exactly this reason.
🤔 Check yourself: A freshness check fails at 6 a.m. on the revenue table that feeds both the CFO dashboard and an ML pricing model. Same check — same posture for both consumers?
No. The dashboard wants blocking semantics: stale-but-correct, with the morning number delayed rather than wrong. The pricing model may prefer flowing data with an alert — a few hours of slightly stale features usually beats serving from nothing. Which is the argument for making blocking a property of the *consumer edge*, not only the check: one producer, per-consumer failure economics.
Hand-written checks scale with team effort; warehouses scale faster. The structural answer is auto-generation: most quality checks belong to standard classes — freshness, completeness, duplicates, cross-datacenter consistency — whose parameters are derivable from metadata, and lineage tells you *where* checks matter (upstream of critical assets). Uber's UDQ runs this at full scale: roughly 100,000 assertions a day from ~18,000 definitions across 2,000+ critical datasets, auto-generated from metadata and lineage, catching about 90% of quality incidents automatically — coverage maintained by both push (metadata-change events) and pull (batch refresh).
The most senior idea in this whole space rides alongside: UDQ measures itself — precision and recall against real incidents. "Catches 90% of incidents" is a recall claim; without precision (what fraction of alerts were real?) it's compatible with an alert firehose nobody reads. The alert-engineering that protects precision: *sustain periods* (a condition must hold before alerting — trading detection latency for fewer flappy false positives), *dependency-based suppression* (one root cause upstream produces one alert, not fifty downstream echoes), and a grace period before new alerts page anyone.
⚠️ Gotcha: auto-generation has two standing maintenance costs. Lineage goes stale as ETL evolves — UDQ prunes obsolete tables daily — and primary keys still need human declaration. And no automated coverage is complete: a user-reported incident channel remains part of the system, not an admission of failure.
🤔 Check yourself: A vendor pitches "our platform detected 90% of your incidents." What's the missing number, and what's the fastest way it could be gamed?
Precision. Recall alone is gameable by alerting on everything — 100% detection, 2% precision, and a team trained to ignore the channel (the alert-fatigue scenario this cohort runs makes you do this arithmetic: hundreds of alerts, a handful real). A platform claim needs both numbers, measured against a labeled incident history.
The last organ is incentive design. Enforcement doesn't scale — you can't review every table — but visible quality state changes producer behavior without a mandate, because producers respond to what their consumers can see. Airbnb's Data Quality Score is the worked example: an automated 0–100 score on *every* warehouse asset across four weighted dimensions (Accuracy, Reliability, Stewardship, Usability), shown in the catalog beside the data, with concrete improvement steps attached — and certification preserved as the top tier, since only Midas-certified assets can score above 90. The score gives the 95% of assets that will never be certified a visible, improvable number.
Two companions complete the pattern. Twitter's platform lands every check result in BigQuery as a *queryable dataset* — and that storage decision is itself a mechanism: a pass/fail ping is consumed once and gone, while check results as rows support trend lines, week-over-week drift comparisons, and "when did this start?" forensics during incidents (their core served-impressions table is consumed by 400+ internal customers, each a potential drift victim). And Criteo's DataDoc adds the discovery layer: partition-level lineage captured by instrumenting Hadoop containers, plus an availability calendar reconciling what different systems *believe* about readiness — because Hive, HDFS, and Vertica routinely disagree.
🏭 In production at Criteo: the trigger for the lineage investment was an outage they couldn't trace through the dependency chain. Lineage isn't documentation — it's the incident-response map, and it's also what UDQ-style generation and alert suppression consume. Every organ in this week's figure feeds the others.
🤔 Check yourself: Why does Airbnb's score ship with *improvement steps* attached — what fails if you publish the number alone?
A bare score produces disputes, not improvement: producers who can't see *why* they scored 61 contest the metric instead of fixing the asset, and the score itself loses trust. Actionability is what converts visibility into behavior — the same reason a check without an owner is noise. Score, cause, and next step travel together or the incentive mechanism stalls.
| Concept | Mechanism in one line | Number to remember | Production proof |
|---|---|---|---|
| Platform > checks | Ownership + consequence + visibility turn assertions into trust | 70%+ DAG code cut by declaring checks | Airbnb Wall on all critical pipelines |
| Blocking semantics | Halt = stale-but-correct; alert = fresh-but-suspect; chosen by failure economics | thousands of checks daily | Wall's blocking/non-blocking split |
| Scaled coverage | Metadata + lineage generate standard checks; precision/recall grade the platform | ~100K assertions/day, ~90% incidents caught | Uber UDQ over 2,000+ datasets |
| Visible incentives | Scores in the catalog change producers where mandates can't | Only certified assets score >90 | Airbnb DQ Score; Twitter results-as-data |
Each objective maps to an interview move:
