The Backfill That Skipped the AuditQuality · medium · ~13 minA pricing-logic fix required recomputing 90 days of order_margin history. The engineer ran the backfill Friday evening — writing directly into the production table — and signed off when the job succeeded.The job did succeed. It also carried a join bug that duplicated rows for orders with multiple discounts. For six hours on Saturday, finance dashboards and two ML training jobs read margins inflated by the duplicates — until an analyst noticed margin > 100% on a big account.Remediate today's mess, then redesign backfills so this class of failure can't recur.Evidence

Metrics

Logs

Config

Checkpoints

The job reported SUCCESS. Why did that mean nothing?

Which existing, unused capability would have caught the bug BEFORE any consumer read it?

Row count went 41.9M → 55.6M on a table keyed by order_id. Roughly what percentage of the PUBLISHED table's rows were duplicates introduced by the join bug? (whole number, %)

Write the remediation for today AND the standing backfill policy: immediate rollback, consumer cleanup, and the rule that makes backfills safe by default.