Pipeline & Orchestration EngineeringAirflow at scale, backfills and idempotency, DAG discipline, and the in-house engines — Netflix Maestro to Pinterest Spinner.Orchestration is where data engineering's software-engineering discipline shows: Shopify's multi-tenant Airflow lessons, Snap's multi-cluster DAG switching, Netflix rebuilding Maestro's engine 100× faster, and the backfill/idempotency machinery every senior interview probes.Week 2: Backfills, idempotency & incremental stateThe rerun-safety stack: the idempotency contract (intervals, not wall clocks), blue-green reloads, the incremental-state boundary rule, and lineage-driven blast-radius control.

Figure 1 — the rerun-safety stack: the idempotency contract at the bottom, consumer-safe swaps above it, bounded incremental state above that, and lineage-driven blast-radius control on top. By the end of this week you can say whether any given pipeline is safe to re-run — and what breaks if it isn't

A pricing bug corrupted eleven days of a core table. The fix is deployed; now someone has to re-run history. You type the backfill command and stop — because you've seen this movie. Last year, a re-run *duplicated* three days of records: the job appended without checking. The year before, a re-run produced *different numbers* than the original: the code called datetime.now(), and "now" had moved. The command is one line. Whether it repairs your data or corrupts it twice depends entirely on properties someone did — or didn't — design in months ago. Which properties?After this week you canThe full week 2 brief is part of LeetData Pro.Case studies

Eppo — Incremental Pipelines: Managing State at Scale

Airbnb — How Airbnb Achieved Metric Consistency at Scale

OpenLineage — Backfilling Airflow DAGs using Marquez

Astronomer — DAG writing best practices in Apache Airflow | Astronomer Documentation

Storyblocks — Blue-Green ETLs with Airflow Task Groups

Line — Introducing Frey: LINE’s new self-service batch ingestion system

Exercises

Backfills & Idempotency — Week 2 Check

The catchup=True Flood

A Week of Double-Writes

Late Data vs the Execution Date

Which Day Is This Run For?