Airflow at scale, backfills and idempotency, DAG discipline, and the in-house engines — Netflix Maestro to Pinterest Spinner.
Orchestration is where data engineering's software-engineering discipline shows: Shopify's multi-tenant Airflow lessons, Snap's multi-cluster DAG switching, Netflix rebuilding Maestro's engine 100× faster, and the backfill/idempotency machinery every senior interview probes.
The ceilings that force a move, fork-vs-build-vs-adopt with production proofs, the actor-model engine hot path, and migrations with per-workflow rollback — decisions a platform lives with for a decade.
The platform review lands on one slide: scheduling delay has crept to seventy seconds, the vendor-default dashboard shows the scheduler "healthy," and last month it deadlocked while its process stayed green — alive to the health check, dead to every workflow. Someone says the word everyone's been avoiding: *migrate*. Three engineers propose three futures — fork what we have, adopt something newer, build our own — each certain, none with numbers. You've seen this meeting end badly at two companies. What evidence would make it end well?
The full week 4 brief is part of LeetData Pro.
