The Migration That Almost Re-Ran HistoryPipeline · hard · ~13 minYour Airflow deployment hit the metadata-database wall, and the plan is approved: a second cluster, with DAGs migrating over in batches. You're designing the DAG Switching runbook before anyone moves anything.The pilot migration is billing_rollup — a daily DAG with catchup=True (it must self-heal schedule gaps), 3 years of run history, and a downstream that charges customers.A teammate's draft runbook is three steps: pause on cluster A, copy the DAG file to cluster B, unpause on B. In staging, that runbook just tried to launch 1,095 historical runs. You're rewriting it before the production attempt.Evidence

Staging dry-run

Runbook draft + cluster facts

Constraints

Checkpoints

Why did cluster B try to launch 1,095 runs?

The fix that keeps catchup=True (it's required) while preventing the history re-run is to seed cluster B with something before unpausing. What?

If the switch happens at 14:00 and the anchor is A's run for yesterday, how many runs should cluster B schedule in the first 24 hours after unpause (daily schedule, one interval completing at midnight)?

Write the production runbook for the fleet: the ordered steps with the exactly-one-cluster invariant, where coordination state lives (and why NOT Airflow Variables here), and the rollback story mid-migration.