Shuffle Explosion Under an SLASpark · hard · ~15 minThe revenue-attribution pipeline must land by 06:00 for finance. It currently finishes 06:40-07:10. Inputs total ~500GB, yet the job writes 8.1TB of shuffle. Leadership's proposed fix is doubling the cluster — which the infra budget explicitly caps. Find the 16x shuffle amplification and get under the SLA without new hardware.Evidence

Plan Walkthrough

Constraints

Checkpoints

Rank the fixes by leverage. Which is FIRST?

And the supporting moves, in sensible order after the filter pushdown?

Finance asks for confidence: give the senior narration — expected shuffle after fixes, why no new hardware is needed, and how you'll verify before the next 06:00.