Salting a Hot-Key AggregationSpark · hard · ~14 minA groupBy(customer_id).agg(...) over clickstream data has one brutal customer: an internal load-test account generating 31% of all events. AQE's skew-join handling doesn't apply — this is an aggregation, and every event for that key must reach the same reducer. The stage straggles for 50 minutes on one task.Filtering the account is forbidden (its numbers are reported). Design the two-phase salted aggregation.Evidence

Stage Evidence

Config

Checkpoints

Why doesn't AQE (or more partitions) save an aggregation the way it can save a join?

The two-phase salted design?

What are the costs and the correctness trap? When would you NOT bother salting? Interview-style.