Partition Count: Too Few, Too ManySpark · easy · ~10 minTwo jobs, two opposite diseases. Job A aggregates a 2TB shuffle with the default 200 shuffle partitions — tasks average 10GB each, spill constantly, and some OOM. Job B processes 3GB of data with a hand-set 40,000 partitions — 75KB per task, and the runtime is dominated by scheduling overhead.Fix both with one principle.Evidence

Job Evidence

Config

Checkpoints

Diagnose the pair.

Using a 256MB-per-task target, roughly how many shuffle partitions should Job A use for its 2TB shuffle? (a round number is fine)

What makes this whole class of tuning mostly disappear?

Job B's 40,000 was set 'because a blog said more parallelism is better'. Give the interview-style explanation of why partition count is a Goldilocks number.