Spark Internals — Senior ScreeningSenior-level Spark: the tradeoffs and internals that separate people who've operated it from people who've read about it.

Why can't raising spark.sql.shuffle.partitions fix a single hot key?

AQE can convert a sort-merge join to broadcast at runtime because...

A broadcast join OOM'd the fleet after months of working. Most likely cause?

Remote shuffle-spill appears in a stage's profile. The FIRST lever to consider?

Dynamic Partition Pruning applies when...

One 88GB-heap executor per node is slower than five 16GB executors mainly because...

For a nightly batch on spot/preemptible executors, the resilient pattern is...

A job shuffles 16x its input size. The most likely class of cause?