Apache Spark in ProductionShuffle services, fleet-scale upgrades, cost engineering, and performance forensics — from Uber's 2M jobs to Wix's −60% bill.Spark tuning folklore is endless; this cohort sticks to what companies actually did. Uber upgraded 2M+ jobs fleet-wide with shadow execution, Wix cut costs 60% moving 5,000 workflows to EMR-on-EKS, LinkedIn rebuilt shuffle around sequential reads, PayPal cut a flagship job's cost 70%, and Amazon rebuilt its exabyte compactor on Ray — each taught as a mechanism first, with the company as proof.Week 2: Performance forensics: memory, skew, shuffleSymptom → subsystem: the UDF codegen cliff, join algorithms diagnosed from where they fail, memory forensics at both ends, and why shuffle physics eats 10–20% of clusters.

Figure 1 — the forensics map: each symptom on the left points at one subsystem on the right. By the end of this week you can run this diagram in your head — from "driver OOM" or "20% of runtime is fetch-wait" straight to the mechanism responsible

The job ran in forty minutes yesterday and four hours today. Same code, same cluster, roughly the same data volume. The Spark UI shows one stage holding everything: 197 of its 200 tasks finished in seconds, and three have been running for hours, spilling to disk the whole time. A colleague suggests doubling executor memory; another suggests doubling the cluster. Both suggestions cost real money, and neither names a mechanism. Before you spend anything: what, specifically, made three tasks different from the other 197?After this week you canThe full week 2 brief is part of LeetData Pro.Case studies

LinkedIn — Magnet: A scalable and performant shuffle architecture for Apache Spark | LinkedIn Engineering

Uber — RemoteShuffleService/docs/server-high-level-design.md at master

LinkedIn — Right-sizing Spark executor memory

State Farm — When the Spark Execution Plan Gets Too Big

UnSkew Data — Distributed Data Systems - Understanding Join Algorithms

Sem Sinchenko — Why Apache Spark is often considered as slow?

Exercises

Executor OOM on a Skewed Join

The Loop That Slowed Every Lap

Performance Forensics — Week 2 Check

GC Thrash

199 Tasks Done, 1 Task Forever

Salting a Hot-Key Aggregation

The Job That Killed Its Own Driver

The Accidental Cartesian