Shuffle services, fleet-scale upgrades, cost engineering, and performance forensics — from Uber's 2M jobs to Wix's −60% bill.
Spark tuning folklore is endless; this cohort sticks to what companies actually did. Uber upgraded 2M+ jobs fleet-wide with shadow execution, Wix cut costs 60% moving 5,000 workflows to EMR-on-EKS, LinkedIn rebuilt shuffle around sequential reads, PayPal cut a flagship job's cost 70%, and Amazon rebuilt its exabyte compactor on Ray — each taught as a mechanism first, with the company as proof.
Past a handful of teams every serious Spark shop independently grows the same four organs — a submission front door, an application-aware scheduler, remote shuffle, and shadow-run upgrade machinery.
You inherit "the Spark platform." It is, on inspection, eleven clusters: each team launched its own, sized for its worst day, running whichever Spark version was current when the team formed. A new cluster takes a quarter of an hour to start, so nobody ever shuts one down. One team is pinned three major versions back by a library nobody dares touch. Finance reports half the fleet idle; the teams report they can't get capacity. Every company that runs Spark seriously hit this exact wall — and, independently, they all built their way out with the same four organs. What are they?
The first organ looks bureaucratic and is actually load-bearing: a submission service — one internal API through which every Spark job enters the platform, owning authentication, cluster selection, queue placement, and job state. Users never talk to a cluster; they talk to the platform.
The mechanism matters because of what it decouples. When a job arrives as a spec (code location, resources, schedule) rather than a spark-submit against a named cluster, the platform can *re-route* it — to a different cluster, a different Spark version, a different runtime entirely — without the owner changing anything. Wix's PlatySpark, Pinterest's Archer, and Uber's Drogon are all this pattern: Airflow operators hand a spec to the service; the service turns it into whatever the current infrastructure wants (at Pinterest, a Kubernetes SparkApplication resource picked up by an operator; the service layer holds auth and job state, the worker layer talks to the cluster API).
spark-submit
SparkApplication
The front door is also where hidden bottlenecks concentrate, which is the cautionary half of the mechanism. Wix initially kept Apache Livy as the submission path onto EMR-on-EKS and found Livy itself was the choke point — it spawns a Java process per driver launch and doesn't scale horizontally, forcing one peak-sized instance to run all day. They removed it and called the cluster API directly.
🏭 In production at Wix: the front door is what made their entire EMR → EKS migration invisible. PlatySpark redirected jobs queue by queue behind the API; 99% of 3,500+ daily shared-cluster applications moved with zero user changes.
🤔 Check yourself: Your company submits Spark jobs from forty repos, each with its own spark-submit wrapper and hardcoded cluster endpoint. Leadership wants the fleet on a cheaper runtime within a year. What does the *first* engineering quarter buy you if you spend it on a submission service instead of the migration itself?
Re-routability. With every job entering through one API, the migration becomes a platform-side routing change executed gradually and reversibly — per queue, per team, with rollback. Without it, the migration is forty coordinated team projects, each with its own timeline and its own way to fail.
Kubernetes won the infrastructure war, but its default scheduler places *pods*, one at a time, with no concept of a user, a queue, or an application. Batch platforms need exactly those concepts, and this gap is why every serious Spark-on-K8s shop runs a second scheduler.
The mechanics you need: hierarchical queues — capacity carved along org/team/project lines so tenants get guarantees and bounded burst; gang scheduling — an application starts only when its whole resource gang (driver plus executors) can be placed, because a driver that starts alone burns its queue slot doing nothing; preemption — lower-tier pods get evicted when higher-tier SLOs are at risk. YuniKorn supplies these on Kubernetes. Below it, a *node* autoscaler (Karpenter, or the Cluster Autoscaler) provisions machines. That's the two-layer design: YuniKorn decides *which pods, in what order, into which queue capacity*; the node layer decides *what machines exist*. Neither layer alone suffices — Karpenter has no queues; YuniKorn doesn't make nodes.
Why this beats YARN on cost is a capacity argument you can run yourself. A YARN cluster is sized ahead of demand and pays for its valleys; if launching capacity takes 10–15 minutes, you keep headroom running all day. Cut node provisioning to 2–3 minutes with pods starting in seconds — Wix's measured numbers on EKS — and capacity can *follow* demand: the platform binpacks jobs onto fewer nodes and returns the rest. That, not any per-node discount, is where Wix's result came from: a 20% cost-reduction goal, a 60% actual reduction on the shared cluster, 35–50% on dedicated ones.
Pinterest closed the remaining manual loop: per-application usage summaries (a capability they contributed upstream) land in S3, feed insight tables, and a constraint solver regenerates queue allocations from usage-versus-capacity gaps — replacing percentile-threshold tuning that took several manual iterations. Allocations deploy as Git-reviewed config. Capacity management became a pipeline.
⚠️ Gotcha: Kubernetes at Spark scale has failure modes nobody meets in microservice land. Pinterest hit PodGC deleting *driver pods* once the cluster crossed the garbage-collection threshold — jobs misreported FAILED because the pod vanished before final status was read (fixed with finalizers). Admission-webhook latency degraded the control plane as pod churn grew (fixed by moving customization to pod templates). Spark's pod bursts exhausted IP space, forcing network redesign. And Joom found the Cluster Autoscaler freezes downscaling across *all* node groups after one group's launch failure — their fix: one autoscaler instance per group.
⚖️ Tradeoff: shuffle needs fast local disk, and on Kubernetes that's a real decision. Wix's EBS-backed executors regressed shuffle-heavy jobs — network-attached storage added a hop vanilla EMR didn't pay — and only local-NVMe instance families restored parity, at the price of constraining instance choice.
🤔 Check yourself: A job needs 1 driver + 200 executors. The cluster has room for 120 executors right now. Predict what happens with the default Kubernetes scheduler, and what gang scheduling changes.
Default: the driver and ~120 executors start; the job runs badly underprovisioned (or deadlocks waiting), occupying capacity while delivering a fraction of its throughput — and two such jobs can each hold half of what the other needs. Gang scheduling: the application waits, whole, until 201 pods fit — capacity is either fully productive or genuinely free.
The third organ gets its full mechanics next week, but it belongs in the platform map because the *reason* for it is platform-level. Executor-local shuffle couples three things that want to be independent: a node's disk, a job's intermediate data, and every co-tenant's stability. One shuffle-heavy job fills a shared disk and unrelated jobs crash; a node can't be reclaimed while any downstream stage might fetch from it — which quietly disables the autoscaling you just built; and at fleet scale the write volume physically consumes hardware (Uber measured shuffle killing SSDs rated for three years in about six months).
A remote shuffle service — executors write shuffle to a dedicated fleet instead of local disk — cuts all three couplings at once: noisy neighbors lose their weapon, executors become stateless enough for aggressive scale-down, and disks are procured for the job they're actually doing. Pinterest's Moka runs Celeborn in exactly this role; average job performance improved a modest 5%, but the prize was the decoupling — dynamic allocation works, shuffle timeouts faded, and packing tightened.
💡 Note: hold the "how" — push-merge, partition ownership, dedup on retries — for Week 2, where Magnet and Uber's RSS get the mechanism treatment. This week's takeaway is *why platforms adopt one*: shuffle locality is a tax on elasticity.
The fourth organ exists because engine upgrades at fleet scale broke every naive strategy. "Ask teams to test on the new version" fails for a reason worth internalizing: engine versions change semantics silently. Uber's canonical example — array_contains(array(1), 1.34D) returns TRUE on Spark 2.4 and FALSE on 3.3, because type coercion changed. No test suite written against the old behavior catches what its authors didn't know could change; production outputs simply differ, quietly.
array_contains(array(1), 1.34D)
So the shops that succeeded all converged on the same mechanism: run the real jobs on the new engine against non-production targets, and diff the outputs. Slack's version — dual-stack: one upgraded Hive Metastore (3.1.0) serving old and new clusters simultaneously so metadata never forked; a boolean flag in their Airflow operators routing each DAG to EMR 5 or EMR 6; validation by exact EXCEPT/COUNT diffs, executed on Trino for speed, with timestamp columns excluded and deterministic ordering imposed. Uber's version — at 2M+ daily applications there was no staging environment, so they built one *at runtime*: Iron Dome intercepts the catalog and the output committer so a production job's writes transparently rewrite from /db/tbl to /stgdb/tbl, guardrail interceptors block any accidental production write, telemetry records every touched table, and comparison jobs diff shadow outputs against production. For the 2,100+ applications needing source changes, an AST-rewriting tool (Piranha) matched Spark-2-only patterns and patched them — codemods, not tickets.
EXCEPT
COUNT
/db/tbl
/stgdb/tbl
The numbers say the machinery pays: Uber moved 85% of 20,000 workflows in six months, with roughly 50% overall runtime/resource reduction and millions saved; Slack landed Spark 3 across 60+ clusters and thousands of DAGs with typical 30–60% task speedups and zero incidents.
⚠️ Gotcha: the flip side of upgrade machinery is the *legacy flag ledger*. Slack shipped with storeAssignmentPolicy=Legacy to keep pre-ANSI casting; Uber's codemods injected flags like allowUntypedScalaUDF. Each flag preserves old semantics cheaply — and accumulates as debt someone must retire, because flags mask exactly the semantic drift the diffing was built to catch.
storeAssignmentPolicy=Legacy
allowUntypedScalaUDF
🤔 Check yourself: Design the validation leg for upgrading 3,000 nightly jobs from Spark 3.3 to 4.0 with no staging environment. Name the three components you'd steal from this section and the order you'd deploy them.
(1) Path-rewriting interception first — run real jobs on 4.0 with writes redirected to staging namespaces, plus a write guardrail; (2) output diffing next — exact EXCEPT/COUNT comparisons against production outputs, with nondeterminism (timestamps, row order) explicitly handled; (3) flag-based routing last — flip jobs to 4.0 cohort by cohort as they certify, keeping instant rollback. Codemods enter only for the subset the diffs prove broken.
| Concept | One-line mechanism | Number to remember | Production proof |
|---|---|---|---|
| Front door | jobs are specs into one API; platform owns routing, auth, state | 99% of 3,500 daily jobs migrated invisibly | Wix PlatySpark; Pinterest Archer |
| Two-layer scheduling | app-aware queues/gangs/preemption above, node provisioning below | 10–15 min clusters → 2–3 min nodes | Wix −60% vs 20% goal; Pinterest CP-SAT loop |
| Remote shuffle | intermediate data leaves the node; executors become stateless | SSDs: ~3-year rating, ~6-month life | Uber wear data; Moka runs Celeborn |
| Shadow upgrades | real jobs, rewritten output paths, exact diffs | 2M+ jobs; 85% in 6 months; 0 incidents at Slack | Uber Iron Dome; Slack dual-stack |
Platform-anatomy questions test whether you've operated Spark beyond one job. Expect: *"Your company has eleven team-owned clusters — what do you build first and why?"* (the front door; argue from re-routability and the migrations it later enables). *"Why do Spark-on-K8s shops run YuniKorn when Kubernetes already has a scheduler?"* (name the missing concepts — queues, gangs, preemption — and walk the 200-executor gang example). *"Where does the cost saving actually come from in a YARN → K8s migration?"* (provisioning speed → capacity follows demand → binpacking; not a per-node discount — and Pinterest's finding that consolidation and packing beat the hardware swap). *"How do you upgrade 20,000 jobs when outputs can silently change?"* (shadow execution with path rewriting, exact diffing with nondeterminism handled, flag-routed gradual cutover). The senior tell in every one of these is mechanism-plus-number: candidates who can say *why* the organ exists and quote what it measurably bought.
Open week 1 →
Symptom → subsystem: the UDF codegen cliff, join algorithms diagnosed from where they fail, memory forensics at both ends, and why shuffle physics eats 10–20% of clusters.
The job ran in forty minutes yesterday and four hours today. Same code, same cluster, roughly the same data volume. The Spark UI shows one stage holding everything: 197 of its 200 tasks finished in seconds, and three have been running for hours, spilling to disk the whole time. A colleague suggests doubling executor memory; another suggests doubling the cluster. Both suggestions cost real money, and neither names a mechanism. Before you spend anything: what, specifically, made three tasks different from the other 197?
The full week 2 brief is part of LeetData Pro.
Open week 2 →
Three attacks on the bill — fewer fatter tasks, cheaper cores that survive reclaims, vectorized cores behind a fallback boundary — each with the arithmetic that predicts its payoff.
The job is healthy. It finishes inside its window every night, nothing spills, nothing retries. Then finance tags you: this one pipeline is a third of the platform's compute bill. You open the Spark UI looking for a villain and find none — just an input stage that ran two hundred thousand tasks, each completing in under two seconds, on a cluster of a hundred and forty machines that all went home on time. Nothing is broken. It just *costs*. Where, exactly, is the money going?
The full week 3 brief is part of LeetData Pro.
Open week 3 →
Working-set arithmetic decides single-node vs cluster; copy-by-reference compaction and driver-serial fan-out mark where specialists beat generality; table formats hand you a maintenance loop.
Your nightly job aggregates eight gigabytes of events. It runs on a six-worker cluster that takes four minutes to start, executes for eleven, and shuts down. The pipeline has run this way for two years, nobody complains, and the code is clean. Then a new teammate asks the question you realize you've never actually answered: *why is there a cluster here at all?* You start to say "because it's Spark," and stop. What would it take to answer with arithmetic instead of habit?
The full week 4 brief is part of LeetData Pro.
Open week 4 →
