The Cluster That Watched One Core WorkSpark · medium · ~13 minThe weekly budget-allocation pipeline "runs on Spark": feature ETL over 800 GB, then an optimization step that solves a small constrained-allocation problem per city — 4,200 cities — then aggregation and writeback.Total runtime has crept from 2.5 to 4 hours and the optimization step is the growing part. The team's proposals: double the cluster (40 → 80 executors), or move to a bigger driver instance "since the driver seems busy."You pull the utilization data mid-run before approving either spend.Evidence

Metrics

Driver log

Job structure

Checkpoints

Executors sit at ~5% CPU for nearly two hours while one driver core is pegged. What is actually happening?

Sanity-check the bottleneck's size: at ~1.5 s per city × 4,200 cities run sequentially, roughly how many minutes does the driver-serial optimization step take?

Which redesign fixes the bottleneck while preserving the working Spark investment?

Write the design note for the hybrid: the data-movement contract between the two engines, what stays serial and why that bounds the speedup, and the operational cost you're signing up for.