The Job That Killed Its Own DriverSpark · easy · ~10 minA nightly Spark job that builds a customer export has started failing near the end. Executors look healthy — plenty of memory headroom — but the driver crashes with an OutOfMemoryError right after the final stage completes. The job worked fine months ago when the customer table was smaller.Read the evidence and find why the driver, not the executors, is dying.Evidence

Metrics

Driver Logs

Job Snippet

Checkpoints

Why is the DRIVER running out of memory while executors are fine?

What is the correct fix?

Explain, interview-style, why 'just increase driver memory' is the wrong instinct here.