The Loop That Slowed Every LapSpark · medium · ~12 minA feature-engineering job applies ~300 transformations to a DataFrame in a Python for loop — one enrichment per feature family, same data volume throughout.Each iteration is slower than the last. By iteration 200 the job spends more time *between* stages than in them, and the Spark UI gives up rendering the SQL DAG entirely. Data volume is flat.A teammate added cache() every 10 iterations — no improvement. Another found a benchmark showing localCheckpoint() is the fastest lineage fix and wants to ship it today. The job runs on spot executors with dynamic allocation.Evidence

Metrics

Logs

Job & cluster

Checkpoints

Why didn't `cache()` help, even though the data is genuinely reused every iteration?

The benchmark is right that localCheckpoint() is fastest. Why is it the wrong choice for THIS job?

Plan analysis grows with accumulated plan size. If analysis cost ~1.5s at iteration 10, and growth is roughly linear in iteration count, estimate the analysis cost in seconds at iteration 200.

Write the fix: what you change in the loop, why it survives spot reclaims, the housekeeping it adds, and how you'll verify the plan is actually truncated (one known fake fix looks truncated but isn't).