Spark · hard · ~15 min
Product wants the recommendation-features pipeline's freshness SLA tightened from 'by 08:00' to 'by 05:00' — a 3-hour improvement. The job currently runs 02:00-07:30 (5.5h) on 20 nodes ($0.55/node/hr). Infra's quick quote: 'roughly 3x the nodes to go ~3x faster: ~$29k/month'. Before anyone signs that, the profile shows: 40% of runtime is one skewed stage, and 25% is recomputing features for users with NO activity since yesterday.
Find the cheaper path to 05:00 — and know when to say the money is actually worth it.