Two Hundred Terabytes of WARNSpark · hard · ~12 minYou own logging for a Spark platform that runs 250K+ jobs a day across hundreds of thousands of executors, emitting up to 200TB of unstructured logs daily (~5.38PB per 30 days).Last quarter's cost fix: retention cut to 3 days and verbosity dropped from INFO to WARN. Since then, three incident retros have stalled on "the logs we needed were already gone." The HDFS line still runs ~$180K/yr at 3-day retention — a month would be ~$1.8M/yr — and the small-write pattern is chewing through SSDs.Two proposals are on the table:

"Turn gzip up to 9 and buy the month of retention."

"Parse the logs into a structured form first, then compress — a compression ratio in the hundreds is claimed."

Finance wants a decision; on-call wants INFO back.Evidence

The logging bill

What the raw stream looks like

Current appender + constraints

Checkpoints

The benchmarks show gzip and zstd plateauing around 10-12x on the raw stream while parse-then-compress claims 169x. Why does parsing first unlock an order of magnitude?

At the claimed 169x, roughly how many TB does a 30-day window (5.38PB) of logs compress down to?

Where should phase-1 compression run?

Write the adoption plan: the correctness risk that must be engineered around (not accepted), what breaks for log consumers and how you handle it, and what you spend the savings on.