The Broker That Filled Its DiskKafka · easy · ~10 minBroker kafka-2 just went down hard. Its disk hit 100% and the process exited. The clickstream-raw topic's traffic doubled two weeks ago, and someone also set its retention to 30 days 'to be safe'. Producers are still running against the surviving brokers.Get kafka-2 back safely and make sure this never pages you again.Evidence

Metrics

Broker Logs

Config

Checkpoints

Why did the broker actually die?

What's the safe way to bring kafka-2 back?

What prevention do you put in place, and how would you narrate this incident in an interview?