Kafka · hard · ~15 min
Black Friday, 20:00, peak traffic. A rack-level power event took down brokers kafka-2 and kafka-3 at nearly the same time. For 4 of the checkout-events partitions, both in-sync followers were on those brokers — only the leader on kafka-1 survived, but for 2 partitions even the leader is gone and no in-sync replica remains.
kafka-2
kafka-3
checkout-events
kafka-1
Producers using acks=all are now failing with NOT_ENOUGH_REPLICAS for the affected partitions, and 2 partitions are fully offline. Checkout is degraded. The SRE bridge is asking you for a call. Hardware recovery ETA is 25–40 minutes.
acks=all
NOT_ENOUGH_REPLICAS
There is no clean answer here — decide, and be ready to defend the tradeoff.