How Uber, PayPal, Agoda, and 30+ companies actually run Kafka — scale, failure, cost, and the judgment calls.
Past a few teams Kafka becomes an internal product; every big shop independently grows the same four organs — discovery, standard clients, self-serve, filtered alerting.
Tonight's maintenance window needs three brokers rebooted. You ask the obvious question: which clients will notice? The honest answer is that nobody knows — forty teams connect with bootstrap addresses copied from a wiki, a dozen homegrown consumer wrappers, and configs last reviewed at onboarding. The tutorial made Kafka look like a cluster you run. Why does it now feel like a product you're failing to support?
That gap is this week's subject. The case studies converge on something remarkable: every company that crossed roughly a trillion events a day independently grew the *same four organs* — and each one exists because a specific mechanical problem forced it.
Start with what every Kafka client physically holds: bootstrap broker addresses, plus a few dozen configs (batching, timeouts, retries, security) that each team copied from somewhere. That coupling is invisible with five services and crushing with eight hundred — replace a broker and some client somewhere holds its address; tune a default and you're filing forty pull requests. The support surface isn't the brokers. It's every copy of the client configuration in the company.
The structural fix is indirection: a config service — a stateless service that hands every client its bootstrap addresses and vetted configs at startup. Clients know a name, not an address; brokers become replaceable mid-day without a single client change. The second half is standard client libraries — resilience, metrics, security baked in — so a thousand engineers can't each invent their own consumer loop. Run the counterfactual at PayPal's scale to feel the leverage: 85+ clusters, 1,500+ brokers, 800+ applications at 99.99% availability with Black Friday peaking around 1.3 trillion messages a day. Without the indirection, one broker replacement is 800 application configs under review; with it, the number is zero.
🏭 In production at Agoda: the same idea taken one step further — producers don't talk to Kafka *at all*. Applications write serialized events to local disk, and a node-local *forwarder daemon* ships batched, compressed events to the clusters. That's how 1.8 trillion events a day get produced by teams who just "write to a log file," while the platform team owns connectivity, batching, and upgrades in one place. The price is ~10 seconds of p99 end-to-end latency — which 85% of producers happily accept; the latency-sensitive 15% connect directly.
⚠️ Gotcha: PayPal's scar from the years *before* the discipline: fast onboarding with no access records created who-uses-this-topic blindness that took a retroactive ACL rollout to fix. The discovery layer isn't only convenience — it's where you learn who your clients *are*.
🤔 Check yourself: Your company has 30 services connected with hand-copied bootstrap addresses. Walk the maintenance procedure for replacing broker 7 — then walk it again with a config service in place. What changed, mechanically?
Without: find every config holding broker 7's address (you can't, reliably), coordinate rolling restarts with each owning team, accept that stragglers will fail on reconnect. With: replace the broker; clients resolve fresh addresses from the config service at startup, and connected clients ride normal metadata refresh. The procedure collapses from a cross-team project to a platform-internal operation — that's what "brokers became replaceable" means.
Here's the constraint that shapes every large consumer fleet. A consumer group — Kafka's mechanism for sharing a topic's partitions across cooperating consumer instances — assigns each partition to at most one member. Consequence: a 40-partition topic supports at most 40 active consumers; number 41 sits idle. Your parallelism ceiling is a *broker-side* number chosen at provisioning time, and raising it later has producer-side costs (section 4). Worse, membership is dynamic: every deploy, autoscale event, or GC pause triggers a rebalance — the group pausing to reassign partitions (week 2 dissects this machinery). At Walmart's scale — 25,000+ consumers on Kubernetes — everyday pod churn kept rebalance cycles breaching near-real-time SLAs long before throughput was a problem.
Their answer inverts the model: a messaging proxy consumes each topic *once* and POSTs messages to stateless REST services. Consumer capacity now scales with pod replicas — a Kubernetes number you change in seconds — while partition count returns to being a broker-side sizing concern (each partition costs roughly 5–10 MB of broker memory, so right-sizing counts is real money too). The proxy absorbs the hard parts: per-key ordering preserved by keeping only one in-flight message per key, retries with backoff, poison pills diverted to a dead-letter queue.
⚠️ Gotcha: the proxy commits only the *contiguous prefix* of processed offsets — it cannot tell Kafka "I did 1–99 and 101" — so one stuck message holds back commit progress for everything behind it in that partition. This same contiguous-prefix mechanic is why poison-pill handling (dead-lettering after bounded retries) is a correctness feature, not an operational nicety. You'll meet it again in week 2's stalled-consumer scenarios.
🤔 Check yourself: A 200-partition topic needs 400 workers' worth of processing. Enumerate your options without a proxy, and what the proxy version looks like.
Without: you can't — 200 partitions caps the group at 200 consumers. Either repartition to 400+ (a produce-side disruption with batching consequences — section 4), or make each consumer internally multi-threaded (now *you* own per-key ordering and offset correctness). With a proxy: one consumer group reads 200 partitions; 400 stateless REST replicas process behind it; scaling is a replica count.
The mechanism here is queueing arithmetic, not technology. Ticket-driven provisioning costs platform-engineer hours *linearly in team count* — at a handful of teams it's fine; at fifty it's the platform team's whole calendar. DoorDash's numbers make the shape concrete: topic provisioning via tickets and Terraform took about 12 hours of elapsed time with infra engineers in the loop; their portal flow takes under 5 minutes, and 23,335 resources have been onboarded through it — including one engineer provisioning 20 production topics in half a day, previously a multi-team effort.
The design discipline is what makes self-serve safe rather than reckless: expose only essential capacity parameters, enforce best practices as *defaults* (validated at creation), auto-approve low-risk requests, and gate the rest. The platform team's job shifts from vending to building the vending machine — and the same guardrail thinking extends to testing: DoorDash rides test traffic on *production* topics using header-based tenancy (an OpenTelemetry-propagated tenant tag; sandbox consumer groups auto-suffixed so they can't steal production partitions), avoiding a parallel test-pipeline estate entirely.
⚖️ Tradeoff: abstraction cuts both ways. Hide too many knobs and power users leave the paved road; expose too many and defaults stop protecting anyone. The working split: users own capacity intent (throughput, retention, consumer count); the platform owns everything with a wrong answer (replication factor, security, naming, cleanup policies).
🤔 Check yourself: Design the self-serve topic form. Which three fields do users fill in, and name three settings you'd refuse to expose — with the failure each hidden setting prevents.
User-facing: expected throughput, retention need, consumer-group intent (the capacity contract). Platform-enforced: replication factor and min.insync.replicas (a wrong value silently trades away durability), ACL/naming conventions (tenancy and discoverability depend on them), and cleanup/compaction policy defaults (a wrong one deletes data or grows disks unbounded). The test is "does this field have a wrong answer a reasonable team could pick?" — if yes, it's a default, not a field.
The cheapest monitoring habit in the corpus: chart producer *outgoing* bytes against consumer *incoming* bytes for the same topic on one panel. They should track; a divergence is a misconfiguration made visible. Adobe's pipeline (~310 billion messages a day) caught a replication service showing 2.6–3× more ingress than egress — an upstream producer misconfiguration whose fix removed ~130 MB/s each way and let an edge cluster scale down 50%.
The deeper story is the partition-increase trap, and it's worth the mechanism. Producers don't send records one at a time; the record accumulator batches records *per partition*, flushing a batch when it fills or when linger.ms expires. Grow the partition count 2.5× on a hot topic and each record lands in one of 2.5× more per-partition buffers — every batch now fills 2.5× slower, so batches flush by timeout, *under-filled*. Smaller batches mean worse compression and more request overhead: ingress, egress, replication traffic, and storage all inflate — same event volume, bigger bill. Adobe's fix — a sticky partitioner that fills one batch at a time for keyless records — reclaimed roughly 78 TB of storage and hundreds of MB/s of traffic.
linger.ms
The client side has an equally quiet failure mode: a consumer group can stop consuming *with zero errors in logs* — stuck in a rebalance loop because poll gaps exceed timeouts. The metrics that expose it are client-side: coordinator join-rate and seconds-since-last-poll. If you alert on nothing else from consumers, alert on those.
⚠️ Gotcha: partition count is not a free parallelism dial. Producer batching (this section), rebalance blast radius (week 2), and per-partition broker memory all move with it. Treat a partition increase on a hot topic as a capacity change with a rollback plan, not a config tweak.
🤔 Check yourself: You increase a hot keyless topic from 100 to 250 partitions. Predict the producer-side effect chain, and name the panel that catches it.
Records spread across 2.5× more accumulator buffers → batches fill 2.5× slower → linger.ms expires first → smaller, worse-compressed batches → producer request rate and byte rates rise while event volume is flat — visible as producer-out bytes climbing on the paired panel with no matching event-count change, then echoed in replication and storage. Mitigation: sticky partitioning (or restoring batch economics with linger.ms/batch.size), and questioning whether the increase was needed at all.
batch.size
| Concept | Mechanism in one line | Number to remember | Production proof |
|---|---|---|---|
| Config indirection | Clients resolve addresses/configs at startup; brokers become replaceable | 800+ apps, zero client changes per swap | PayPal config service at 1.3T msgs/day peak |
| Partition ceiling | One partition ↔ at most one group member; churn triggers rebalances | 5–10 MB broker memory per partition | Walmart proxy: 25K consumers scale by replicas |
| Self-serve | Guardrails as validated defaults; platform builds the vending machine | ~12h → <5 min provisioning | DoorDash: 23,335 resources onboarded |
| Paired metrics | Producer-out vs consumer-in on one panel; divergence = misconfig | ~78 TB reclaimed from one partitioner fix | Adobe's three monitoring-driven wins |
Each objective maps to an interview move:
Open week 1 →
The canonical outage chain — slow sink → GC pause → missed heartbeat → rebalance storm — and why failback, not failover, is the real DR interview question.
Friday, 4 p.m., a routine deploy. Twenty minutes later the lag graph goes vertical. Kubernetes says every consumer pod is healthy. The application logs are clean — no errors, no exceptions, nothing. But consumption is zero, and it stays zero while lag climbs into the millions. Nothing crashed. So why is nothing moving?
This week is the consumer group under fire: the failure chain that runs through *your* code, *your* JVM, and *your* Kubernetes settings before it ever touches a broker — and the disaster-recovery question (failback, not failover) that most designs skip.
The full week 2 brief is part of LeetData Pro.
Open week 2 →
Cloud Kafka's dominant cost is network, not storage — follower fetching, tiered storage, and the diskless KIPs are one continuous attack on cross-AZ traffic.
Finance flags your streaming platform's spend, so you audit it expecting waste: oversized brokers, forgotten topics, bloated retention. You find none. Brokers are right-sized, storage is modest — and the biggest line on the bill is "inter-AZ data transfer," roughly half the total. Nothing is misconfigured. Every one of those bytes is doing exactly what Kafka's design says it should. So where, mechanically, is the money going?
This week is the Kafka bill: deriving where the bytes cross zone boundaries, and reading a decade of architecture work — follower fetching, tiered storage, diskless topics — as one continuous attack on that line item.
The full week 3 brief is part of LeetData Pro.
Open week 3 →
Exactly-once is a layered defense, contracts beat repair on immutable topics, and the strongest senior signal is knowing Kafka's boundaries.
The PM asks a simple question: "Can you guarantee every order event is counted exactly once?" You check the boxes — acks=all, idempotent producer on, transactions enabled, sinks reading read_committed — and say yes. Two weeks later finance finds duplicated orders in the warehouse. Every config was right. Every component honored its contract. So what, exactly, did those configs promise — and where did the promise end?
acks=all
read_committed
The final week is about boundaries: where exactly-once guarantees actually stop, how events get into Kafka atomically, why immutable topics make prevention beat repair — and where Kafka itself ends.
The full week 4 brief is part of LeetData Pro.
Open week 4 →
