How Uber, PayPal, Agoda, and 30+ companies actually run Kafka — scale, failure, cost, and the judgment calls.
Past a few teams Kafka becomes an internal product; every big shop independently grows the same four organs — discovery, standard clients, self-serve, filtered alerting.
Tonight's maintenance window needs three brokers rebooted. You ask the obvious question: which clients will notice? The honest answer is that nobody knows — forty teams connect with bootstrap addresses copied from a wiki, a dozen homegrown consumer wrappers, and configs last reviewed at onboarding. The tutorial made Kafka look like a cluster you run. Why does it now feel like a product you're failing to support?
That gap is this week's subject. The case studies converge on something remarkable: every company that crossed roughly a trillion events a day independently grew the *same four organs* — and each one exists because a specific mechanical problem forced it.
Start with what every Kafka client physically holds: bootstrap broker addresses, plus a few dozen configs (batching, timeouts, retries, security) that each team copied from somewhere. That coupling is invisible with five services and crushing with eight hundred — replace a broker and some client somewhere holds its address; tune a default and you're filing forty pull requests. The support surface isn't the brokers. It's every copy of the client configuration in the company.
The structural fix is indirection: a config service — a stateless service that hands every client its bootstrap addresses and vetted configs at startup. Clients know a name, not an address; brokers become replaceable mid-day without a single client change. The second half is standard client libraries — resilience, metrics, security baked in — so a thousand engineers can't each invent their own consumer loop. Run the counterfactual at PayPal's scale to feel the leverage: 85+ clusters, 1,500+ brokers, 800+ applications at 99.99% availability with Black Friday peaking around 1.3 trillion messages a day. Without the indirection, one broker replacement is 800 application configs under review; with it, the number is zero.
🏭 In production at Agoda: the same idea taken one step further — producers don't talk to Kafka *at all*. Applications write serialized events to local disk, and a node-local *forwarder daemon* ships batched, compressed events to the clusters. That's how 1.8 trillion events a day get produced by teams who just "write to a log file," while the platform team owns connectivity, batching, and upgrades in one place. The price is ~10 seconds of p99 end-to-end latency — which 85% of producers happily accept; the latency-sensitive 15% connect directly.
⚠️ Gotcha: PayPal's scar from the years *before* the discipline: fast onboarding with no access records created who-uses-this-topic blindness that took a retroactive ACL rollout to fix. The discovery layer isn't only convenience — it's where you learn who your clients *are*.
🤔 Check yourself: Your company has 30 services connected with hand-copied bootstrap addresses. Walk the maintenance procedure for replacing broker 7 — then walk it again with a config service in place. What changed, mechanically?
Without: find every config holding broker 7's address (you can't, reliably), coordinate rolling restarts with each owning team, accept that stragglers will fail on reconnect. With: replace the broker; clients resolve fresh addresses from the config service at startup, and connected clients ride normal metadata refresh. The procedure collapses from a cross-team project to a platform-internal operation — that's what "brokers became replaceable" means.
Here's the constraint that shapes every large consumer fleet. A consumer group — Kafka's mechanism for sharing a topic's partitions across cooperating consumer instances — assigns each partition to at most one member. Consequence: a 40-partition topic supports at most 40 active consumers; number 41 sits idle. Your parallelism ceiling is a *broker-side* number chosen at provisioning time, and raising it later has producer-side costs (section 4). Worse, membership is dynamic: every deploy, autoscale event, or GC pause triggers a rebalance — the group pausing to reassign partitions (week 2 dissects this machinery). At Walmart's scale — 25,000+ consumers on Kubernetes — everyday pod churn kept rebalance cycles breaching near-real-time SLAs long before throughput was a problem.
Their answer inverts the model: a messaging proxy consumes each topic *once* and POSTs messages to stateless REST services. Consumer capacity now scales with pod replicas — a Kubernetes number you change in seconds — while partition count returns to being a broker-side sizing concern (each partition costs roughly 5–10 MB of broker memory, so right-sizing counts is real money too). The proxy absorbs the hard parts: per-key ordering preserved by keeping only one in-flight message per key, retries with backoff, poison pills diverted to a dead-letter queue.
⚠️ Gotcha: the proxy commits only the *contiguous prefix* of processed offsets — it cannot tell Kafka "I did 1–99 and 101" — so one stuck message holds back commit progress for everything behind it in that partition. This same contiguous-prefix mechanic is why poison-pill handling (dead-lettering after bounded retries) is a correctness feature, not an operational nicety. You'll meet it again in week 2's stalled-consumer scenarios.
🤔 Check yourself: A 200-partition topic needs 400 workers' worth of processing. Enumerate your options without a proxy, and what the proxy version looks like.
Without: you can't — 200 partitions caps the group at 200 consumers. Either repartition to 400+ (a produce-side disruption with batching consequences — section 4), or make each consumer internally multi-threaded (now *you* own per-key ordering and offset correctness). With a proxy: one consumer group reads 200 partitions; 400 stateless REST replicas process behind it; scaling is a replica count.
The mechanism here is queueing arithmetic, not technology. Ticket-driven provisioning costs platform-engineer hours *linearly in team count* — at a handful of teams it's fine; at fifty it's the platform team's whole calendar. DoorDash's numbers make the shape concrete: topic provisioning via tickets and Terraform took about 12 hours of elapsed time with infra engineers in the loop; their portal flow takes under 5 minutes, and 23,335 resources have been onboarded through it — including one engineer provisioning 20 production topics in half a day, previously a multi-team effort.
The design discipline is what makes self-serve safe rather than reckless: expose only essential capacity parameters, enforce best practices as *defaults* (validated at creation), auto-approve low-risk requests, and gate the rest. The platform team's job shifts from vending to building the vending machine — and the same guardrail thinking extends to testing: DoorDash rides test traffic on *production* topics using header-based tenancy (an OpenTelemetry-propagated tenant tag; sandbox consumer groups auto-suffixed so they can't steal production partitions), avoiding a parallel test-pipeline estate entirely.
⚖️ Tradeoff: abstraction cuts both ways. Hide too many knobs and power users leave the paved road; expose too many and defaults stop protecting anyone. The working split: users own capacity intent (throughput, retention, consumer count); the platform owns everything with a wrong answer (replication factor, security, naming, cleanup policies).
🤔 Check yourself: Design the self-serve topic form. Which three fields do users fill in, and name three settings you'd refuse to expose — with the failure each hidden setting prevents.
User-facing: expected throughput, retention need, consumer-group intent (the capacity contract). Platform-enforced: replication factor and min.insync.replicas (a wrong value silently trades away durability), ACL/naming conventions (tenancy and discoverability depend on them), and cleanup/compaction policy defaults (a wrong one deletes data or grows disks unbounded). The test is "does this field have a wrong answer a reasonable team could pick?" — if yes, it's a default, not a field.
The cheapest monitoring habit in the corpus: chart producer *outgoing* bytes against consumer *incoming* bytes for the same topic on one panel. They should track; a divergence is a misconfiguration made visible. Adobe's pipeline (~310 billion messages a day) caught a replication service showing 2.6–3× more ingress than egress — an upstream producer misconfiguration whose fix removed ~130 MB/s each way and let an edge cluster scale down 50%.
The deeper story is the partition-increase trap, and it's worth the mechanism. Producers don't send records one at a time; the record accumulator batches records *per partition*, flushing a batch when it fills or when linger.ms expires. Grow the partition count 2.5× on a hot topic and each record lands in one of 2.5× more per-partition buffers — every batch now fills 2.5× slower, so batches flush by timeout, *under-filled*. Smaller batches mean worse compression and more request overhead: ingress, egress, replication traffic, and storage all inflate — same event volume, bigger bill. Adobe's fix — a sticky partitioner that fills one batch at a time for keyless records — reclaimed roughly 78 TB of storage and hundreds of MB/s of traffic.
linger.ms
The client side has an equally quiet failure mode: a consumer group can stop consuming *with zero errors in logs* — stuck in a rebalance loop because poll gaps exceed timeouts. The metrics that expose it are client-side: coordinator join-rate and seconds-since-last-poll. If you alert on nothing else from consumers, alert on those.
⚠️ Gotcha: partition count is not a free parallelism dial. Producer batching (this section), rebalance blast radius (week 2), and per-partition broker memory all move with it. Treat a partition increase on a hot topic as a capacity change with a rollback plan, not a config tweak.
🤔 Check yourself: You increase a hot keyless topic from 100 to 250 partitions. Predict the producer-side effect chain, and name the panel that catches it.
Records spread across 2.5× more accumulator buffers → batches fill 2.5× slower → linger.ms expires first → smaller, worse-compressed batches → producer request rate and byte rates rise while event volume is flat — visible as producer-out bytes climbing on the paired panel with no matching event-count change, then echoed in replication and storage. Mitigation: sticky partitioning (or restoring batch economics with linger.ms/batch.size), and questioning whether the increase was needed at all.
batch.size
| Concept | Mechanism in one line | Number to remember | Production proof |
|---|---|---|---|
| Config indirection | Clients resolve addresses/configs at startup; brokers become replaceable | 800+ apps, zero client changes per swap | PayPal config service at 1.3T msgs/day peak |
| Partition ceiling | One partition ↔ at most one group member; churn triggers rebalances | 5–10 MB broker memory per partition | Walmart proxy: 25K consumers scale by replicas |
| Self-serve | Guardrails as validated defaults; platform builds the vending machine | ~12h → <5 min provisioning | DoorDash: 23,335 resources onboarded |
| Paired metrics | Producer-out vs consumer-in on one panel; divergence = misconfig | ~78 TB reclaimed from one partitioner fix | Adobe's three monitoring-driven wins |
Each objective maps to an interview move:
