Your service is running fine at 10 RPS. At 1000 RPS it falls over, and nobody told the upstream to slow down. You add more replicas, but the database is now the bottleneck. You add a queue, but the consumer can’t drain it fast enough and memory explodes. Sound familiar?
This is the fast-producer, slow-consumer problem — and it’s one of the most common ways distributed systems die in production. The fix isn’t "just scale up". The fix is backpressure: a mechanism for the slow part of your system to signal upstream that it needs breathing room.
Backpressure isn’t new. Hydraulic engineers have used it since the 19th century — it’s literally what happens when a pipe tries to push more fluid than a valve can pass. In software, the valve is your consumer. The pipe is everything upstream. If you don’t design the valve to communicate its limits back to the source, you get burst pipes.
Let’s break down how three of the most common infrastructure pieces — Kafka, gRPC, and HTTP/2 — handle this problem, where they shine, and where they’ll betray you at the worst possible moment.
The Taxonomy of Saturation
Before diving into specifics, it’s worth naming the failure modes so you can recognize them:
Queue overflow: unbounded or too-small buffers fill up and either block producers, drop messages, or crash the process. Classic.
Consumer lag accumulation: the consumer is alive but perpetually behind. Messages are processed eventually, but "eventually" means hours or days late — which for most systems means "never usefully".
Cascade: the slow consumer holds a connection, the upstream times out and retries, now you have double the load, rinse, repeat until everything is on fire.
OOM kill: you buffered "just in case" in memory instead of applying backpressure at the protocol level. Linux disagrees with your optimism.
Kafka: Backpressure Through Pull and Pause
Kafka’s architecture is fundamentally pull-based. Consumers decide when to fetch and how much. The broker never pushes data at you — you ask for it. This is a deliberate design choice that gives consumers natural flow control by default.
But "natural" doesn’t mean "automatic". You still need to tune it correctly.
The Core Controls
# Consumer config
fetch.max.bytes=52428800 # Max total bytes per fetch request (50MB)
max.partition.fetch.bytes=1048576 # Max bytes per partition per fetch (1MB)
max.poll.records=500 # Max records returned by a single poll()
fetch.max.wait.ms=500 # How long broker waits if there's not enough data
max.poll.records is your primary throttle. Reduce it and your consumer processes smaller batches, which means each processing cycle is cheaper but you call poll() more often. This directly controls how much work you bite off at once.
The relationship between max.poll.records and max.poll.interval.ms is where most people get burned:
max.poll.interval.ms=300000 # 5 minutes default — dangerously long
If your consumer takes longer than this between two poll() calls, Kafka considers it dead and triggers a rebalance. Rebalances are expensive. One sluggish consumer processing a large batch can stall the entire consumer group.
The pause() / resume() Pattern
When your downstream (a database write, an HTTP call, a file write) is saturated, you need to tell Kafka to stop handing you records — without leaving the consumer group. This is what pause() is for:
// Java example — same concept applies to librdkafka, confluent-kafka-python, etc.
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (TopicPartition partition : records.partitions()) {
List<ConsumerRecord<String, String>> partitionRecords = records.records(partition);
for (ConsumerRecord<String, String> record : partitionRecords) {
boolean accepted = downstreamProcessor.tryProcess(record);
if (!accepted) {
// Downstream is saturated — pause this partition
consumer.pause(Collections.singleton(partition));
// Re-queue the failed record for retry
pendingRecords.put(partition, record);
}
}
}
// Attempt to drain pending records and resume paused partitions
for (Map.Entry<TopicPartition, ConsumerRecord<String, String>> entry : pendingRecords.entrySet()) {
if (downstreamProcessor.tryProcess(entry.getValue())) {
consumer.resume(Collections.singleton(entry.getKey()));
pendingRecords.remove(entry.getKey());
}
}
}
When a partition is paused, poll() still runs (keeping the consumer group heartbeat alive) but returns zero records for that partition. You’re not throttling the broker — you’re throttling yourself, which is exactly right.
Gotchas
Zombie consumers: a consumer that’s paused everything and never resumes is still in the group. It holds partition assignments, preventing other consumers from taking over. Always set a maximum pause duration and dead-man-switch logic.
Offset commit timing: if you commit offsets before processing completes and then crash, you lose records. If you never commit because processing is always "pending", you’ll reprocess everything on restart. Commit offsets only after successful downstream write.
Lag is a lagging indicator: monitoring consumer group lag tells you you’re behind. It doesn’t tell you why. A consumer that’s paused correctly and recovering looks identical in lag metrics to a consumer that’s stuck in an infinite retry loop. Track pause state explicitly.
Consumer count isn’t always the answer: more consumers mean more parallelism, but only up to the number of partitions. Ten consumers on a three-partition topic means seven of them do nothing. Partition count is your scaling ceiling.
Production Setup: Lag Monitoring
# prometheus-kafka-consumer-group-exporter or use kafka_exporter
- alert: KafkaConsumerGroupLagHigh
expr: kafka_consumer_group_lag > 100000
for: 10m
labels:
severity: warning
annotations:
summary: "Consumer group {{ $labels.consumergroup }} is lagging on {{ $labels.topic }}"
Set your lag alerting threshold based on how much latency your SLA tolerates, not a round number.
HTTP/2: Backpressure in the Transport Layer
HTTP/2 introduced binary framing and multiplexed streams over a single TCP connection. Along with that came explicit, mandatory flow control — unlike HTTP/1.1 where you got TCP’s window and nothing else.
How the Windows Work
Every HTTP/2 connection has two levels of flow control:
- Connection-level: a shared window across all streams on that connection
- Stream-level: individual window per request/response stream
Both start at the initial window size (default: 65535 bytes — 64KB). A sender cannot transmit DATA frames beyond what the receiver’s window allows. To receive more data, the receiver sends a WINDOW_UPDATE frame.
Sender Receiver
| |
|--HEADERS (stream 1)------------>|
|--DATA [30KB] (stream 1)-------->|
|--DATA [30KB] (stream 1)-------->|
| (window exhausted: 64KB used) |
| [processing...] |
|<--WINDOW_UPDATE [65535]---------|
|--DATA [remaining]-------------->|
If you’re doing high-throughput transfers — think log shipping, file uploads, streaming ML inference — the 64KB default is a joke. You need to tune it:
// Go net/http example
server := &http.Server{
Addr: ":443",
TLSConfig: tlsCfg,
}
// Using golang.org/x/net/http2 for direct tuning
http2Server := &http2.Server{
MaxConcurrentStreams: 250,
InitialWindowSize: 1 << 20, // 1MB per stream
MaxUploadBufferPerConnection: 1 << 23, // 8MB connection window
MaxUploadBufferPerStream: 1 << 20, // 1MB stream window
}
http2.ConfigureServer(server, http2Server)
On the client side with curl for testing:
# Check negotiated HTTP version and flow control behavior
curl -v --http2 https://yourservice.example/stream 2>&1 | grep -E "HTTP|WINDOW"
What HTTP/2 Flow Control Doesn’t Do
HTTP/2 flow control is byte-level, not semantic. It prevents the transport from being overwhelmed, but it doesn’t know anything about your application’s processing state.
You can’t say "I’m busy with other requests, stop sending me new ones" at the HTTP/2 layer. For that, you use SETTINGS frames and MAX_CONCURRENT_STREAMS:
SETTINGS frame:
SETTINGS_MAX_CONCURRENT_STREAMS = 100
When the limit is hit, the client receives a REFUSED_STREAM error and must retry. This is the closest HTTP/2 gets to application-level backpressure — it limits concurrency, not rate.
Gotchas
TCP-level HOL blocking: HTTP/2 multiplexes streams over one TCP connection. TCP itself doesn’t know about H2 streams. If a single large DATA frame gets stuck in TCP retransmission, every other stream on that connection stalls. This is why HTTP/3 (QUIC) exists.
Window exhaustion is silent by default: if your sender hits the flow control window and blocks, most HTTP/2 implementations just… wait. No error, no log, no metric. Your throughput drops and you have no idea why unless you’re instrumenting at the frame level.
Proxies break flow control: nginx, envoy, and most L7 proxies buffer upstream responses before sending them to the client. This means flow control signals from the client don’t propagate to your backend — the proxy absorbs them. Your backend can be completely saturated while the proxy happily accepts more connections.
gRPC: Flow Control Built on HTTP/2, Plus Streaming Semantics
gRPC runs on HTTP/2, so it inherits everything above. But gRPC adds its own layer: structured streaming with server-streaming, client-streaming, and bidirectional RPCs. This is where backpressure gets interesting.
Unary vs Streaming
For unary RPCs (one request, one response), backpressure is mostly handled by concurrency limits and deadlines. The server either responds or doesn’t; there’s no flow to control.
For streaming RPCs, especially bidirectional, you have explicit control over when to read and when to write.
// Server-side streaming — server controls write rate
func (s *MyServer) StreamData(req *pb.Request, stream pb.MyService_StreamDataServer) error {
for _, item := range s.getData(req) {
// Check if client can receive more
select {
case <-stream.Context().Done():
return stream.Context().Err()
default:
}
if err := stream.Send(item); err != nil {
return err // This returns when the client-side window is full
}
// Optional: yield to allow flow control to propagate
// time.Sleep(time.Millisecond) // only if you actually need rate limiting
}
return nil
}
stream.Send() blocks when the HTTP/2 flow control window is exhausted. This is your backpressure signal from the network layer. The server is naturally throttled by the client’s ability to consume.
The onReady Pattern (Client-Side)
On the client side of a bidirectional stream in Java (grpc-java exposes this most explicitly):
// grpc-java bidirectional streaming with manual flow control
StreamObserver<Request> requestObserver = stub.bidiStream(new StreamObserver<Response>() {
@Override
public void onNext(Response response) {
// Process response
// Call request(1) to signal you're ready for the next one
call.request(1);
}
// ...
});
// In ClientCall.Listener
@Override
public void onReady() {
// Called when the channel has capacity to send
// Only send when this is called — don't buffer unboundedly
if (outstandingRequests.hasMore()) {
requestObserver.onNext(outstandingRequests.next());
}
}
The onReady / request(n) pattern is gRPC’s explicit flow control API. request(n) tells the gRPC runtime "deliver me n more messages". Without it, gRPC buffers incoming messages unboundedly. With it, you have full control.
Most gRPC tutorials don’t mention this. Most production codebases don’t use it. This is why those codebases OOM under load.
Deadlines, Not Retries
gRPC’s recommended pattern for handling downstream saturation isn’t retry loops — it’s deadline propagation:
// Propagate the parent deadline downstream
func (s *Gateway) ProcessRequest(ctx context.Context, req *pb.Request) (*pb.Response, error) {
// Don't create a new timeout — use what the caller gave you
// If the caller has 500ms left, don't give the backend 5 seconds
resp, err := s.backendClient.DoWork(ctx, &pb.BackendRequest{
Data: req.Data,
})
if err != nil {
st := status.FromContextError(ctx.Err())
return nil, st.Err()
}
return resp, nil
}
If your backend is saturated and you retry with fresh timeouts, you make it more saturated. Propagate the original deadline. Let it expire naturally. The upstream caller gets a clean timeout error, not a cascade of retries.
Gotchas
Many streams, each within window limit, still saturates a server: HTTP/2 flow control is per-stream. One stream sending at 1MB is throttled. One hundred streams each sending at 1MB is not — they each get their own window. MAX_CONCURRENT_STREAMS is your only defense here.
keepalive and flow control interact badly: if your connection is idle (flow control window full, nothing to send), keepalive pings can fail because the TCP buffer is also full. This causes spurious connection resets. Set GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS=0 unless you specifically need it.
Load balancers don’t see gRPC stream multiplexing: an L4 load balancer sees one TCP connection per client. All 100 streams from one client go to one backend instance. Your "load balanced" system is actually all traffic hitting one pod. Use client-side load balancing with a service mesh, or use grpc.experimental.gevent / the built-in round-robin picker.
// Force round-robin at the client — don't let one backend absorb everything
conn, err := grpc.Dial(
"dns:///my-service.default.svc.cluster.local:50051",
grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
Cross-Cutting Patterns
Adaptive Concurrency
Rather than guessing at fixed thread pool or connection pool sizes, measure and adjust. The principle: if latency is increasing, you’re adding load faster than you can handle it — reduce concurrency.
import time
from threading import Semaphore
class AdaptiveSemaphore:
"""Dead simple adaptive concurrency limiter."""
def __init__(self, initial=10, min_limit=2, max_limit=100):
self.limit = initial
self.min_limit = min_limit
self.max_limit = max_limit
self._sem = Semaphore(initial)
self.ema_latency = None
def acquire(self):
self._sem.acquire()
return time.monotonic()
def release(self, start_time: float, success: bool):
latency = time.monotonic() - start_time
if self.ema_latency is None:
self.ema_latency = latency
else:
self.ema_latency = 0.9 * self.ema_latency + 0.1 * latency
# If we're trending worse, back off
if success and latency < self.ema_latency * 1.1:
if self.limit < self.max_limit:
self.limit += 1
self._sem.release() # Add a token
elif not success or latency > self.ema_latency * 1.5:
if self.limit > self.min_limit:
self.limit -= 1
# Don't release the token — shrink the pool
return
self._sem.release()
This is a toy implementation — production-grade versions exist in libraries like resilience4j (Java), concurrency-limits (Netflix), or via Envoy’s adaptive concurrency filter.
Explicit Queue Depth Limits
Kafka has consumer lag, HTTP/2 has flow control windows, gRPC has request(n). But in your own application code, you’re probably still using unbounded queues somewhere. Stop.
// Bounded channel — this IS your backpressure mechanism in Go
workQueue := make(chan Job, 1000) // If this fills up, callers block or drop
go func() {
for job := range workQueue {
process(job)
}
}()
// Non-blocking enqueue with explicit backpressure
select {
case workQueue <- job:
// queued
default:
// Queue full — reject with 429, or block, or drop (your call)
return ErrBackpressure
}
The size of that channel is a SLA commitment. Make it deliberately, not by default.
Putting It Together
When you operate across all three of these systems simultaneously — say, HTTP/2 ingress → gRPC internal call → Kafka publish — backpressure only works if every layer propagates it:
- HTTP/2 client hits
WINDOW_UPDATEdelay → latency spikes visible to caller - gRPC stream
Send()blocks → deadline propagates upstream - Kafka
pause()fires → consumer stops asking for records
The failure mode is when one layer swallows the signal. A proxy with an unbounded buffer eats the HTTP/2 flow control. A retry loop ignores the deadline. An unbounded in-memory queue hides the Kafka lag.
Backpressure is only as strong as its weakest propagation point. Audit every buffer in your system — every in-memory queue, every HTTP proxy configuration, every thread pool — and ask whether it has a bound, and what happens when that bound is hit.
If the answer is "it grows until memory runs out," you’ve found your next 3 AM incident.