Most teams start with fixed rate limits. “100 requests per second per user.” It sounds controlled. But it doesn’t protect a service when latency rises, queues grow, or clients retry aggressively.

A request that takes 20 milliseconds counts the same as one that holds a database connection for five seconds. Both consume one unit of your limit. When the expensive requests pile up, the service runs out of capacity. The rate limiter happily lets more in right up until the service collapses.

Static limits answer “how many?” They don’t answer “how much?” And in production systems, “how much” matters far more than “how many.”

The problem with static rate limits

Static rate limits were designed for traffic control, not service protection. They regulate the flow of incoming requests without caring what happens after those requests enter your service.

Consider a payment service. You set 200 RPS per merchant. Everything looks fine until one merchant triggers a slow query. That query holds a database connection for four seconds. Meanwhile, the client retries on timeout. Each retry spawns another slow query. The rate limiter doesn’t catch this because the client is still under 200 RPS.

The service exhausts its connection pool. Memory fills with queues. CPU saturates. The service doesn’t gracefully degrade, it dies.

This is the core limitation: requests are not equal.

Why RPS alone is misleading

Rates don’t measure resource consumption. A single request that triggers a cache miss, three database calls, and an external API fetch uses far more resources than 100 lightweight requests hitting a warm cache. The rate limiter treats them identically. The expensive request is the one that breaks the service, but the limit can’t tell the difference.

How retry storms make overload worse

When a service under load returns errors, retries amplify the problem. A client retrying three times triples its effective load. If the service is already struggling, those retries push it further past the breaking point.

Aggressive retries without jitter synchronize across clients. Ten clients retrying at the same moment create waves of traffic that swamp services not built for sudden spikes.

Why queues hide failure

Most services queue requests when they’re busy. The queue sits between incoming traffic and actual work. It creates an illusion of health because requests keep getting accepted. But the queue is just storing the problem.

Eventually the queue fills up. Memory consumption spikes. Consumer threads block. Latency balloons from milliseconds to seconds. The service appears fine until it suddenly isn’t, and by then recovery takes much longer than prevention would have.

Traffic control vs. service protection

Traffic control shapes the volume of incoming requests. It asks “who can come in?” Service protection asks “can I handle this right now?” These are different questions. A rate limiter addresses the first. Adaptive load shedding addresses the second.

You need both. Traffic control throttles abusive clients. Service protection keeps the system healthy during genuine demand spikes.

Define the overload signals

Before you can intelligently shed load, you need to know when your service is actually struggling. Watch these signals:

In-flight requests. The number of requests currently executing tells you how much work your service is actively doing. This is more meaningful than incoming RPS because it measures actual resource consumption.

P95/P99 latency. Rising latency means your service is becoming slower. If your P99 jumps from 100ms to 800ms, something is wrong long before errors start appearing for end users.

Queue depth. Growing queues mean your consumers can’t keep up. A queue that normally holds five items but suddenly has 200 requests is screaming for attention.

CPU and memory pressure. High utilization leaves no room for new work. If your service is at 90% CPU, accepting another request might push it past the tipping point.

Database connection pool saturation. If all connections are in use, additional requests wait or fail. Connection pool exhaustion is one of the fastest ways a service degrades because recovery requires waiting for existing transactions to complete.

Error rate. Rising errors often precede total failure. Catching them early lets you shed load before the cascade becomes unstoppable.

Timeout rate. More timeouts mean your SLAs are slipping and downstreams are backing up. Timeouts are an early warning system.

Downstream dependency health. If your payment service is slow, your checkout service shouldn’t keep accepting orders that will fail anyway. Route health should influence load acceptance.

Adaptive concurrency limits

The model is simple. A server allows only a safe number of concurrent requests. If latency rises, the allowed concurrency drops. If latency recovers, the limit slowly increases.

Envoy’s adaptive concurrency filter does exactly this. It dynamically adjusts allowed outstanding requests using latency samples. The implementation isn’t machine learning. It’s feedback.

Here’s the logic:

import collections

class AdaptiveConcurrencyLimiter:
    def __init__(self, min_c=10, max_c=100, target_latency=0.1):
        self.min = min_c
        self.max = max_c
        self.target = target_latency
        self.current = max_c / 2
        self.latencies = collections.deque(maxlen=20)
        self.in_flight = 0
        self.lock = threading.Lock()

    def allow_request(self):
        with self.lock:
            if self.in_flight < self.current:
                self.in_flight += 1
                return True
            return False

    def release_request(self):
        with self.lock:
            self.in_flight -= 1

    def record_latency(self, latency):
        self.latencies.append(latency)
        if len(self.latencies) < 5:
            return

        avg = sum(self.latencies) / len(self.latencies)
        if avg > self.target * 1.5:
            self.current = max(self.min, self.current * 0.9)
        elif avg < self.target * 0.8:
            self.current = min(self.max, self.current * 1.05)

If latency is 50% above target, drop concurrency by 10%. If it’s 20% below target, raise it by 5%. Use averages so transient spikes don’t trigger unnecessary thrashing.

This is reactive, not predictive. You’re not forecasting future traffic. You’re watching what your service is actually experiencing and adjusting accordingly. That simplicity is exactly what makes it work in production.

The request decision path

Every incoming request goes through this sequence:

1. Is the client allowed? Check authentication and quotas. If the client has exceeded their budget or is unauthorized, reject immediately.
2. Is the route healthy? Some endpoints are more expensive than others. If a route is in a degraded state, shed more aggressively.
3. Is the dependency healthy? If the downstream service is down or slow, routing traffic there wastes resources.
4. Is the concurrency budget available? Check current in-flight count against the adaptive limit.
5. What’s the outcome?

Accept: All checks pass. Execute the request. Queue: If the request can tolerate delay and the queue has room. Degrade: Return a simplified response. Reject: Return 429 or 503 with retry guidance.

This path gives you five distinct responses, not just “accept” or “reject.” That flexibility is the difference between a service that crashes and one that keeps running.

Graceful degradation

Accepting that you can’t serve everyone perfectly is the first step toward resilience. Here are concrete patterns:

Return cached data. If you have a recent cache hit, serve it. Stale data is almost always better than an error.

Disable expensive sorting. Return results in a cheaper order. Skip the full-text ranking or use a precomputed score.

Use stale reads. If replication lag is within your tolerance, serve from the read replica instead of waiting for the primary.

Skip non-critical enrichment. Drop the recommendation engine, personalization layer, or analytics tracking for this request. Fetch the core data and return it.

Accept write intent and process later. When write capacity is exhausted, queue the intent and acknowledge the client. Process it when resources free up. The client continues. The service recovers.

Return 429 with retry guidance. Tell the client to back off and for how long. A Retry-After header gives clients better behavior than generic errors.

Return 503 when you’re done. If the service is genuinely overwhelmed, 503 is honest. It tells the client the service is temporarily unavailable and they should try elsewhere or wait.

The key is matching the degradation mode to the request type. A GET request for a public profile can return stale data. A POST request to charge a credit card cannot skip any safety checks.

Fairness

If one tenant can consume your entire budget, adaptive limits don’t help much.

Per-route budgets. Your payment processing route should have a larger budget than your batch export route. Critical paths get more capacity.

Per-tenant budgets. Enterprise customers shouldn’t suffer because one noisy tenant is scraping your entire API. Cap each tenant at a reasonable fraction of the total budget.

Priority lanes. Internal health checks, admin endpoints, and monitoring dashboards should always pass. They need to run even when the service is stressed. Separate these from public traffic so a spike in user requests doesn’t block your operations team from debugging.

Separate read and write limits. Reads are usually latency-tolerant. Writes are not. Give each its own budget so a spike in reads doesn’t starve writes, or vice versa.

Split the total concurrency into pools. A checkout service might allocate 60% to payment routes, 30% to browse operations, and 10% to admin. If one pool is exhausted, only that pool rejects traffic. The rest continue operating.

Testing the design

You can’t trust adaptive limits without testing them under stress. Here’s how to validate your design:

Load testing. Generate traffic beyond peak capacity and watch how the system reacts. Does concurrency drop appropriately? Do rejection rates climb linearly or exponentially?

Latency injection. Artificially delay downstream dependencies. Your concurrency limit should notice and decrease. If it doesn’t, your latency threshold is set too high.

Dependency failure simulation. Kill or delay a downstream service. Routes that depend on it should stop accepting traffic or degrade gracefully within a reasonable time window.

Retry storm simulation. Create clients that retry aggressively with short backoffs. Verify your system holds up and doesn’t enter a positive feedback loop where retries increase load, which causes more errors, which trigger more retries.

Queue depth tests. Fill queues to capacity and verify rejection logic triggers before memory exhaustion.

Brownout testing. Gradually reduce capacity and ensure graceful degradation activates in the right order. Serve cache first, then skip enrichment, then queue writes, then reject.

Middleware: in-flight request limiter

Here’s a Python middleware that counts in-flight requests and rejects when the limit is reached:

import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

class ConcurrencyLimitMiddleware:
    def __init__(self, max_concurrency=50):
        self.max = max_concurrency
        self.in_flight = 0
        self.lock = threading.Lock()
        self.stats = {"rejected": 0, "accepted": 0}

    def wrap(self, handler):
        original_do_METHOD = getattr(handler, 'do_GET', None) or getattr(handler, 'do_POST', None)

        def limited_handler():
            with self.lock:
                if self.in_flight >= self.max:
                    self.stats["rejected"] += 1
                    handler.send_response(503)
                    handler.send_header('Retry-After', '1')
                    handler.end_headers()
                    handler.wfile.write(b'Service temporarily at capacity')
                    return
                self.in_flight += 1
                self.stats["accepted"] += 1

            try:
                original_handler = handler.__class__.do_GET if hasattr(handler, 'do_GET') else handler.__class__.do_POST
                return original_handler(handler)
            finally:
                with self.lock:
                    self.in_flight -= 1

        return limited_handler

This is the static foundation. The adaptive limiter just changes the maximum dynamically based on observed latency.

Client retry behavior

Clients need to retry carefully. Aggressive retries turn isolated failures into storms.

import random
import time

def exponential_backoff_with_jitter(attempt, base_delay=0.1, max_delay=30):
    delay = min(base_delay * (2 ** attempt), max_delay)
    delay = delay * (0.5 + random.random())
    return delay

def safe_retry_request(request_fn, max_retries=3, idempotent_only=True):
    if idempotent_only and not getattr(request_fn, 'idempotent', False):
        raise ValueError("Cannot retry non-idempotent request")

    for attempt in range(max_retries + 1):
        try:
            return request_fn()
        except (ServiceBusyError, TimeoutError) as e:
            if attempt < max_retries:
                time.sleep(exponential_backoff_with_jitter(attempt))
                continue
            raise

AWS notes that APIs with side effects are not safe to retry unless they explicitly provide idempotency keys. Your retry logic should inspect request methods or headers. Safe to retry: GET, PUT with idempotency key, DELETE. Unsafe to retry: POST without idempotency guarantees.

The bottom line

Static rate limits are necessary but insufficient. They prevent abuse. They don’t prevent collapse.

Adaptive load shedding adds a feedback loop. Your service watches its own health. When it struggles, it does less. When it recovers, it does more. It’s not smart. It’s not predictive. But it works.

The goal isn’t to reject as much traffic as possible. It’s to keep the service running for as many requests as possible without failing hard. Sometimes that means rejecting one request so ten others can succeed.

That trade-off is worth making.