The Multi-Tenant Neighbor Problem: Why Traditional API Rate Limiters Fail Under Webhook-Heavy Bursts

Modern software architectures rely heavily on webhooks. Whether your application processes payments via Stripe, synchronizes inventory with Shopify, or reacts to repository events from GitHub, webhooks are how many real-time, event-driven systems learn that something happened. As applications scale, engineering teams run into an architectural problem known as the multi-tenant neighbor problem, often called the noisy neighbor problem.

Consider a hypothetical scenario. Your SaaS platform serves thousands of businesses, each a tenant with its own data and API credentials. One of your largest customers launches a flash sale on Shopify, and your ingestion endpoint suddenly receives tens of thousands of webhook requests per second. For illustration, this article uses 50,000 requests per second (RPS). That figure is an assumption, not a published Shopify number. It is a plausible order of magnitude for a platform that aggregates events from many merchants, and a single merchant would rarely produce it alone.

If your system uses rate-limiting strategies designed for ordinary user traffic, it may crash, bottleneck, or starve unrelated customers.

This guide explains why traditional rate limiters struggle with webhook spikes, how the neighbor problem plays out in a real ingestion pipeline, and which patterns help you build a resilient one. Provider behavior described here (timeouts, retries) comes from the providers’ own documentation, linked at the end.

1. Understanding the Multi-Tenant Neighbor Problem in Webhooks

In a shared multi-tenant architecture, CPU, memory, database connection pools, and network capacity are pooled across customers. This improves hardware efficiency and lowers cost, but it exposes every tenant to the behavior of the loudest one.

How Webhook Traffic Differs from User Traffic

Ordinary user traffic is spread across many endpoints (login, dashboard, search) and paced by human interaction. Webhook traffic behaves differently:

• Event-driven bursts. Providers push data when events occur. Bulk operations, such as a flash sale, a mass product update, or a batch import, can generate thousands of events in a very short window. For scale, Shopify reported that during Black Friday–Cyber Monday 2025 its API requests peaked at 31.8 million per minute and merchant sales peaked at $5.1 million per minute. Those are platform-wide figures rather than webhook counts, but they show how sharply commerce traffic can spike.
• Tight response deadlines. Providers expect a fast acknowledgment. Shopify fails a delivery if your app doesn’t respond within five seconds, and GitHub gives your server 10 seconds before it terminates the connection and records a failure. Stripe’s documentation asks endpoints to return a 2xx status code quickly, before doing complex work.
• Provider-side retries. Retry behavior varies by provider, and it can amplify load on a struggling receiver:
  • Stripe, in live mode, attempts delivery for up to three days with exponential backoff. In sandbox mode it retries three times over a few hours.
  • Shopify retries failed deliveries up to eight times over four hours. If failures persist, it removes the webhook subscription.
  • GitHub does not automatically redeliver failed deliveries. You have to redeliver them manually or with a script.
• Per-request work. Webhook handlers typically verify a cryptographic signature, resolve which tenant the event belongs to, and persist or enqueue the payload.

When one tenant generates a traffic surge, it competes with every other tenant for the same shared resources. The others see higher latency, timeouts, or dropped events even though their own traffic hasn’t changed.

2. Why Traditional API Rate Limiters Fail

Common algorithms such as token bucket, leaky bucket, and fixed or sliding window counters, often backed by Redis, work well for protecting CRUD APIs from scrapers and runaway client scripts. They break down for webhook ingestion for several reasons.

A. IP-Based Limiting Doesn’t Identify Tenants

Webhooks from a given provider come from that provider’s infrastructure, not from your tenant’s. Every tenant’s events arrive from the same pool of source addresses. Limiting by IP therefore either throttles the provider as a whole, which punishes every tenant using that integration, or fails to single out the one tenant causing the burst. IP allowlists are still useful as an additional security layer where providers publish ranges (GitHub, for example, exposes its addresses through its GET /meta endpoint), but IP is the wrong key for fairness between tenants.

B. Global or Per-Route Limits Cause Cross-Tenant Starvation

Suppose you cap /api/webhooks/shopify at 10,000 RPS. If Tenant A generates 12,000 RPS, the limiter has no concept of who is responsible. Tenant B’s legitimate events are throttled or rejected alongside A’s, which causes missed order updates and out-of-sync data.

C. A Centralized Counter Becomes a Hot Spot

Distributed rate limiters commonly rely on Redis atomic operations (INCR, EXPIRE, or Lua scripts). Redis executes commands on a single main thread, so the problem is not lock contention. It is that one hot key lives on one shard and every limiter check is a network round trip to it. At tens of thousands of checks per second, the limiter’s own latency, network bandwidth, and availability become part of your request path. If the limiter is unreachable, you have to decide whether to fail open (and risk overload) or fail closed (and reject legitimate traffic).

D. Rejecting a Webhook Doesn’t Make It Go Away

Traditional limiters are binary: allow the request or reject it with 429 Too Many Requests. Webhook senders aren’t ordinary API clients. Shopify counts any non-2xx response as a failed delivery and retries it, and it may eventually remove the subscription. Stripe retries for days. A rejected event is therefore either redelivered later, adding to future load, or lost if the provider gives up or never retries (as with GitHub). Neither outcome is what you want from a load-protection mechanism.

Providers’ documentation does not state that they honor a Retry-After header on webhook deliveries, so don’t design your protection strategy around it. The more reliable approach is to acknowledge quickly and absorb the burst in a durable buffer.

3. Anatomy of a Webhook Ingestion Failure

Here is how an unprotected, synchronous pipeline typically fails under a sudden burst from one tenant:

1. Edge saturation. The load balancer or gateway (NGINX, Envoy, a cloud API gateway) handles a flood of new connections. TLS handshakes and connection handling consume CPU.
2. Per-request overhead. Each request needs signature verification and a lookup of the tenant’s signing secret. The HMAC computation itself is cheap relative to TLS and JSON parsing, but if the secret comes from a database on every request, that lookup becomes the bottleneck.
3. Connection pool exhaustion. Each handler tries to read tenant metadata or write the event to a relational database. Connection pools (for example HikariCP, or PgBouncer in front of PostgreSQL) fill up, threads block waiting for connections, and timeouts cascade.
4. Memory pressure. Requests pile up in application memory while waiting for workers. Eventually the process or container exceeds its memory limit and the kernel’s OOM killer, or the orchestrator, restarts it. A restarted instance that immediately faces the same backlog can fall into a crash loop.
5. Provider retries amplify the damage. Once responses exceed the provider’s timeout (five seconds for Shopify, ten for GitHub), the provider records failures. Where it retries, as Stripe and Shopify do, it adds more load while you are already down.

4. Modern Architectural Patterns to Solve the Neighbor Problem

The core shift is from synchronous processing and flat rate limits to asynchronous ingestion, per-tenant isolation, and tenant-keyed quotas.

Pattern 1: Tenant-Aware Rate Limiting with Dynamic Quotas

Key limits by tenant rather than by route or IP. Depending on the provider, the tenant can be identified from the URL path (for example /webhooks/shopify/{tenant_id}), a subdomain, or a header. Shopify, for instance, includes the originating shop’s domain in a request header.

One caution: the tenant identifier is unauthenticated until the signature has been verified. Anyone who can reach your endpoint can forge a header or URL. Use the identifier to look up the signing secret and verify the signature before charging the request to that tenant’s quota or trusting the payload. Otherwise an attacker can exhaust a victim’s quota with forged requests.

• Hierarchical limiting. Use two tiers:
  1. A global safety cap that protects the infrastructure as a whole.
  2. A per-tenant quota, set by plan tier or historical volume, so Tenant A cannot consume capacity reserved for Tenant B.
• Shed after acknowledging, not before. Because rejected webhooks come back as retries (or are lost), apply per-tenant limits primarily to how fast you process events from the buffer, not to whether you accept them.

Pattern 2: Decouple Ingestion from Processing

Don’t process webhook payloads synchronously. Both GitHub and Shopify advise responding quickly and handling the work in the background, and GitHub specifically suggests a queue for this. The ingestion endpoint should do the minimum:

1. Read the raw request body.
2. Verify the signature against the raw bytes.
3. Write the event to a durable buffer.
4. Return a 2xx response.

• Durable buffers. Apache Kafka, Redpanda, Amazon Kinesis, or a managed queue such as Amazon SQS all work. Choose based on throughput, ordering, and operational needs.
• Partitioning by tenant. In Kafka-style logs, keying messages by tenant ID sends all of a tenant’s events to the same partition, which preserves per-tenant ordering. Ordering is guaranteed only within a partition, and several tenants can share one. A very large tenant can also create a hot partition, so consider sub-keying (for example by entity ID) for heavy tenants if you don’t need strict tenant-wide ordering.
• Idempotency. Delivery is at-least-once, so consumers must tolerate duplicates. Use the provider’s delivery identifier, such as X-Shopify-Webhook-Id for Shopify or X-GitHub-Delivery for GitHub, as an idempotency key. GitHub keeps the same X-GitHub-Delivery value on manual redeliveries.

[Webhook Provider]
       │  burst of requests
       ▼
[Edge / API Gateway]
       │  verify signature, enqueue, return 2xx
       ▼
[Durable Log / Queue]  (partitioned by tenant)
       │
       ▼
[Per-Tenant Rate-Limited Worker Pools]
       │
       ▼
[Tenant Data Stores]

Pattern 3: Prioritization and Graceful Degradation

When queues or worker pools approach saturation, degrade deliberately instead of failing everywhere.

• Priority lanes. Route business-critical events (for example, payment confirmations) to a high-priority queue and lower-value events (analytics, profile updates) to a lower-priority one that can lag during congestion.
• Backlog-aware processing. Slow or pause consumption for the offending tenant while others continue, and alert on queue depth and consumer lag rather than only on request rates.
• Treat non-2xx as a last resort. If you must reject traffic, such as when the buffer itself is unavailable, remember that Shopify caps retries at eight and may remove the subscription, and that GitHub won’t retry at all. Build a reconciliation path: GitHub lets you list and redeliver failed deliveries through its API, and Shopify recommends re-importing data from the outage window after downtime.

Pattern 4: Resource Isolation (Bulkheading)

Bulkheads, borrowed from ship design and resilience engineering, partition resources so that failure or overload in one compartment doesn’t sink everything.

• Dedicated worker pools or queues for high-volume enterprise tenants, with smaller shared pools for everyone else.
• Per-tenant concurrency caps so one tenant cannot occupy all database connections.
• Data partitioning. Shard or partition tenant data so heavy write load from one tenant is confined to its own partitions or databases where practical.

5. A Token Bucket Rate Limiter in Redis (Lua)

To avoid race conditions and extra round trips when enforcing per-tenant limits, run the token bucket logic as an atomic Lua script. Redis executes a script as a single unit, so concurrent callers can’t interleave between the read and the write.

This version reads the clock from Redis itself (TIME) so that clock skew between application servers cannot corrupt the bucket. It requires Redis 5.0 or later, where scripts that call non-deterministic commands are replicated by their effects. It also returns a suggested wait time you can use to pace your own consumers.

-- Tenant token bucket
-- KEYS[1]: bucket key, e.g. ratelimit:tenant:12345
-- ARGV[1]: bucket capacity (maximum burst)
-- ARGV[2]: refill rate (tokens per second)
-- ARGV[3]: tokens requested (usually 1)

local t = redis.call('TIME')
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])

local bucket = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(bucket[1])
local ts = tonumber(bucket[2])

if tokens == nil or ts == nil then
  tokens = capacity
  ts = now
end

-- Refill based on elapsed time
tokens = math.min(capacity, tokens + math.max(0, now - ts) * refill_rate)

local allowed = 0
local retry_after = 0

if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
else
  retry_after = (requested - tokens) / refill_rate
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
-- Let idle buckets expire (twice the time needed to fully refill)
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / refill_rate) * 2)

return {allowed, tostring(retry_after)}

Two practical notes:

• Where to apply it. Use this limiter to meter processing per tenant, such as consumers pulling from the queue, rather than as a gate that rejects webhook deliveries at the edge (see Pattern 3).
• Limiter failure mode. Decide up front what happens if Redis is slow or unavailable. For ingestion, failing open (accept and buffer) is usually safer than failing closed, because the queue provides the real protection. Keep local, in-process fallback limits as a backstop.

6. Summary and Best Practices Checklist

Protecting your ingestion pipeline from the multi-tenant neighbor problem means moving from reactive scaling to proactive isolation:

1. Acknowledge fast, process later. Verify the signature, write to a durable buffer, and return 2xx well inside the provider’s deadline (five seconds for Shopify, ten for GitHub).
2. Key quotas by tenant, not by IP or route. Authenticate the request before charging a tenant’s quota.
3. Verify signatures on the raw body, and cache signing secrets so verification doesn’t hit your database on every request.
4. Meter consumption, not acceptance. Apply per-tenant rate limits and concurrency caps to workers reading from the queue.
5. Design for duplicates. Use delivery IDs for idempotency, since providers retry and consumers may reprocess.
6. Isolate with bulkheads. Give heavy tenants dedicated pools, and cap per-tenant database concurrency.
7. Plan for reconciliation. Know how to recover missed events: GitHub supports redelivery through its API, and Shopify recommends re-importing data after an outage.
8. Monitor the right signals. Watch queue depth, consumer lag, and per-tenant processing time, along with request rate.

A system built this way absorbs bursts in the buffer, throttles the noisy tenant at the processing stage, and keeps its neighbors online.

Sources

• Shopify, Troubleshoot webhooks: five-second response limit, retry behavior, and subscription removal.
• Stripe, Receive Stripe events in your webhook endpoint: retry behavior in live and sandbox modes.
• GitHub Docs, Best practices for using webhooks: 10-second response window, queue recommendation, delivery header.
• GitHub Docs, Handling failed webhook deliveries: no automatic redelivery.
• Shopify, BFCM 2025 data: peak sales and API request volumes.