High-Velocity Webhook Engineering: Low-Level Ingress, Dynamic Flow Control, and Multi-Tenant Isolation

Webhook traffic is bursty by nature. A payment provider, a Git host, or a SaaS platform can fire a large batch of events at your endpoint at once. When a router fans those events out to many downstream customers, three problems show up: the cost of network I/O at the edge, one slow customer stalling everyone else, and noisy senders consuming shared capacity.

This guide covers three building blocks of a resilient multi-tenant webhook router:

1. Efficient ingress with Linux io_uring: multishot receive and provided buffer rings, plus the kernel’s separate zero-copy receive feature.
2. Dynamic flow control: bounded per-tenant queues, credit-style backpressure, and load shedding to stop “slow consumer” cascades.
3. Per-tenant rate limiting: an atomic token bucket in Redis (Lua), with notes on Redis Cluster.

One caveat applies throughout. Many webhook workloads are limited by TLS termination, HTTP parsing, signature verification, and downstream I/O rather than by recv() calls. Profile before you adopt anything in section 1.

1. Ingress with io_uring: Multishot Receive and Provided Buffers

Where the standard epoll model spends its time

Most high-performance Linux network services use readiness-based event loops. Tokio, for example, runs on epoll through the mio library. Epoll tells you a socket is ready, and your code then issues the actual system call. Reading one chunk of a webhook request looks like this:

1. The NIC DMA-copies the packet into kernel memory, and the kernel’s network stack processes it and queues the payload on the socket.
2. epoll_wait() returns, telling userspace the socket is readable.
3. Userspace calls read() or recv(), which crosses the user/kernel boundary.
4. The kernel copies the payload from the socket buffer into the application’s buffer.
5. The application parses and processes the bytes.

Each step is cheap. At very high request rates, though, per-call overhead and the kernel-to-user copy become measurable. Gains from io_uring vary widely with workload, kernel version, mitigations, and batching, so benchmark your own stack instead of relying on generic figures.

What io_uring changes

io_uring is a Linux asynchronous I/O interface built around two ring buffers shared between the kernel and the application: a submission queue (SQ) and a completion queue (CQ). Instead of waiting for readiness and then making a call, the application submits operations and later reaps completions. Many operations can be submitted and completed with a single io_uring_enter() call.

Two features matter most for webhook ingestion:

• Multishot receive. The kernel’s io_uring_prep_recv documentation describes a multishot variant in which one receive request repeatedly posts completions as data arrives, instead of needing a new submission per receive. Multishot receive is available since kernel 6.0.
• Provided buffer rings. The application registers a pool of buffers with the kernel, and each completion consumes one buffer from the pool. The completion reports which buffer was used. The application hands buffers back to the ring when it has finished with them. This allows memory to be allocated per-pool rather than per-connection.

A correction on “zero-copy”

Multishot receive with provided buffers is not zero-copy. The kernel still receives packets into its own memory, runs the TCP stack, and copies the payload into your provided buffer. What you save is submission overhead and per-connection buffer pinning.

True zero-copy receive is a separate feature, io_uring zero-copy Rx (zcrx), added in Linux 6.15. The kernel documentation says it removes the kernel-to-user copy on the receive path, so payload data lands directly in userspace memory. Unlike full kernel-bypass stacks such as DPDK, packet headers are still processed by the kernel TCP stack. It also has strict hardware requirements:

• The NIC must support header/data split, so headers go to kernel memory and payloads go to userspace memory.
• Flow steering is needed to direct only the intended flows to the zero-copy queues.
• RSS is needed to steer all other traffic away from those queues.

The kernel API does not configure the NIC for you; you have to set it up yourself. For a typical webhook gateway behind a load balancer on cloud VMs, zcrx may simply be unavailable. Multishot receive with provided buffers is the more practical starting point.

How the data flows

 Userspace                                   Kernel
 ┌──────────────────────┐   shared rings    ┌──────────────────────────┐
 │ Submission queue (SQ)│ ───────────────►  │ Executes multishot recv  │
 │ Completion queue (CQ)│ ◄───────────────  │ on the socket            │
 │ Provided buffer ring │ ◄── payload ───── │ Copies payload from the  │
 └──────────────────────┘    copied into    │ socket buffer into a     │
                             a free buffer  │ provided buffer          │
                                            └────────────▲─────────────┘
                                                         │
                                        NIC → kernel network stack (TCP)

A Rust sketch

Rust has several io_uring options. The io-uring crate from the tokio-rs project is a low-level binding, and several runtimes (for example tokio-uring, monoio, and glommio) build on or implement io_uring. Tokio’s main runtime itself uses epoll.

The sketch below submits a multishot receive and handles completions. It is simplified: it omits buffer-ring registration and error handling, and the exact API differs between crate versions, so check the docs for the version you use. Buffer rings are registered through the Submitter, for example with register_buf_ring_with_flags, and require careful unsafe memory management.

use io_uring::{cqueue, opcode, types, IoUring};
use std::{io, os::fd::RawFd};

const BUF_GROUP: u16 = 1; // buffer group id of a registered provided buffer ring

fn submit_multishot_recv(ring: &mut IoUring, conn_fd: RawFd) -> io::Result<()> {
    // One submission; the kernel posts a completion each time data arrives.
    let sqe = opcode::RecvMulti::new(types::Fd(conn_fd), BUF_GROUP)
        .build()
        .user_data(conn_fd as u64);

    unsafe {
        ring.submission()
            .push(&sqe)
            .map_err(|_| io::Error::other("submission queue full"))?;
    }
    ring.submit()?;
    Ok(())
}

fn reap_completions(ring: &mut IoUring) {
    for cqe in ring.completion() {
        let bytes = cqe.result(); // number of bytes received, or a negative errno
        let buf_id = cqueue::buffer_select(cqe.flags()); // which provided buffer was filled
        let multishot_still_armed = cqueue::more(cqe.flags());

        // 1. If bytes > 0 and buf_id is Some, read the payload from that buffer,
        //    then return the buffer to the ring.
        // 2. If the multishot request is no longer armed (no MORE flag),
        //    re-submit it for this connection.
        let _ = (bytes, buf_id, multishot_still_armed);
    }
}

Security and operational caveats

io_uring has a large kernel attack surface. In June 2023, Google’s security team reported that 60% of the Linux kernel exploits submitted to its bug bounty program in 2022 involved io_uring. As a result, Google disabled it for Android apps, on ChromeOS, and on its servers. Docker also removed it from its default seccomp profile, which means io_uring calls are blocked by default in many container environments. Check your runtime’s seccomp policy before assuming it will work in production. Administrators can also restrict it system-wide using the kernel.io_uring_disabled sysctl. For a service that handles untrusted internet input, such as a webhook gateway, weigh the throughput gains against that exposure.

2. Preventing the Slow Consumer Cascade

Head-of-line blocking in a multi-tenant router

A webhook router accepts events and forwards them to many customer endpoints. If one customer’s endpoint slows or goes offline and the router does not isolate that customer, the failure spreads:

1. Shared workers block while trying to deliver to the slow tenant.
2. Healthy tenants’ events queue behind them.
3. Threads, connections, and memory run out, and the failure becomes platform-wide.

Incoming events
   │
   ├─► Tenant A queue (full) ──► Endpoint A (slow or failing)  → circuit opens, A's events shed or parked
   │
   └─► Tenant B queue (healthy) ─► Endpoint B (fast)           → unaffected

The core rule is that a delivery problem for one tenant must never consume a resource that another tenant needs.

Bounded queues as credits

Credit-based flow control is an established idea in networking. It appears in technologies such as Fibre Channel, InfiniBand, and PCIe, where a sender may transmit only while it holds credits granted by the receiver. The same idea works at the application layer.

Give each tenant a fixed budget of in-flight events:

• Accepting an event consumes one credit from that tenant’s budget.
• A credit is returned only when the event leaves the queue, either after successful delivery or after it is moved to durable storage.
• When a tenant has no credits left, the ingress path does not block. It sheds load or spills to durable storage.

A bounded Tokio channel is a natural fit for this. Its capacity is the credit budget, and try_send (or try_reserve) fails immediately instead of waiting.

What “shedding” means depends on who is sending

How you reject an event matters because the upstream provider’s retry behavior varies. Stripe, for instance, documents that in live mode it retries delivery for up to three days with exponential backoff, and recommends returning a 2xx quickly before doing heavy work. If your router returns 429 or 503 to such a sender, the event will come back later. Other senders may retry only briefly or not at all. The safer pattern is:

1. Verify the signature and persist the event durably first, then return 2xx quickly.
2. Deliver downstream asynchronously from a per-tenant queue.
3. When a tenant’s in-memory queue is full, spill to a disk-backed or broker-backed queue (RocksDB, Kafka, SQS, or similar) rather than dropping.
4. Reserve 429 or 503 with a Retry-After header for cases where durable storage itself is saturated, and only for providers you know will retry.

Per-tenant isolation in Rust

The sketch below gives each tenant a bounded channel and its own consumer task. It uses a read-write lock so lookups for different tenants do not serialize on a single mutex.

use std::{collections::HashMap, sync::Arc, time::Duration};
use tokio::sync::{mpsc, RwLock};

#[derive(Debug)]
pub struct WebhookPayload {
    pub event_id: String,
    pub body: Vec<u8>,
}

#[derive(Debug)]
pub enum IngestError {
    UnknownTenant,
    TenantSaturated, // out of credits: spill to durable storage or shed
    TenantClosed,
}

pub struct RouterManager {
    tenants: RwLock<HashMap<String, mpsc::Sender<WebhookPayload>>>,
}

impl RouterManager {
    pub fn new() -> Arc<Self> {
        Arc::new(Self { tenants: RwLock::new(HashMap::new()) })
    }

    pub async fn register_tenant(&self, tenant_id: String, capacity: usize) {
        let (tx, mut rx) = mpsc::channel::<WebhookPayload>(capacity);
        let id = tenant_id.clone();

        // One isolated consumer per tenant: a slow endpoint only stalls its own queue.
        tokio::spawn(async move {
            while let Some(event) = rx.recv().await {
                // Bound every delivery attempt so a hung endpoint cannot block forever.
                let _ = tokio::time::timeout(
                    Duration::from_secs(10),
                    deliver(&id, &event),
                )
                .await;
                // On timeout or error: schedule a retry, and trip a per-tenant circuit breaker.
            }
        });

        self.tenants.write().await.insert(tenant_id, tx);
    }

    pub async fn ingest(&self, tenant_id: &str, event: WebhookPayload) -> Result<(), IngestError> {
        let tenants = self.tenants.read().await;
        let tx = tenants.get(tenant_id).ok_or(IngestError::UnknownTenant)?;

        match tx.try_send(event) {
            Ok(()) => Ok(()),
            Err(mpsc::error::TrySendError::Full(_)) => Err(IngestError::TenantSaturated),
            Err(mpsc::error::TrySendError::Closed(_)) => Err(IngestError::TenantClosed),
        }
    }
}

async fn deliver(_tenant_id: &str, _event: &WebhookPayload) {
    // HTTP POST to the customer's endpoint with signing, timeouts, and retry scheduling.
}

Two design notes. First, a bounded channel gives you fairness in memory, but you still need per-tenant concurrency limits on outbound connections, otherwise one tenant with thousands of queued events can monopolize the shared HTTP client pool. Second, a circuit breaker per endpoint, which stops attempts after repeated failures and probes occasionally for recovery, prevents wasted work against a dead endpoint.

3. Per-Tenant Rate Limiting with a Redis Token Bucket

Why rate limit

Failures and bugs can create floods without any malicious intent. A customer’s endpoint that returns errors triggers retries, and a buggy integration can loop and resend events. Per-tenant limits keep one sender’s problem from consuming shared capacity.

Atomic token bucket in Lua

A token bucket stores the number of tokens and the time of the last refill. A read-modify-write sequence done through separate GET and SET commands is racy when several gateway nodes run it concurrently. A Redis Lua script runs atomically as a single command, so no other command executes in the middle of it.

The script below improves on the common version in three ways:

• It reads time from Redis with TIME instead of trusting each gateway’s clock, which avoids skew between nodes. Reading time in scripts works with current Redis replication behavior, where scripts replicate by their effects.
• It uses HSET, because HMSET has been deprecated since Redis 4.0.
• It returns floats as strings, because Redis converts Lua numbers in replies to integers and truncates them.

-- KEYS[1]: bucket key, e.g. "{tenant_xyz}:rate_limit"
-- ARGV[1]: bucket capacity (maximum burst)
-- ARGV[2]: refill rate in tokens per second
-- ARGV[3]: tokens requested for this call (usually 1)

local key         = KEYS[1]
local capacity    = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested   = tonumber(ARGV[3])

local t   = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local bucket = redis.call("HMGET", key, "tokens", "ts")
local tokens = tonumber(bucket[1])
local ts     = tonumber(bucket[2])

if tokens == nil or ts == nil then
  tokens = capacity
else
  local elapsed = math.max(0, now - ts)
  tokens = math.min(capacity, tokens + elapsed * refill_rate)
end

local allowed = 0
local retry_after = 0
if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
else
  retry_after = (requested - tokens) / refill_rate
end

redis.call("HSET", key, "tokens", tokens, "ts", now)
-- Expire idle buckets once they would have fully refilled anyway.
redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate * 1000))

return { allowed, tostring(tokens), tostring(retry_after) }

On a denied request, use retry_after to populate a Retry-After header.

Redis 7.0 also introduced Redis Functions, which are invoked with FCALL. Unlike EVAL scripts, functions are persisted with the dataset and replicated to replicas, so you do not need to resend script source from every client. Consider them if you run Redis 7 or later.

Redis Cluster considerations

• Declare every key in KEYS. In Redis Cluster, all keys a script touches must be passed in KEYS and must hash to the same slot. Accessing undeclared keys inside a script breaks cluster routing.
• Use hash tags when a script touches multiple keys. The script above uses one key per tenant, so it works as is. If you extend it to also read, say, a per-tenant configuration key, name both with the same tag (for example {tenant_xyz}:bucket and {tenant_xyz}:config) so they land in the same slot.
• Expect hot slots. Each tenant’s bucket lives on one node, so a single very busy tenant is bounded by what that node can serve. Spread tenants across slots and monitor slot-level load.
• Limit round trips. Redis executes commands on a main thread per node, so every limiter call costs capacity there. For very high-volume tenants, let each gateway keep a local bucket and lease tokens from Redis in batches (for example, take 100 tokens at a time and refill when they run low). This reduces Redis calls and latency at the cost of precision: the global limit becomes approximate by up to the total size of outstanding leases. Choose the batch size according to how strict the limit must be. The reduction in Redis traffic depends on your batch size and traffic pattern, so measure it.

Summary

Further Reading

• Linux kernel documentation: io_uring zero copy Rx
• liburing manual: io_uring_prep_recv, including the multishot variant
• Stripe documentation: Receive Stripe events in your webhook endpoint
• Redis documentation: Lua scripting with EVAL and Redis Functions