Lock-Free Ring Buffers in Rust for Multi-Tenant Webhook Ingress: Avoiding Mutex Contention at High Request Rates

Webhook ingestion engines are bursty by nature. A single tenant can send a flood of events while thousands of others send a trickle, and the ingress path has to hand each payload from a network-facing thread to a processing thread without stalling either side. At very high request rates (this article uses 1,000,000 requests per second as its reference point), the queue between those two stages becomes one of the hottest shared data structures in the process.

This article explains why a lock-protected queue can become a bottleneck under heavy contention, what bounded lock-free ring buffers are and how they work, and how to build one in Rust. It also covers the production alternatives (crossbeam-queue and crossbeam-channel) and how to keep one noisy tenant from starving the others. It does not publish benchmark numbers, because queue performance depends heavily on hardware, payload size, contention pattern and kernel version. A closing section explains how to measure it yourself.

The Bottleneck: What Actually Goes Wrong with Mutexes

First, a correction to a common back-of-the-envelope calculation. 1M RPS does not mean “1 microsecond per request” for your application as a whole. It means one request arrives every microsecond on average. With N cores working in parallel, each core has roughly N microseconds of CPU time per request. The real constraint is the shared queue: every request passes through it, so any per-operation cost on that queue is multiplied by the full request rate, and any serialization limits total throughput no matter how many cores you add.

A mutex-protected queue serializes all producers and consumers on one lock. Under heavy contention, three things tend to go wrong.

1. Parking and context switches

Since Rust 1.62, the standard library’s std::sync::Mutex on Linux is built on a futex word rather than a pthread mutex. The implementation spins briefly while the lock is held, then falls back to a futex_wait system call that puts the thread to sleep. An uncontended lock or unlock is just an atomic operation, so mutexes are cheap when contention is low. The trouble starts when many threads compete for the same lock: waiters end up in the kernel, and the unlocking thread has to issue a wake-up call.

Published microbenchmarks put the direct cost of a Linux thread context switch at roughly 1–4 microseconds. One measurement on x86 found about 1.2–1.5 µs when pinned to a single core and about 2.2 µs unpinned. Another reported futex-based hand-offs at around 1.3–1.9 µs. The figure varies with the CPU, kernel and measurement method, and it excludes the indirect cost of reloading caches after the switch. Even at the low end, a futex sleep and wake-up is orders of magnitude more expensive than the tens of nanoseconds a cache-resident atomic operation typically costs.

A note on tokio::sync::Mutex: this mutex does not park OS threads. Its lock().await yields the task while the lock is held elsewhere. The Tokio documentation explains that it exists mainly so a guard can be held across .await points, and that this makes it more expensive than a blocking mutex, so the blocking mutex should be preferred where it can be used. The thread-parking behavior discussed above applies to std::sync::Mutex (and equivalents such as parking_lot), not to the async mutex.

2. Cache-line ping-pong

When several cores repeatedly write to the same cache line, such as a lock word or a queue index, the line has to move between cores. Each write invalidates other cores’ copies, and each subsequent access by another core is slower as a result. This coherence traffic is a major cost of contended shared state. It is also why the layout of the queue’s metadata matters, which the next section addresses.

3. Head-of-line blocking across tenants

If all tenants share one queue and one lock, a spike from one tenant slows everyone else. The same is true if the queue has a single capacity limit: a flood from Tenant A can fill it and cause Tenant B’s webhooks to be rejected. Lock-free structures reduce the first problem but do nothing for the second, which is a queue-topology problem addressed in the multi-tenant section below.

 Worker 1 ---> [ holds lock ]   ---> [ lock line owned by Core 0 ]
 Worker 2 ---> [ spin, then futex_wait ] ---> [ sleeps in kernel ]
 Worker 3 ---> [ spin, then futex_wait ] ---> [ sleeps in kernel ]
                         |
                         v
        Throughput bounded by the one thread holding the lock

Queue Shapes: SPSC, SPMC and MPMC

Choose the queue by how many threads touch each end.

A common ingress design is one network I/O thread per core (optionally pinned with OS thread affinity), each pushing into its own queue, with worker threads draining those queues. Since each queue has exactly one producer, the producer side needs no compare-and-swap loop. Only the consumer side does, if more than one worker shares a queue. Pinning is an optimization to measure, not a requirement.

How a Bounded Lock-Free Ring Buffer Works

A bounded ring buffer is a fixed array of slots plus two monotonically increasing positions: where the next item will be enqueued and where the next item will be dequeued. A position maps to a slot with pos & (capacity - 1) when capacity is a power of two, which is cheaper than a modulo.

The design used below follows Dmitry Vyukov’s bounded MPMC queue, which is also the basis of crossbeam-queue’s ArrayQueue and the array flavor of crossbeam-channel. Each slot carries a sequence number (a “stamp”) that tells threads whether the slot is ready for a producer or for a consumer:

• Initially, slot i has sequence i.
• A producer at position pos may write when slot.sequence == pos. After writing, it stores pos + 1 with Release ordering.
• A consumer at position pos may read when slot.sequence == pos + 1. After reading, it stores pos + capacity with Release ordering, marking the slot free for the next lap.

Threads claim positions by advancing the shared enqueue or dequeue counter with compare_exchange, and publish data through the slot’s sequence number. No thread ever holds a lock, and a thread that loses a race simply retries with an updated position.

Cache-line padding

The enqueue and dequeue counters are written by different threads, so they must not share a cache line, or every enqueue would invalidate the line the consumers are polling. crossbeam_utils::CachePadded aligns and pads a value to avoid this.

The padding size is not always 64 bytes. According to the crossbeam-utils documentation, CachePadded assumes 128-byte cache lines on x86-64, aarch64 and powerpc64. On modern Intel CPUs the spatial prefetcher pulls pairs of 64-byte lines, so crossbeam pessimistically aligns to 128 bytes. It uses 32 bytes on arm, mips, mips64, sparc and hexagon, 256 on s390x, 16 on m68k, and 64 elsewhere. The crate itself notes that this is a reasonable guess and not a guarantee about the machine the program runs on.

Memory Orderings, Briefly

Every atomic operation in Rust takes an Ordering argument. The ones used here:

• Relaxed guarantees atomicity of the operation itself but no ordering relative to other memory accesses. It is suitable for counters and for reading a position you will later validate.
• Release (on a store) makes all of the thread’s earlier writes visible to any thread that later performs an Acquire load of the same atomic and observes the stored value. In the queue, the producer writes the payload first, then stores the sequence number with Release.
• Acquire (on a load) pairs with that Release. A consumer that observes the new sequence number is guaranteed to see the payload written before it.
• AcqRel applies both semantics to a read-modify-write operation such as compare_exchange.

In this queue, the Acquire/Release pair on each slot’s sequence number carries all of the data synchronization. The compare_exchange calls on the shared counters only arbitrate which thread owns a position, so Relaxed is sufficient for them. That is also how the reference algorithm and crossbeam’s implementations use them.

A Bounded MPMC Ring Buffer in Rust

The implementation below is a straightforward port of Vyukov’s algorithm. It is a teaching example and a starting point, not a substitute for a battle-tested crate. If you use it, run it under Miri and loom, and stress-test it. The sequence-number logic, the use of MaybeUninit and the unsafe blocks are exactly where subtle bugs hide.

use std::cell::UnsafeCell;
use std::mem::MaybeUninit;
use std::sync::atomic::{AtomicUsize, Ordering};

use crossbeam_utils::CachePadded;

struct Slot<T> {
    sequence: AtomicUsize,
    value: UnsafeCell<MaybeUninit<T>>,
}

/// Bounded multi-producer multi-consumer queue (Vyukov-style).
pub struct RingBuffer<T> {
    buffer: Box<[Slot<T>]>,
    mask: usize,
    enqueue_pos: CachePadded<AtomicUsize>, // advanced by producers
    dequeue_pos: CachePadded<AtomicUsize>, // advanced by consumers
}

// Slots are handed between threads only through the sequence protocol.
unsafe impl<T: Send> Sync for RingBuffer<T> {}

impl<T> RingBuffer<T> {
    /// `capacity` must be a power of two and at least 2.
    pub fn new(capacity: usize) -> Self {
        assert!(
            capacity >= 2 && capacity.is_power_of_two(),
            "capacity must be a power of two >= 2"
        );

        let buffer = (0..capacity)
            .map(|i| Slot {
                sequence: AtomicUsize::new(i),
                value: UnsafeCell::new(MaybeUninit::uninit()),
            })
            .collect::<Vec<_>>()
            .into_boxed_slice();

        Self {
            buffer,
            mask: capacity - 1,
            enqueue_pos: CachePadded::new(AtomicUsize::new(0)),
            dequeue_pos: CachePadded::new(AtomicUsize::new(0)),
        }
    }

    /// Try to enqueue. Returns the value back if the buffer is full.
    pub fn push(&self, value: T) -> Result<(), T> {
        let mut pos = self.enqueue_pos.load(Ordering::Relaxed);
        loop {
            let slot = &self.buffer[pos & self.mask];
            let seq = slot.sequence.load(Ordering::Acquire);
            let diff = seq.wrapping_sub(pos) as isize;

            if diff == 0 {
                // Slot is free for this position: try to claim it.
                match self.enqueue_pos.compare_exchange_weak(
                    pos,
                    pos.wrapping_add(1),
                    Ordering::Relaxed,
                    Ordering::Relaxed,
                ) {
                    Ok(_) => {
                        unsafe { (*slot.value.get()).write(value) };
                        // Publish the payload to consumers.
                        slot.sequence.store(pos.wrapping_add(1), Ordering::Release);
                        return Ok(());
                    }
                    Err(actual) => pos = actual, // lost the race; retry
                }
            } else if diff < 0 {
                // Slot still holds an unconsumed item from the previous lap.
                return Err(value);
            } else {
                // Another producer already claimed this position; reload.
                pos = self.enqueue_pos.load(Ordering::Relaxed);
            }
        }
    }

    /// Try to dequeue. Returns `None` if the buffer is empty.
    pub fn pop(&self) -> Option<T> {
        let mut pos = self.dequeue_pos.load(Ordering::Relaxed);
        loop {
            let slot = &self.buffer[pos & self.mask];
            let seq = slot.sequence.load(Ordering::Acquire);
            let diff = seq.wrapping_sub(pos.wrapping_add(1)) as isize;

            if diff == 0 {
                match self.dequeue_pos.compare_exchange_weak(
                    pos,
                    pos.wrapping_add(1),
                    Ordering::Relaxed,
                    Ordering::Relaxed,
                ) {
                    Ok(_) => {
                        let value = unsafe { (*slot.value.get()).assume_init_read() };
                        // Mark the slot free for the next lap.
                        slot.sequence
                            .store(pos.wrapping_add(self.mask + 1), Ordering::Release);
                        return Some(value);
                    }
                    Err(actual) => pos = actual,
                }
            } else if diff < 0 {
                // Producer has not published this slot yet: empty.
                return None;
            } else {
                pos = self.dequeue_pos.load(Ordering::Relaxed);
            }
        }
    }
}

impl<T> Drop for RingBuffer<T> {
    fn drop(&mut self) {
        // Drop any items still in the buffer.
        while self.pop().is_some() {}
    }
}

The single-producer variant

If exactly one thread ever calls push, the producer-side compare_exchange is unnecessary, because no other thread competes for the enqueue position. That is the SPMC case. Because the compiler cannot enforce “only one producer”, the method is unsafe and the caller takes on that contract:

impl<T> RingBuffer<T> {
    /// # Safety
    /// At most one thread may call this method (or `push`) at any time.
    pub unsafe fn push_single_producer(&self, value: T) -> Result<(), T> {
        let pos = self.enqueue_pos.load(Ordering::Relaxed);
        let slot = &self.buffer[pos & self.mask];

        // The slot is free only when its sequence equals the position.
        if slot.sequence.load(Ordering::Acquire) != pos {
            return Err(value); // full
        }

        (*slot.value.get()).write(value);
        slot.sequence.store(pos.wrapping_add(1), Ordering::Release);
        // Only this thread writes enqueue_pos, so a plain store suffices.
        self.enqueue_pos.store(pos.wrapping_add(1), Ordering::Relaxed);
        Ok(())
    }
}

Consumers still use the CAS-based pop, so multiple workers can safely share the queue. If you instead want one consumer per queue, you can build the mirror-image single-consumer pop.

Production Options: crossbeam-queue and crossbeam-channel

For most services, a maintained crate is a better choice than custom unsafe code.

crossbeam_queue::ArrayQueue

ArrayQueue is a bounded multi-producer multi-consumer queue that allocates a fixed-capacity buffer on construction. Pushing into a full queue fails and returns the element, and force_push lets it act as an overwriting ring buffer. Its implementation is based on Vyukov’s bounded MPMC queue, with CachePadded head and tail counters, the same design described above. For raw “push a payload, pop a payload” semantics with no blocking, it is the closest off-the-shelf equivalent of the code in this article.

use crossbeam_queue::ArrayQueue;

let q = ArrayQueue::new(1024);
assert!(q.push(42u64).is_ok());
assert_eq!(q.pop(), Some(42));

crossbeam-channel

crossbeam-channel is a full MPMC channel library. Its bounded channels use an array flavor that is, again, based on Vyukov’s bounded MPMC queue and preallocates its buffer. Unlike a bare queue, it supports blocking send/recv and select!. Blocking operations can put threads to sleep, so on a latency-critical ingress path use the non-blocking try_send and try_recv.

Two points of terminology to keep straight: crossbeam-channel is MPMC (it has no dedicated SPMC type), and “lock-free” describes its fast path. The blocking paths still involve waker bookkeeping and thread parking when a channel is empty or full.

Reference ingress architecture

        HTTP clients          I/O threads (one per core)       Bounded queues          Workers
   +-----------------+      +----------------------------+    +----------------+    +-----------+
   | Tenant A reqs   | ---> | I/O thread 0 (parse, auth) | -> | Queue 0 (SPMC) | -> | Worker 0  |
   | Tenant B reqs   | ---> | I/O thread 1 (parse, auth) | -> | Queue 1 (SPMC) | -> | Worker 1  |
   +-----------------+      +----------------------------+    +----------------+    +-----------+
                                    try_push, never blocks         pop (CAS)

Non-blocking dispatch with crossbeam-channel

use crossbeam_channel::{bounded, Receiver, Sender, TrySendError};

#[derive(Debug)]
pub struct WebhookPayload {
    pub tenant_id: String,
    pub body: Vec<u8>,
    pub received_at_epoch_ms: u64,
}

#[derive(Debug)]
pub enum IngressError {
    QueueFull,
    Offline,
}

pub struct WebhookIngress {
    tx: Sender<WebhookPayload>,
}

impl WebhookIngress {
    pub fn new(capacity: usize) -> (Self, Receiver<WebhookPayload>) {
        let (tx, rx) = bounded(capacity);
        (Self { tx }, rx)
    }

    /// Never blocks: a full queue becomes backpressure (e.g. HTTP 429/503).
    pub fn handle_incoming(&self, payload: WebhookPayload) -> Result<(), IngressError> {
        self.tx.try_send(payload).map_err(|e| match e {
            TrySendError::Full(_) => IngressError::QueueFull,
            TrySendError::Disconnected(_) => IngressError::Offline,
        })
    }
}

A note on the Vec<u8> and String fields: each payload owns heap allocations made when the request was parsed. The queue itself does not allocate per message, but the messages do. If allocation shows up in profiles, consider pooled buffers or bytes::Bytes.

What about tokio::sync::mpsc?

If your producers and consumers are both Tokio tasks, tokio::sync::mpsc is usually the right default. It stores elements in blocks organized as a linked list, reuses emptied blocks where it can, and gives you async backpressure for bounded channels. It is a multi-producer, single-consumer channel, so it does not fit a shared worker pool without a dispatcher task. It is designed for async-to-async communication, not for the thread-to-thread hand-off this article focuses on. A reasonable decision rule:

• Async tasks on both ends: tokio::sync::mpsc.
• OS threads on both ends, latency-critical: ArrayQueue or a bounded crossbeam-channel with try_send.
• Async producer, blocking consumer pool: bridge deliberately and measure, since the Tokio documentation itself discusses mixing sync and async channel endpoints.

Multi-Tenant Isolation

A fast queue does not fix noisy neighbors. If Tenant A fills a shared queue, Tenant B’s webhooks are rejected no matter how efficient the queue is. Fairness has to be designed in at the queue-topology and admission-control level.

1. Separate queues per tier or per shard

Give each service tier its own bounded queue and its own workers, so a free-tier flood cannot consume enterprise capacity.

pub struct TieredRouter {
    enterprise: crossbeam_channel::Sender<WebhookPayload>,
    standard: crossbeam_channel::Sender<WebhookPayload>,
}

impl TieredRouter {
    pub fn route(&self, tier: &str, payload: WebhookPayload) -> Result<(), IngressError> {
        let tx = match tier {
            "enterprise" => &self.enterprise,
            _ => &self.standard,
        };
        tx.try_send(payload).map_err(|e| match e {
            TrySendError::Full(_) => IngressError::QueueFull,
            TrySendError::Disconnected(_) => IngressError::Offline,
        })
    }
}

For very large tenant counts, per-tenant queues become unwieldy. Hash tenants onto a fixed number of shards instead, and give the largest tenants dedicated shards.

2. Rate limit before the queue

Shed load at the edge, before a payload consumes queue capacity. A token bucket can be implemented with atomics, for example by packing a token count and a timestamp into a single AtomicU64 and updating it with a compare-exchange loop. Per-tenant limits keep a spike from reaching the shared buffers at all. Return a clear status code (such as 429 Too Many Requests) so well-behaved senders back off and retry.

3. Reserve capacity for high-priority traffic

Pair tiered queues with worker assignment: dedicate some workers exclusively to the enterprise queue, and let the remaining workers drain the standard queue. Work stealing between tiers is possible but reintroduces the cross-tenant contention you are trying to avoid.

Measure Before You Commit

Lock-free is not automatically faster. A lock-free queue under heavy contention still suffers cache-line transfers, and CAS retry loops can burn CPU. Whether a mutex-protected VecDeque, crossbeam-channel, ArrayQueue or a custom queue wins depends on payload size, the number of producers and consumers, core topology (including NUMA), and how bursty the traffic is. Some practical guidance:

• Benchmark with realistic producer/consumer counts and payload sizes, using a harness such as Criterion.rs. Report tail latencies (p99, p99.9), not just averages.
• Run the benchmark on the hardware you deploy on. Cache-line sizes, core counts and interconnects differ between x86 and ARM servers.
• Use perf (for example perf stat and perf c2c) to look for cache-line contention and context-switch counts in the real process.
• Test correctness separately: use Miri for undefined behavior and loom for exhaustive interleaving checks of any custom lock-free code.
• Compare against the simplest baseline first. An uncontended or lightly contended Mutex is cheap, and a design with sharded queues and few shared structures may beat any single lock-free queue.

Conclusion and Checklist

At very high request rates, a single contended mutex can limit how many webhooks you can ingest, mostly because of futex parking, wake-ups and cache-line transfers. A bounded lock-free ring buffer based on per-slot sequence numbers avoids the locking and keeps the hot path to a few atomic operations, but it only works well if you also handle layout, backpressure and tenant fairness.

1. Pad shared cursors. Wrap the enqueue and dequeue counters in crossbeam_utils::CachePadded, and do not hard-code 64 bytes: crossbeam assumes 128 on x86-64 and aarch64.
2. Use power-of-two capacities so that index wrapping is a bitwise AND, not a modulo.
3. Prefer proven implementations. crossbeam_queue::ArrayQueue or a bounded crossbeam-channel covers most needs. Write custom queues only when profiling justifies it, and test them with Miri and loom.
4. Never block at ingress. Use try_send or push, and turn a full queue into explicit backpressure (HTTP 429 or 503) rather than parking I/O threads.
5. Isolate tenants structurally with tiered or sharded queues and rate limiting before the queue.
6. Match the tool to the runtime. Use tokio::sync::mpsc between async tasks, and lock-free thread queues between OS threads.
7. Measure tail latency on your own hardware before and after every change.