Ingress Architecture & Low-Level Performance: Zero-Copy Pipelines, Adaptive Backpressure, and Shared-Memory Buffering for Webhook Gateways

High-volume webhook ingress is an asymmetric systems problem. A typical north-south API gateway routes a synchronous request to a local service and discards its state. A webhook relay has a different job. It must accept untrusted, bursty HTTP payloads from external producers such as Stripe, GitHub, Shopify, or Twilio, and verify their cryptographic signatures. It must also record the event somewhere safe before returning a 2xx, then dispatch the payload asynchronously to downstream consumers.

The senders’ own documentation shows why the acknowledgment path matters. GitHub expects a 2xx within 10 seconds and recommends queueing payloads for background processing. Shopify allows 1 second to connect and 5 seconds in total. Stripe’s exact timeout isn’t publicly documented, and its guidance is to verify the signature and return a 2xx before running slow logic. A relay that stalls on its own downstream dependencies pushes that latency straight back onto the sender.

At high request rates, per-request overheads that are invisible at 1,000 RPS start to dominate. These include redundant kernel-to-user memory copies, per-request heap allocation, syscall transitions, and naive 503 responses that invite retry traffic from senders that retry. This guide covers three low-level patterns for high-throughput ingress on modern Linux (kernel 6.1 or newer; examples assume a current LTS such as 6.6 or 6.12):

1. Zero-copy ingestion with Go splice(2) and io_uring: reducing user-space copies and syscalls between the socket and storage.
2. Adaptive backpressure with Little’s Law and a token bucket: shaping admission during downstream degradation without provoking avoidable retries.
3. RAM-constrained buffering with tmpfs and shared memory: a memory-mapped ring buffer that survives process crashes, with explicit caveats about what it does not survive.

Throughput figures in this article are arithmetic examples, not benchmarks. Measure on your own hardware before relying on any of them.

1. Zero-Copy Webhook Ingress with Go splice(2) and io_uring

The Anatomy of User-Space Copying Overhead

In a conventional Go net/http handler, persisting a request body to a write-ahead log (WAL) or forwarding it to a broker involves several steps:

1. NIC to kernel socket buffer: the NIC DMAs incoming packets into kernel memory, where the TCP stack assembles them into socket buffers (sk_buff). This step involves no CPU copy.
2. Kernel to user space (read(2)): the Go runtime, woken by epoll, calls read, which copies the payload into a []byte on the Go heap.
3. User-space processing: the application parses headers, validates the signature, and allocates buffers.
4. User space to kernel (write(2)): the data is copied again, either into the page cache for a file or into another socket buffer for a downstream connection.

At 100,000 webhooks per second with 16 KB payloads, each CPU copy moves 1.6 GB/s, so the two copies in steps 2 and 4 move about 3.2 GB/s of memory traffic. The per-request []byte allocations also add garbage-collection work. Go’s collector is concurrent and its pauses are usually short, but allocation rate still costs CPU and can widen tail latency under load.

Using splice(2) from Go

The Linux splice(2) call moves data between a file descriptor and a pipe without copying it through user space. Go’s standard library already uses it: io.Copy between two *net.TCPConn values takes the splice path automatically. A webhook gateway, however, rarely proxies blindly. It must read headers such as X-Hub-Signature-256 or Stripe-Signature, find the body length, and separate the body from the HTTP framing before routing it.

Two caveats apply before the code:

• What “zero-copy” means here. Splicing from a socket into a pipe moves page references. Splicing from a pipe into a regular file or tmpfs still copies the data once, inside the kernel, into the page cache. You save the kernel-to-user copy and the syscall round trips, not every copy. Socket-to-socket splicing can avoid the CPU copy entirely.
• Protocol constraints. This approach fits HTTP/1.1 with a known Content-Length. Chunked encoding and HTTP/2 framing require user-space parsing, and you must handle any body bytes that the HTTP parser has already buffered. In practice this means hijacking the connection (http.Hijacker) or writing a custom HTTP/1.1 front end.

The function below transfers an exact byte count from a TCP connection to a destination file descriptor through an anonymous pipe. SPLICE_F_MOVE is deliberately omitted, because it has been a no-op since Linux 2.6.21.

package ingress

import (
	"fmt"
	"net"

	"golang.org/x/sys/unix"
)

// SplicePayload transfers exactly byteCount bytes from an incoming TCP connection
// to a destination file descriptor through a kernel pipe, avoiding user-space copies.
// dstFd should be a regular file or other descriptor that never returns EAGAIN
// on splice; for non-blocking sockets, drive the write side through rawConn.Write.
func SplicePayload(srcConn *net.TCPConn, dstFd int, byteCount int64) (int64, error) {
	rawConn, err := srcConn.SyscallConn()
	if err != nil {
		return 0, fmt.Errorf("failed to acquire raw conn: %w", err)
	}

	var pipeFds [2]int
	if err := unix.Pipe2(pipeFds[:], unix.O_CLOEXEC|unix.O_NONBLOCK); err != nil {
		return 0, fmt.Errorf("pipe2 failed: %w", err)
	}
	defer unix.Close(pipeFds[0])
	defer unix.Close(pipeFds[1])

	// Raise pipe capacity to 1 MiB (the default unprivileged ceiling,
	// /proc/sys/fs/pipe-max-size) to reduce the number of splice calls per payload.
	_, _ = unix.FcntlInt(uintptr(pipeFds[1]), unix.F_SETPIPE_SZ, 1<<20)

	var total int64
	var spliceErr error

	// Running inside rawConn.Read lets Go's netpoller manage readiness.
	err = rawConn.Read(func(srcFd uintptr) bool {
		for total < byteCount {
			remaining := byteCount - total

			// Socket -> pipe: moves page references, no copy into user space.
			n, err := unix.Splice(int(srcFd), nil, pipeFds[1], nil,
				int(remaining), unix.SPLICE_F_NONBLOCK)
			if err != nil {
				if err == unix.EAGAIN {
					return false // yield to the netpoller until readable
				}
				spliceErr = err
				return true
			}
			if n == 0 {
				spliceErr = fmt.Errorf("unexpected EOF after %d/%d bytes", total, byteCount)
				return true
			}

			// Pipe -> destination fd.
			var drained int64
			for drained < n {
				w, wErr := unix.Splice(pipeFds[0], nil, dstFd, nil,
					int(n-drained), 0)
				if wErr != nil {
					if wErr == unix.EINTR {
						continue
					}
					spliceErr = wErr
					return true
				}
				drained += w
			}
			total += drained
		}
		return true
	})

	if err != nil {
		return total, err
	}
	return total, spliceErr
}

// DuplicatePipePages uses tee(2) to duplicate pipe contents into a second pipe
// without consuming the source, so one copy of the page references can be hashed
// while the other continues on to storage. Both descriptors must be pipes.
func DuplicatePipePages(srcPipeFd, dstPipeFd, length int) (int64, error) {
	return unix.Tee(srcPipeFd, dstPipeFd, length, unix.SPLICE_F_NONBLOCK)
}

Verifying Signatures Without Pulling Bytes into User Space

Webhook ingress must validate an HMAC (GitHub’s X-Hub-Signature-256, Shopify’s X-Shopify-Hmac-Sha256, Stripe’s Stripe-Signature) over the exact raw body before acknowledging. If the body goes straight from socket to storage, how do you compute the digest? There are two approaches.

1. Kernel crypto API (AF_ALG) plus tee(2). Linux exposes hashing through AF_ALG sockets, including hmac(sha256). tee(2) duplicates pipe page references into a second pipe without copying data. One pipe can be spliced into the AF_ALG operation socket to compute the digest, while the other is spliced to storage. Some senders sign more than the body (Stripe signs a timestamp prefix followed by the payload), so feed the prefix into the hash socket first. Because the bytes reach storage before the digest is known, treat the stored frame as uncommitted until verification succeeds.
2. Verify from a memory-mapped buffer in user space. If verification needs user-space logic, such as multi-tenant key lookup or RFC 9421 HTTP Message Signatures, splice the body into a tmpfs-backed file (Section 3) that the verifier has already mapped with mmap(2). The verifier hashes the mapped bytes directly, with no read(2) copy. This still reads the memory, but it avoids an extra copy and allocation.

Reducing Syscall Overhead with io_uring

Even without copies, high-concurrency ingress pays for syscall transitions, and mitigations for speculative-execution vulnerabilities can make each transition more expensive on some CPUs. io_uring lets you queue many operations through shared-memory rings and amortize or avoid those transitions.

Calling io_uring directly from Go is awkward. Go’s M:N scheduler moves goroutines between OS threads, which conflicts with ring-per-thread designs and with IORING_SETUP_SINGLE_ISSUER. A common approach is to put the network and storage data plane in a small Rust (or C) component and keep Go as the control plane. The two share state through memory-mapped rings or a narrow C ABI.

These io_uring features matter most for ingress. Kernel versions are the minimum for each:

• Multishot accept (5.19): one submission entry keeps accepting connections and posts a completion for each. This covers TCP accept only. TLS handshakes are a separate matter.
• Provided buffer rings (IORING_REGISTER_PBUF_RING, 5.19): the application registers a ring of buffers and the kernel picks one for each receive (IOSQE_BUFFER_SELECT), returning the buffer ID in the completion. This supersedes the older IORING_OP_PROVIDE_BUFFERS mechanism.
• Multishot receive (6.0): one receive request keeps producing completions as data arrives, each consuming a buffer from the ring.
• Submission queue polling (IORING_SETUP_SQPOLL): a kernel thread polls the submission queue so the application can submit work without a syscall while the thread is awake. The polling thread consumes a CPU core, and it sleeps after an idle timeout, after which submission needs a wakeup call.
• IORING_SETUP_SINGLE_ISSUER and IORING_SETUP_DEFER_TASKRUN (6.0 and 6.1): these reduce overhead for a ring driven by a single thread.

// Rust io_uring ingress ring: multishot receive with provided buffers.
// The buffer ring itself must be registered beforehand (IORING_REGISTER_PBUF_RING;
// registration code not shown) under buffer group ID `bgid`.
use io_uring::{opcode, types, IoUring};
use std::os::unix::io::RawFd;

pub struct IngressRing {
    ring: IoUring,
    bgid: u16, // buffer group ID for kernel-managed buffer selection
}

impl IngressRing {
    pub fn new(entries: u32, bgid: u16) -> std::io::Result<Self> {
        let ring = IoUring::builder()
            .setup_sqpoll(1000) // SQ thread idles out after 1000 ms without submissions
            .build(entries)?;
        Ok(Self { ring, bgid })
    }

    /// Arms a multishot receive on an accepted webhook sender socket.
    /// The kernel picks buffers from the registered ring for each completion.
    pub unsafe fn arm_multishot_recv(&mut self, client_fd: RawFd, conn_token: u64) -> std::io::Result<()> {
        let recv_e = opcode::RecvMulti::new(types::Fd(client_fd), self.bgid)
            .build()
            .user_data(conn_token);

        self.ring
            .submission()
            .push(&recv_e)
            .map_err(|_| std::io::Error::new(std::io::ErrorKind::Other, "SQ full"))?;

        // With SQPOLL, submit() only needs a wakeup syscall if the SQ thread is asleep.
        self.ring.submit()?;
        Ok(())
    }
}

SINGLE_ISSUER is omitted from this snippet because its rules about which thread may submit are easy to violate in a hybrid Go/Rust layout. Add it only once ownership of the ring is unambiguous.

Truly Zero-Copy Receive: io_uring zcrx

Provided buffers keep data out of the Go heap, but the kernel still copies each packet payload into your buffer. Linux 6.15 added io_uring zero-copy receive (zcrx), which removes that kernel-to-user copy so payloads land directly in application memory. It has demanding prerequisites: the NIC must support header/data split, and you must configure flow steering and RSS so that only the intended flows reach a hardware queue bound to the io_uring instance. The NIC is not configured by the kernel API, so this setup is done out of band. It also requires SINGLE_ISSUER, DEFER_TASKRUN, and 32-byte completion entries. Treat zcrx as a specialized option for dedicated ingress hosts, not a default.

Architecture Comparison

Production note on TLS. Webhook traffic is HTTPS, and user-space TLS (crypto/tls or OpenSSL) decrypts in user space, which defeats splice. Kernel TLS (kTLS) moves record encryption and decryption into the kernel after a user-space handshake. Once active, splice and io_uring see plaintext. Decryption is still CPU work, so it is not “free,” and Go’s crypto/tls does not natively support kTLS, so you would need a TLS library that does (OpenSSL 3.x can). Many deployments terminate TLS at a load balancer or proxy in front of the ingress, which sidesteps the issue.

2. Adaptive Backpressure with a Token Bucket and Little’s Law

Why Naive Throttling Can Trigger Retry Storms

When a downstream database, Kafka cluster, or tenant destination slows, an unthrottled gateway exhausts connection pools and memory. The reflex is to return 503 or 429 to everyone. Whether that helps or hurts depends on how each sender treats a failure, and the senders differ significantly:

• Stripe retries failed deliveries with exponential backoff for up to three days in live mode. Any non-2xx response triggers a retry, and in sandbox mode it retries three times over a few hours. Stripe does not publish its exact schedule or timeout.
• Shopify allows 1 second to connect and 5 seconds for the full response. It retries up to 8 times over 4 hours, and its documentation says subscriptions can be removed if failures persist.
• GitHub requires a 2xx within 10 seconds and does not automatically redeliver failed deliveries. Redelivery is manual or scripted through its API.

These differences have two consequences. For senders that retry, rejections come back later, on a schedule you don’t control, on top of fresh traffic. Retries from many failed events can cluster in time, especially when a sender’s backoff has little jitter, so recovery can be slower than the outage that caused it. For senders that don’t retry, such as GitHub, a rejection is not a delay but a loss until someone redelivers. A uniform shedding policy therefore treats very different senders as equals.

The strategy is to absorb short latency spikes by delaying the acknowledgment within the sender’s timeout, rather than rejecting immediately. At the same time, cap concurrency using Little’s Law and modulate admission with a feedback-driven token bucket.

Modeling Capacity with Little’s Law

Little’s Law states that, in a stable system, the average number of items in the system (L) equals the average arrival rate (λ) times the average time each item spends in the system (W):

L = λ × W

For a webhook gateway, L is in-flight requests awaiting persistence, λ is accepted webhooks per second, and W is the time from receipt to durable commit.

Suppose a node handles λ = 20,000 RPS with W = 5 ms. The in-flight count is 20,000 × 0.005 = 100. If storage degrades and W rises to 250 ms while the gateway keeps admitting 20,000 RPS, the in-flight requests needed become 20,000 × 0.25 = 5,000, which is 50 times higher. That surge can exhaust connection pools and lock capacity, which pushes W higher still, a feedback loop that collapses throughput.

Inverting the relationship shows what a fixed concurrency cap implies: sustainable throughput is L / W. With a cap of 100:

A gateway that enforces the cap keeps the database from being overloaded. It does not, by itself, tell you what to do with the arrivals that exceed the sustainable rate, which is the job of the hold queue below.

An Adaptive Concurrency Limit

Rather than hard-coding the cap, you can adapt it from observed latency. Netflix’s open-source concurrency-limits library popularized gradient-based limiters. Its Gradient2 variant compares a long-term exponentially smoothed latency to the current latency:

gradient  = clamp(tolerance × W_long / W_current, 0.5, 1.0)
new_limit = gradient × limit + queue_allowance
limit     = limit × (1 − smoothing) + new_limit × smoothing

In that library the defaults are a tolerance of 1.5 and a smoothing factor of 0.2. queue_allowance is a small headroom term that lets the limit keep growing while latency is healthy. Its exact form (a constant or a function of the limit, such as a square root) depends on version and configuration. The implementation below is a variant that uses a slowly decaying minimum latency as its baseline. Two details deserve emphasis:

• The 0.5 floor on the gradient is a per-step bound, not a limit floor. Repeated steps at that floor will drive the limit toward the configured minimum, so set a sensible minimum limit.
• Smoothing and tolerance matter. Without them the limit reacts to every latency blip. Tolerance keeps mild jitter from shrinking the limit, and smoothing slows both shrink and growth.

Coupling the Limit to a Token Bucket and a Timeout-Aware Hold Queue

Little’s Law bounds in-flight concurrency, while senders produce arrivals at their own rate. An adaptive token bucket bridges the two. Its refill rate follows the controller’s estimate:

refill_rate = limit / W_observed

When a request arrives and no token is available, the gateway does not reject immediately. It places the request in a bounded hold queue. The hold budget must stay well inside the sender’s timeout, including time for TLS, parsing, and persistence. A 2.5-second budget leaves headroom under Shopify’s 5-second total limit and GitHub’s 10-second limit. Stripe’s timeout is undocumented, so budget conservatively there.

1. Absorb micro-bursts. If a token frees up within the budget, the gateway persists the webhook and returns 200. The sender just saw slightly higher latency, and no retry occurs.
2. Shed retries before first attempts, but only for senders that retry. Identify repeat deliveries by an identifier you have already seen: the event id in a Stripe payload, X-Shopify-Webhook-Id (which Shopify keeps the same across retries), or X-GitHub-Delivery. Under severe pressure, shed likely retries first, since the sender will try again later. Do not apply the same logic to a sender that never retries on its own, such as GitHub, where a shed request may be lost.
3. Bound the queue itself. Held requests consume sockets and memory. Cap the number held, and reject beyond the cap.
4. Shed with a jittered Retry-After. Only when the hold budget is exhausted, return 429 with a randomized Retry-After. Treat the header as a hint, since you can’t assume every sender honors it, and make sure your own downstream recovery doesn’t depend on it.

package backpressure

import (
	"context"
	"math"
	"math/rand/v2"
	"sync"
	"time"
)

// AdaptiveController couples a latency-gradient concurrency limit
// with a dynamic token bucket and a timeout-aware hold budget.
type AdaptiveController struct {
	mu sync.Mutex

	// Concurrency-limit state
	minLatencyMs  float64 // slowly decaying baseline latency
	ewmaLatencyMs float64 // exponentially weighted observed latency
	inFlight      int64   // current in-flight requests
	limit         float64 // adaptive concurrency limit
	minLimit      float64 // floor, to prevent total starvation

	// Token bucket state
	tokens         float64
	bucketCapacity float64
	refillRate     float64 // tokens per second (limit / observed latency)
	lastRefill     time.Time

	// Must stay well below the sender's timeout (e.g. 2.5s for a 5s sender timeout)
	maxHoldBudget time.Duration
}

const (
	rttTolerance = 1.5 // latency growth tolerated before the limit shrinks
	smoothing    = 0.2 // how quickly the limit moves toward its target
)

func NewAdaptiveController(initialLimit, minLimit float64, holdBudget time.Duration) *AdaptiveController {
	return &AdaptiveController{
		minLatencyMs:   5.0,
		ewmaLatencyMs:  5.0,
		limit:          initialLimit,
		minLimit:       minLimit,
		tokens:         initialLimit,
		bucketCapacity: initialLimit * 1.5,
		refillRate:     initialLimit / 0.005, // 200 req/s per concurrency unit at 5 ms
		lastRefill:     time.Now(),
		maxHoldBudget:  holdBudget,
	}
}

// Acquire waits up to maxHoldBudget for admission. On success it returns a release
// callback that records persistence latency. On failure it returns a Retry-After
// value in seconds and an error.
func (c *AdaptiveController) Acquire(ctx context.Context, isRetry bool) (func(success bool), int, error) {
	deadline := time.Now().Add(c.maxHoldBudget)

	for {
		c.mu.Lock()
		c.refillLocked()

		// Likely retries need extra headroom when the bucket is low, which
		// reserves the last tokens for first-attempt traffic.
		requiredTokens := 1.0
		if isRetry && c.tokens < c.bucketCapacity*0.20 {
			requiredTokens = c.bucketCapacity * 0.25
		}

		if c.tokens >= requiredTokens && float64(c.inFlight) < c.limit {
			c.tokens -= 1.0
			c.inFlight++
			start := time.Now()
			c.mu.Unlock()

			return func(success bool) {
				latencyMs := float64(time.Since(start).Microseconds()) / 1000.0
				c.recordCompletion(latencyMs, success)
			}, 0, nil
		}

		// Estimate time until enough tokens exist; poll at a 2 ms minimum
		// (this also covers the case where the in-flight limit, not tokens, is binding).
		deficit := requiredTokens - c.tokens
		waitSec := math.Max(deficit, 0) / math.Max(c.refillRate, 1.0)
		waitDur := time.Duration(waitSec * float64(time.Second))
		if waitDur < 2*time.Millisecond {
			waitDur = 2 * time.Millisecond
		}
		c.mu.Unlock()

		// If waiting would exceed the hold budget, shed with a jittered Retry-After.
		if time.Now().Add(waitDur).After(deadline) {
			return nil, c.computeJitteredRetryAfter(), context.DeadlineExceeded
		}

		select {
		case <-ctx.Done():
			return nil, 1, ctx.Err()
		case <-time.After(waitDur):
		}
	}
}

func (c *AdaptiveController) refillLocked() {
	now := time.Now()
	elapsed := now.Sub(c.lastRefill).Seconds()
	if elapsed <= 0 {
		return
	}
	c.tokens = math.Min(c.bucketCapacity, c.tokens+elapsed*c.refillRate)
	c.lastRefill = now
}

func (c *AdaptiveController) recordCompletion(latencyMs float64, success bool) {
	c.mu.Lock()
	defer c.mu.Unlock()

	c.inFlight--
	if latencyMs <= 0 {
		latencyMs = 0.1
	}

	// Exponentially weighted observed latency (alpha = 0.1).
	c.ewmaLatencyMs = 0.9*c.ewmaLatencyMs + 0.1*latencyMs

	// Baseline: track new minimums immediately, drift upward slowly otherwise
	// so the baseline can adapt to structural shifts.
	if latencyMs < c.minLatencyMs {
		c.minLatencyMs = latencyMs
	} else {
		c.minLatencyMs = c.minLatencyMs*0.999 + latencyMs*0.001
	}

	gradient := rttTolerance * c.minLatencyMs / c.ewmaLatencyMs
	if !success {
		gradient *= 0.7 // extra multiplicative decrease on storage errors
	}
	gradient = math.Max(0.5, math.Min(1.0, gradient))

	target := c.limit*gradient + math.Sqrt(c.limit)
	newLimit := c.limit*(1-smoothing) + target*smoothing
	c.limit = math.Max(c.minLimit, math.Min(2000.0, newLimit))

	// Token refill rate follows Little's Law: rate = L / W.
	obsSec := math.Max(c.ewmaLatencyMs/1000.0, 0.001)
	c.refillRate = c.limit / obsSec
	c.bucketCapacity = math.Max(10.0, c.limit*1.5)
}

func (c *AdaptiveController) computeJitteredRetryAfter() int {
	c.mu.Lock()
	congestionRatio := c.ewmaLatencyMs / math.Max(c.minLatencyMs, 1.0)
	c.mu.Unlock()

	// Base grows with congestion (2s to 15s); add 1-5s of uniform jitter.
	base := int(math.Min(15.0, math.Max(2.0, congestionRatio)))
	return base + rand.IntN(5) + 1
}

What to Expect During a Database Degradation

The exact numbers depend on your tolerance, smoothing, limit floor, and workload. Load-test the controller against a simulated slow database before trusting any set of constants.

3. RAM-Constrained Webhook Buffering with tmpfs and Shared Memory

The Crash-Persistence Dilemma

To keep acknowledgment latency low, a gateway can buffer payloads in a local queue and let an asynchronous drainer flush batches to durable storage (S3, Kafka, PostgreSQL). Keeping that queue in process memory (a Go channel, a slice, a Rust VecDeque) means a panic, an OOM kill, or a binary upgrade loses everything not yet drained.

Writing every request synchronously to local storage with fsync(2) avoids that, but costs a storage round trip per request. The cost varies enormously by device, from tens of microseconds on drives with power-loss protection to milliseconds on others. Under sustained load it also generates significant write volume.

What tmpfs Gives You, and What It Doesn’t

A file on tmpfs (including files under /dev/shm) lives in kernel-managed memory, not in your process’s address space. That gives a useful middle ground:

1. Crash independence. If the process is killed by SIGKILL, a segfault, or the OOM killer, the file stays in the kernel. A restarted process can reopen it, mmap(2) it with MAP_SHARED, read the ring’s indices, and resume draining.
2. No block-device latency. Writes go to RAM, with no fsync stall.
3. What it does not survive. tmpfs is cleared on reboot and unmount, so kernel panics, power loss, and node failure all lose it. If swap is enabled, tmpfs pages can also be swapped out, so disable swap on ingress nodes or lock the mapping with mlock.

This is a durability trade-off you must state explicitly. Acknowledging a webhook once it is in a tmpfs ring means you have promised the sender something that a node failure can break. For senders that retry, that promise is only broken if the node dies before the drainer flushes. For senders that don’t (GitHub), the event is gone unless you redeliver. If you can’t accept that, replicate to a second node or use a durable log before acknowledging. Otherwise, pair the design with reconciliation: Stripe, Shopify, and GitHub all provide APIs or tooling to resend or re-list recent events, within limited windows.

Container platforms add constraints:

• Docker gives containers a small default /dev/shm (64 MB), so size it deliberately with --shm-size or a dedicated mount.
• Kubernetes memory-backed emptyDir volumes (medium: Memory) are tmpfs. Their contents count against the container’s memory limit, and set a sizeLimit to avoid an unbounded volume triggering an OOM kill. The data survives a container restart within the same pod but is deleted with the pod.

Step 1: Provision a Bounded tmpfs Mount

Mount a dedicated volume with an explicit size= ceiling, and pre-allocate the ring file with fallocate(2) at startup so that memory is reserved up front. Without that, a write could hit an out-of-memory condition later and raise SIGBUS on a mapped page.

# Dedicated 1 GiB RAM-backed volume for webhook ingress queuing
mount -t tmpfs -o size=1024M,nr_inodes=64,mode=0700,noatime,nodiratime tmpfs /mnt/webhook_shm

Step 2: Memory Layout of the Ring Buffer

The file starts with a 64-byte header, followed by a power-of-two data region (for example 512 MiB). Frames are variable-length and 8-byte aligned.

+-----------------------------------------------------------------------+
| Offset 0x00 (8B): Magic Number (0x574852494E473031 = "WHRING01")      |
| Offset 0x08 (8B): Ring Data Capacity (power of 2)                     |
| Offset 0x10 (8B): Write Reserve Index (atomic uint64, producers)      |
| Offset 0x18 (8B): Write Commit Index  (atomic uint64, consumer limit) |
| Offset 0x20 (8B): Read Consumer Index (atomic uint64, drainer)        |
| Offset 0x28 (24B): Padding to a full 64-byte header                   |
+-----------------------------------------------------------------------+
| Offset 0x40 ... (Capacity bytes): circular variable-length frames     |
| [Len:4][Flags:4][CRC32C:4][Reserved:4][TimestampNs:8][Payload...]     |
+-----------------------------------------------------------------------+

The header fills exactly one cache line. All three indices share it, so heavily contended deployments may want to move them onto separate cache lines to avoid false sharing between producers and the consumer.

The design separates a reserve index from a commit index:

• Producers claim a contiguous region by advancing Write Reserve Index with compare-and-swap (CAS).
• Only after a producer has written the frame (payload, CRC32C, flags) does it advance Write Commit Index. Commits happen in reservation order, so the consumer never sees a gap.
• Crash recovery: after a crash, any region between commit and reserve was never published. The restarted process sets Write Reserve Index = Write Commit Index, discarding partial frames. Every frame between Read Consumer Index and Write Commit Index is intact. The CRC32C guards against corruption or bugs, not against torn writes, which the commit index already excludes.

Step 3: Go Implementation of the mmap-Backed Ring

The producer side is shown below. A consumer (the drainer) reads frames from readIdx up to writeCommit, verifies each CRC, skips padding frames, writes to durable storage, and only then advances readIdx. A padding frame fills the unused tail of the buffer when a frame won’t fit before the physical end, which is why every frame header begins with a length and flags that fit in 8 bytes.

package shmring

import (
	"encoding/binary"
	"errors"
	"fmt"
	"hash/crc32"
	"os"
	"runtime"
	"sync/atomic"
	"unsafe"

	"golang.org/x/sys/unix"
)

const (
	ringMagic       uint64 = 0x574852494E473031 // "WHRING01"
	headerSize             = 64
	frameHeaderSize        = 24 // [Len:4][Flags:4][CRC32C:4][Reserved:4][TimestampNs:8]
	flagCommitted   uint32 = 1 << 0
	flagWrapPadding uint32 = 1 << 1
)

var (
	ErrRingFull      = errors.New("shared memory ring buffer full")
	ErrRingEmpty     = errors.New("shared memory ring buffer empty")
	ErrCorrupt       = errors.New("frame CRC32C checksum mismatch")
	ErrFrameTooLarge = errors.New("frame larger than ring capacity")

	// Castagnoli uses hardware CRC instructions on amd64 (SSE4.2) and arm64 in Go's stdlib.
	crcTable = crc32.MakeTable(crc32.Castagnoli)
)

// ringHeader maps onto the first 64 bytes of the tmpfs file.
type ringHeader struct {
	magic       uint64
	capacity    uint64
	writeResv   uint64 // claimed by producers via CAS
	writeCommit uint64 // boundary visible to the consumer
	readIdx     uint64 // consumed and flushed to cold storage
	_padding    [24]byte
}

type ShmRing struct {
	file   *os.File
	mmap   []byte
	header *ringHeader
	data   []byte
	mask   uint64
}

// OpenOrRecover maps a tmpfs-backed ring file. If a compatible ring already exists
// (for example after a process crash), uncommitted reservations are discarded and
// committed frames are preserved. A magic or capacity mismatch reinitializes the ring,
// discarding its contents.
func OpenOrRecover(path string, capacityPow2 uint64) (*ShmRing, error) {
	if capacityPow2 == 0 || (capacityPow2&(capacityPow2-1)) != 0 {
		return nil, fmt.Errorf("capacity must be a power of 2, got %d", capacityPow2)
	}

	totalSize := int64(headerSize + capacityPow2)
	f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0600)
	if err != nil {
		return nil, err
	}

	// Reserve tmpfs pages up front so a later write can't fail with SIGBUS.
	if err := unix.Fallocate(int(f.Fd()), 0, 0, totalSize); err != nil {
		_ = f.Close()
		return nil, fmt.Errorf("fallocate failed: %w", err)
	}

	mem, err := unix.Mmap(int(f.Fd()), 0, int(totalSize),
		unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED)
	if err != nil {
		_ = f.Close()
		return nil, fmt.Errorf("mmap failed: %w", err)
	}

	hdr := (*ringHeader)(unsafe.Pointer(&mem[0]))
	ring := &ShmRing{
		file:   f,
		mmap:   mem,
		header: hdr,
		data:   mem[headerSize:],
		mask:   capacityPow2 - 1,
	}

	if atomic.LoadUint64(&hdr.magic) == ringMagic && atomic.LoadUint64(&hdr.capacity) == capacityPow2 {
		// Recovery: drop uncommitted in-flight reservations.
		atomic.StoreUint64(&hdr.writeResv, atomic.LoadUint64(&hdr.writeCommit))
	} else {
		atomic.StoreUint64(&hdr.capacity, capacityPow2)
		atomic.StoreUint64(&hdr.writeResv, 0)
		atomic.StoreUint64(&hdr.writeCommit, 0)
		atomic.StoreUint64(&hdr.readIdx, 0)
		atomic.StoreUint64(&hdr.magic, ringMagic)
	}

	return ring, nil
}

// Enqueue appends one webhook payload to the ring.
func (r *ShmRing) Enqueue(payload []byte, timestampNs uint64) error {
	payloadLen := uint64(len(payload))
	alignedFrameLen := align8(frameHeaderSize + payloadLen)
	capacity := r.header.capacity
	if alignedFrameLen > capacity {
		return ErrFrameTooLarge
	}

	for {
		resv := atomic.LoadUint64(&r.header.writeResv)
		read := atomic.LoadUint64(&r.header.readIdx)

		if (resv-read)+alignedFrameLen > capacity {
			return ErrRingFull
		}

		offset := resv & r.mask
		if offset+alignedFrameLen > capacity {
			// Frame would cross the physical end: claim padding plus the frame together.
			padLen := capacity - offset // always a multiple of 8, at least 8
			if (resv-read)+padLen+alignedFrameLen > capacity {
				return ErrRingFull
			}
			if !atomic.CompareAndSwapUint64(&r.header.writeResv, resv, resv+padLen+alignedFrameLen) {
				continue
			}
			// Padding marker: total bytes to skip, then the wrap flag.
			binary.LittleEndian.PutUint32(r.data[offset:offset+4], uint32(padLen))
			binary.LittleEndian.PutUint32(r.data[offset+4:offset+8], flagWrapPadding)
			r.advanceCommit(resv, resv+padLen)
			resv += padLen
			offset = 0
		} else {
			if !atomic.CompareAndSwapUint64(&r.header.writeResv, resv, resv+alignedFrameLen) {
				continue
			}
		}

		frame := r.data[offset : offset+alignedFrameLen]
		binary.LittleEndian.PutUint32(frame[0:4], uint32(payloadLen))
		binary.LittleEndian.PutUint32(frame[4:8], flagCommitted)
		binary.LittleEndian.PutUint32(frame[8:12], crc32.Checksum(payload, crcTable))
		binary.LittleEndian.PutUint32(frame[12:16], 0)
		binary.LittleEndian.PutUint64(frame[16:24], timestampNs)
		copy(frame[frameHeaderSize:], payload)

		// Publish once all earlier reservations have committed.
		r.advanceCommit(resv, resv+alignedFrameLen)
		return nil
	}
}

// advanceCommit publishes in reservation order, spinning (briefly) until
// earlier producers have committed.
func (r *ShmRing) advanceCommit(expectedOld, nextNew uint64) {
	for !atomic.CompareAndSwapUint64(&r.header.writeCommit, expectedOld, nextNew) {
		runtime.Gosched()
	}
}

func align8(n uint64) uint64 { return (n + 7) &^ 7 }

One operational caution: because commits are strictly ordered, a producer that stalls between reserving and committing blocks every later producer. Keep the reserve-to-commit window tiny (copy and checksum only, with no network calls), and bound payload sizes.

Buffering Tiers Compared

Latencies below are orders of magnitude, not measurements. They vary by hardware, payload size, and contention, so benchmark your own stack.

Putting It All Together

The three pieces compose into one ingress path:

1. Efficient network ingestion. Terminate TLS with kTLS (or in front of the ingress). Receive through a Rust io_uring provided-buffer ring, or a Go RawConn + splice pipeline for fixed-length HTTP/1.1 bodies, keeping payload bytes off the Go heap.
2. Little’s Law admission control. Before reserving buffer space, requests pass the adaptive controller. If the drainer falls behind or storage latency rises, the concurrency limit narrows and the token bucket slows admission. Short spikes are absorbed inside the hold budget so senders still get a 200. Sustained overload is shed, with retries (for senders that retry) shed first.
3. Crash-resilient ring. Admitted payloads are verified and committed to the pre-allocated tmpfs ring with a CRC32C. An independent drainer batches frames into durable storage and advances the read index only after the flush succeeds. If the ingress process restarts, committed-but-undrained frames are still there.

This pipeline is crash-safe, but it is not power-safe. Acknowledging from RAM means a node failure can lose events you have already acknowledged. Decide up front whether that is acceptable. If it isn’t, add replication or a durable log, and add reconciliation against the senders’ own event APIs.