Bypassing the Linux Socket Layer: High-Throughput DPDK, AF\_XDP, and kTLS Architectures for Multi-Tenant Webhook Gateways
Build extreme high-throughput webhook gateways using DPDK, AFXDP ring architectures, and kernel-space kTLS zero-copy decryption to eliminate CPU saturation and kernel crossings.

Bypassing the Linux Socket Layer: High-Throughput DPDK, AF_XDP, and kTLS Architectures for Multi-Tenant Webhook Gateways
Introduction: The Webhook Ingestion Bottleneck
Modern event-driven architectures rely heavily on webhooks. When a payment processor, code host, or IoT fleet emits events in bursts, a multi-tenant webhook gateway sits at the edge of your infrastructure. It absorbs unpredictable HTTP/1.1, HTTP/2, and sometimes HTTP/3 traffic from thousands of senders.
At tens of thousands of requests per second, the standard Linux socket layer (epoll, recv(), send()) paired with a user-space TLS library such as OpenSSL or Rustls works well. Past several hundred thousand requests per second, or tens of millions of packets per second (PPS), per-packet and per-connection CPU costs start to dominate. Modern 100GbE and 400GbE NICs can carry the bandwidth, so the limit is rarely the wire. It is CPU time spent per packet:
- Interrupt and softirq load. Each RX queue raises MSI-X interrupts, and the kernel then does the work in
softirqcontext. NAPI switches drivers to polling under load and interrupt coalescing batches events, but a busy core still spends much of its time in softirq rather than your application. - Per-packet metadata and copies. On the standard path the kernel allocates an
sk_buff(a couple hundred bytes of metadata plus shared info) for every packet. It then walks the packet through Netfilter, routing, and the TCP state machine, and copies payload bytes to user space onrecv(). - Syscall and mode-switch cost. Each transition between user and kernel mode is cheap in isolation, but on CPUs that need mitigations such as page-table isolation it adds up at millions of operations per second. Batching (
io_uring) reduces it. - TLS work. Bulk AES-GCM or ChaCha20-Poly1305 decryption consumes CPU cycles and memory bandwidth. New connections add asymmetric handshake costs (ECDHE key exchange, certificate signing).
A caveat before going further: most webhook gateways do not need kernel bypass. SO_REUSEPORT, io_uring, kTLS, and an XDP-based front line for filtering get you a long way while keeping the Linux tooling you already use. Cloudflare’s experience is instructive. It began exploring XDP in 2016-2017 and shipped its first production XDP deployment (the l4drop filter) in November 2018. XDP replaced its previous DDoS mitigation system, which relied on a proprietary kernel-bypass stack and created a single-vendor dependency. This guide covers what changes at the extreme end, and where each technique helps and where it does not.
1. Where the CPU Cycles Go in the Standard Stack
When a packet arrives, the NIC uses Direct Memory Access (DMA) to write the frame into a pre-posted RX ring buffer in host memory. The driver then builds an sk_buff and hands it up the stack. Netfilter hooks (iptables/nftables), conntrack, routing, and TCP processing each touch different cache lines. If the application thread that reads the socket runs on a different NUMA node from the NIC’s PCIe slot, those memory accesses cross the inter-socket interconnect (UPI on Intel, Infinity Fabric on AMD).
The work a webhook gateway actually performs is modest: terminate TLS, parse HTTP headers, identify the tenant, verify an HMAC signature, and push the payload to an internal bus such as Kafka, Redpanda, or NATS. Almost everything else in the general-purpose stack is overhead for this job.
A webhook request is also not one packet. A typical JSON payload of a few kilobytes spans several packets, plus ACKs, plus TLS records. One million requests per second therefore means several million PPS, and in practice TLS and HTTP processing, not packet I/O, often becomes the bottleneck first. Profile before choosing an architecture.
2. Kernel-Bypass Architectures: DPDK vs. AF_XDP
Two main approaches exist for getting packets to user space without the full socket path: DPDK (Data Plane Development Kit) and AF_XDP (Address Family eXpress Data Path). Both rely on shared-memory ring buffers and DMA, but they trade off isolation, operations, and ecosystem differently.
DPDK: User-Space Poll Mode Drivers
DPDK runs its own Poll Mode Drivers (PMDs) in user space. Dedicated cores spin on the NIC’s RX descriptor rings instead of waiting for interrupts.
- Device binding. For many NICs (for example Intel’s), the device is unbound from its kernel driver and bound to
vfio-pci. Some NVIDIA/Mellanox NICs differ: themlx5PMD works alongside the kernel driver (a “bifurcated” model), so the NIC stays visible to the kernel and Linux tooling. - Hugepages. DPDK allocates packet buffers (
rte_mbuf) from 2 MB or 1 GB hugepages, which reduces TLB pressure. - No TCP/IP stack. DPDK delivers raw frames. For TCP-based protocols you need a user-space stack. Options include F-Stack (the FreeBSD stack ported to DPDK), Seastar’s native stack, and VPP’s host stack. Each adds integration and operational cost.
AF_XDP: Fast Path with Kernel Coexistence
AF_XDP was added in Linux 4.18. It lets an XDP program redirect selected frames into a user-space socket (an XSK) while the NIC stays under kernel control.
- Steering with XDP. An eBPF program attached at the driver’s XDP hook runs before
sk_buffallocation (in native/driver mode). It can pass non-webhook traffic (SSH, ARP, ICMP, BGP) to the kernel withXDP_PASSand redirect selected flows into anXSKMAPwithXDP_REDIRECT. - UMEM. User space registers a contiguous memory region (UMEM) divided into fixed-size chunks. The chunk size must be a power of two, at least 2048 bytes in the common aligned mode, and 2048 or 4096 bytes is typical.
- Zero-copy mode. With
XDP_ZEROCOPYand a supporting driver (for example Inteli40e/ice/ixgbeor NVIDIAmlx5), the NIC DMAs frames directly into UMEM chunks. If the driver lacks support, you fall back toXDP_COPYmode, which still bypasses much of the stack but copies each frame once. - Wakeups and busy polling. The
XDP_USE_NEED_WAKEUPbind flag (kernel 5.3 and later) lets the kernel signal when the application must issue a syscall to make progress.SO_PREFER_BUSY_POLL(kernel 5.11 and later) allows tighter control over busy polling.
DPDK also ships an AF_XDP PMD, so DPDK applications can run on top of AF_XDP sockets.
Comparison
| Dimension | DPDK (native PMDs) | AF_XDP |
|---|---|---|
| Kernel involvement | Driver runs in user space. For many NICs the device is unbound from the kernel; mlx5 is bifurcated. | XDP hook runs in the kernel driver; the OS keeps the control plane. |
| TCP/IP stack | None; requires a user-space stack. | Selected traffic bypasses the kernel stack. The rest can use the normal kernel stack via XDP_PASS. |
| CPU model | Dedicated, busy-polling cores. | Busy-polling or interrupt/wakeup-driven. |
| Tooling | Standard tools (tcpdump, ss) do not see traffic on unbound devices; needs DPDK-specific telemetry. | Works with standard Linux tooling and eBPF maps. |
| Raw performance | Generally the highest, because descriptors go from the NIC ring straight to the application. | Typically lower than native PMDs. The DPDK developer list has a report of roughly 10 Mpps with AF_XDP versus roughly 20 Mpps with a native driver in a drop test on one NIC. |
| Operations | Hugepages, core isolation, custom deployment. | Closer to normal Linux operations. |
Treat any published Mpps figure as hardware- and workload-specific, and benchmark with your own packet sizes and application logic. Packet-drop microbenchmarks say little about a gateway that also parses HTTP and terminates TLS.
3. DMA Ring Buffers and Multi-Queue RSS
AF_XDP’s Four SPSC Rings
AF_XDP uses four single-producer/single-consumer rings in shared memory:
- Fill ring (user to kernel). The application supplies addresses of empty UMEM chunks for incoming packets.
- RX ring (kernel to user). The kernel posts descriptors (address and length) of received frames.
- TX ring (user to kernel). The application posts frames to send, such as ACKs or HTTP
202 Acceptedresponses in a user-space TCP design. - Completion ring (kernel to user). The kernel returns descriptors of transmitted frames so the application can reuse the chunks.
Each ring has one producer and one consumer, so synchronization uses memory barriers and head/tail indices rather than locks.
RSS and Queue Affinity
To scale across cores, NICs use Receive Side Scaling (RSS). The NIC computes a Toeplitz hash over the packet’s flow fields (typically the IP and TCP/UDP 4-tuple) and uses an indirection table to select an RX queue. Packets of one connection then consistently land on the same queue.
For a gateway:
- Bind one AF_XDP socket (or DPDK queue) per RX queue, and pin one worker thread per queue to a core.
- Choose cores on the same NUMA node as the NIC’s PCIe slot.
- Set the number of queues with
ethtool -L, tune the indirection table withethtool -X, and pin IRQs manually (stopirqbalancefrom moving them). - If you stay on the kernel TCP stack (Topology A below), the application threads that accept and read connections need to align with the queue that receives them. Linux offers
SO_REUSEPORT(with a BPF steering program) andSO_INCOMING_CPUfor this. Alignment is not automatic.
For tenant isolation, NICs with flow steering (ntuple filters via ethtool -N, Intel Flow Director, or NVIDIA ConnectX steering rules) can direct a premium tenant’s dedicated IP/port to a dedicated queue and core. This limits noisy-neighbor effects at the hardware level, but it only works if tenants can be distinguished by L3/L4 fields. Tenants sharing one hostname and IP cannot be separated this way. They need isolation at the application layer, such as per-tenant rate limits and queues.
4. Integrating Polling Drivers with Async Rust
Rust suits this work well. Ownership and Drop make it natural to model “who owns this buffer right now”, and the ecosystem includes aya (eBPF), xsk-rs (AF_XDP bindings), and several io_uring runtimes.
The main design clash is scheduling. Tokio’s multi-threaded runtime moves tasks between worker threads, which fights thread-pinned rings and cache locality.
Thread-Per-Core Runtimes
Use a thread-per-core, shared-nothing design. glommio and monoio are io_uring-based runtimes built around this model. You can also run one single-threaded Tokio runtime (current_thread) per pinned OS thread. In each thread:
- Pin the thread to a core.
- Own one AF_XDP socket bound to the matching queue, plus its own UMEM and buffer pool.
- Avoid cross-thread shared state. If threads must communicate, use bounded SPSC or MPSC channels.
Wrapping UMEM Frames Safely
Ownership of a UMEM chunk passes from the kernel to the application when its descriptor appears on the RX ring, and back when you place the address on the Fill ring. A Drop implementation can make that return automatic. In a thread-per-core design, Rc is appropriate because the frame never crosses threads, and Rc makes the type !Send to enforce it:
use std::{cell::RefCell, ptr::NonNull, rc::Rc};
/// Thin wrapper over the AF_XDP fill-ring producer for this queue.
pub struct FillQueue { /* ring producer state */ }
impl FillQueue {
pub fn push(&mut self, _addr: u64) { /* write addr, bump producer index (batched) */ }
}
/// A received frame inside UMEM. The application owns the chunk from the
/// moment its descriptor is read from the RX ring until it is dropped.
pub struct UmemFrame {
addr: u64,
ptr: NonNull<u8>,
len: usize,
fill: Rc<RefCell<FillQueue>>, // Rc => !Send, matching thread-per-core
}
impl UmemFrame {
pub fn payload(&self) -> &[u8] {
// SAFETY: ptr/len describe a chunk inside the registered UMEM that we
// own until Drop returns it to the fill ring.
unsafe { std::slice::from_raw_parts(self.ptr.as_ptr(), self.len) }
}
}
impl Drop for UmemFrame {
fn drop(&mut self) {
self.fill.borrow_mut().push(self.addr); // recycle the chunk
}
}
In production you would batch Fill-ring updates rather than publishing per frame, and decide how frames that must outlive the RX batch (for example, TCP segments waiting for reassembly) are tracked.
Adaptive Polling
A pure busy-poll loop burns a core even when traffic is low. With XDP_USE_NEED_WAKEUP, the kernel sets the XDP_RING_NEED_WAKEUP flag when the application must make a syscall (such as poll() or sendto()) to restart driver processing. Under load the application can process batches of frames without syscalls. When the rings empty, it can fall back to blocking in poll() or io_uring. The right batch size and the switching thresholds depend on your latency goals, so measure them.
5. Offloading TLS with Linux kTLS
Bypassing the socket layer addresses packet I/O. TLS termination is a separate cost.
In a conventional design the kernel hands ciphertext to user space, the TLS library decrypts it, and the application processes the plaintext. Kernel TLS (kTLS) moves the record layer into the kernel while the handshake stays in user space.
How kTLS Works
- Handshake in user space. Your TLS library (OpenSSL 3.0+ built with kTLS support, or
rustlsvia thektlscrate) performs the handshake: key exchange, certificate validation, ALPN. - Keys handed to the kernel. After the handshake, the application enables the
tlsupper-layer protocol on the TCP socket and installs the session keys, IV, and record sequence number withsetsockopt(SOL_TLS, TLS_TX / TLS_RX, ...). The kernel then encrypts and decrypts records onsend()andrecv(). - Version and cipher support. TX support landed in Linux 4.13 and RX in 4.17. TLS 1.3 is supported from 5.1, with AES-GCM-128/256. ChaCha20-Poly1305 arrived in 5.11. Check your kernel and
/proc/net/tls_statto confirm what your build supports. - Zero-copy options. TX can use
sendfile()andsplice(), and kernel 5.19 added a TX zero-copy option. For TLS 1.3 receive,TLS_RX_EXPECT_NO_PAD(kernel 6.0) allows decrypting directly into user buffers when the peer does not pad records. - Key updates. A TLS 1.3
KeyUpdateafter the handshake is a post-handshake record the kernel hands back to user space. Support for installing the new keys has been added in recent kernels (reported as 6.14). Many user-space libraries do not handle re-keying after a kTLS attach, so long-lived connections need care. - Kernel module. The
tlsmodule must be available (modprobe tls). Containers use the host kernel’s module.
What kTLS Does Not Do for a Webhook Gateway
- It does not remove the need to read the payload. Verifying an HMAC signature (for example GitHub’s
X-Hub-Signature-256) and routing by tenant require the headers and body in user space. kTLS removes the user-space decryption step and a user-space copy of ciphertext. It does not mean plaintext never reaches your application.splice()forwarding without inspection only suits plain pass-through proxying, and it cannot help once you terminate HTTP/2, because h2 framing must be parsed. - It does not apply to QUIC. HTTP/3 over QUIC encrypts at the packet level inside user-space transport libraries, so kTLS does not help there.
- It requires the kernel TCP stack. kTLS is a TCP upper-layer protocol. Packets diverted into AF_XDP or DPDK never reach the kernel TCP stack, so kTLS cannot be used on them. This shapes the topologies below.
NIC-Level TLS Offload
The kernel documents three modes:
TLS_SW: the CPU does the crypto.TLS_HW: packet-based NIC offload. The kernel still handles TCP and record framing, and the NIC performs the crypto per packet, provided the packets arrive in order. For RX, if the device decrypted all segments of a record, software decryption is skipped. Otherwise the software path decrypts. This mode integrates with the normal stack and is enabled viaethtoolfeaturestls-hw-tx-offloadandtls-hw-rx-offload.TLS_HW_RECORD: full TCP offload, where NIC firmware replaces the kernel’s TCP handling. The kernel documentation says it is not usable in production environments that rely on the Linux networking stack (for example firewalling or QoS).
In-tree support for TLS_HW comes from specific NICs, notably NVIDIA ConnectX-6 Dx and later. Check your NIC’s driver documentation rather than assuming support. The best-known large-scale example is Netflix’s FreeBSD-based video servers. They use NIC kTLS offload on ConnectX-6 Dx hardware and reached roughly 400 Gb/s per server, with prototype work toward 800 Gb/s. That is a TX-heavy, sendfile-based workload serving large objects, which differs from a webhook ingress path of many small encrypted requests. The gain on your workload may be much smaller, so measure.
Using kTLS from Rust
The ktls crate (now maintained under the rustls organization) configures kTLS for tokio-rustls connections. The server needs enable_secret_extraction = true, a negotiated cipher suite the kernel supports, and the TCP stream wrapped in a CorkStream before the handshake. The example below follows the 6.x API. Pre-release 7.x versions move to a builder-style Setup API, so check the crate documentation for your version.
use ktls::{CorkStream, KtlsStream};
use tokio::net::TcpStream;
use tokio_rustls::TlsAcceptor;
pub async fn upgrade_to_ktls(
tcp: TcpStream,
acceptor: &TlsAcceptor, // built from a ServerConfig with enable_secret_extraction = true
) -> Result<KtlsStream<TcpStream>, Box<dyn std::error::Error>> {
// 1. Handshake in user space over a CorkStream.
let tls = acceptor.accept(CorkStream::new(tcp)).await?;
// 2. Extract the negotiated secrets and install them in the kernel
// (setsockopt SOL_TLS TLS_TX / TLS_RX). Fails if the `tls` module is
// missing or the negotiated cipher suite is unsupported.
Ok(ktls::config_ktls_server(tls).await?)
}
This works with Tokio. A community fork (ktls2) lists optional monoio support. If you adopt glommio or monoio, check TLS and kTLS integration before committing to the runtime.
Handshake Churn
kTLS addresses bulk crypto, not handshakes. Senders that open a new connection per event can still produce thousands of handshakes per second. Mitigations:
- Separate handshake capacity. Run handshakes on dedicated cores or threads so asymmetric math does not starve packet processing.
- Session resumption. Support TLS 1.3 PSK resumption with rotated session-ticket keys shared across the cluster. Do not enable 0-RTT early data for webhook endpoints. Early data is replayable, and webhook POSTs are generally not idempotent. Use 1-RTT resumption only.
- Encourage connection reuse. HTTP keep-alive and HTTP/2 connection pooling on the sender side reduce handshakes the most, but you do not control sender behavior.
- Crypto accelerators. Intel QuickAssist Technology (QAT) can offload asymmetric operations such as RSA, ECDSA, and ECDH through OpenSSL’s QAT engine. This applies to OpenSSL-based termination, not to
rustls. Weigh the gain against the added complexity for ECDSA P-256, which is already fast on modern CPUs.
6. End-to-End Gateway Blueprints
Because kTLS requires the kernel TCP stack and AF_XDP/DPDK bypass it, the two do not combine on the same traffic. Pick one of two topologies.
Topology A: XDP Front Line + Kernel TCP + kTLS + io_uring (the Practical Default)
- XDP for filtering and cheap early decisions. An XDP program drops malformed or obviously hostile packets and enforces coarse per-source rate limits with eBPF maps (for example LRU hash maps), then returns
XDP_PASSfor legitimate traffic. This is the pattern Cloudflare uses for itsl4dropDDoS mitigation, and Meta’s open-source Katran uses XDP for L4 load balancing. Per-tenant limits generally belong in the application, because L3/L4 fields rarely identify a tenant. - Kernel TCP with kTLS. Connections terminate in the kernel TCP stack. Handshakes happen in user space, and bulk record encryption and decryption happen in the kernel, optionally on a
TLS_HW-capable NIC. - Thread-per-core
io_uringserver. Each pinned thread accepts on its ownSO_REUSEPORTlistener and receives data viaio_uring. Multishotrecv(kernel 6.0 and later) with provided buffer rings (5.19 and later; prefer these over the olderIORING_OP_PROVIDE_BUFFERS) lets one submission yield many completions, so syscall overhead stays low. - Application logic. The thread parses HTTP, identifies the tenant (for example via a path, hostname, or custom header), verifies the HMAC, and forwards the event to the message bus.
Tenant isolation here comes from per-tenant queues, quotas, and concurrency limits, plus optional steering of premium tenants to separate listeners or IPs.
Topology B: Full User-Space Data Plane (DPDK or AF_XDP)
Consider this only when profiling shows the kernel stack itself is the limit and you can afford the engineering cost.
- Ingress. DPDK PMDs or AF_XDP zero-copy deliver raw frames to per-core rings.
- User-space TCP and TLS. Each core runs a user-space TCP stack (F-Stack or VPP via FFI,
smoltcp, or a custom stack) and a user-space TLS library such as BoringSSL orpicotls. Decrypt records in place in packet buffers where the library’s API allows it. You lose kTLS and its NIC offload path, so symmetric crypto runs on the CPU, with AES-NI and similar instructions. - Egress. Decrypted payloads go to local sidecar agents over shared-memory rings, or onward via the user-space stack.
You also give up kernel tooling, firewalling, and conntrack for this traffic, and you own TCP behavior, including retransmission, congestion control, and edge cases such as odd middleboxes. This topology is common for specialized load balancers and packet processors, and much less common for HTTP application gateways.
Conclusion
Scaling a webhook gateway to millions of requests per second means treating the kernel as a programmable data plane, not a fixed cost. For most teams the best return comes from Topology A: XDP for early filtering, RSS-aligned thread-per-core design, io_uring with multishot receive and provided buffers, and kTLS (with NIC offload where the hardware supports it). Reserve DPDK and AF_XDP data planes for workloads where profiling shows the kernel TCP path is the limit, accepting that you lose kTLS and take on a user-space TCP stack. In every case, measure with your own request sizes, TLS mix, and handshake rates, because published packet-rate numbers rarely transfer to an HTTPS application workload.
Sources and Further Reading
- Linux kernel documentation: Kernel TLS and Kernel TLS offload
- DPDK documentation: AF_XDP Poll Mode Driver
- DPDK developer list: AF_XDP performance discussion (2023)
- Cloudflare: L4Drop: XDP DDoS mitigations and eBPF replatforming, part 1
- Netflix and kTLS NIC offload: The Next Platform
io_uring: multishot receive and provided buffer rings- Rust:
ktlscrate