Advanced Infrastructure & Distributed Systems: Architectural Deep Dive into High-Performance Webhook Ingress and Durable Execution

Webhooks started as simple fire-and-forget HTTP POST requests. They are now a backbone of event-driven architectures, and the major providers deliver them at very large volume. Stripe, for example, is reported to send billions of them. Stateless handlers that do real work inside the request tend to fail under spikes, slow downstream dependencies, oversized payloads, noisy tenants, and memory-allocation pressure.

A resilient webhook ingestion tier needs more than a basic HTTP router. This guide covers four engineering areas: pairing webhooks with durable execution engines, compressing large payloads, isolating tenants with rate limiting, and reducing heap allocations with zero-copy parsing in Go and Rust.

1. Webhooks and Durable Execution: Managing State and Callbacks

The Timeout Problem

Webhook providers expect a fast acknowledgment. Published or commonly cited limits include:

• Shopify: a five-second timeout, with guidance to respond with a 200 OK within one to two seconds.
• GitHub: a ten-second window (GitHub Enterprise Server historically kept 30 seconds).
• Stripe: about 20 seconds, according to Svix’s webhook reliability guide.

If the endpoint misses the window, the delivery is treated as failed and retried. Real integrations often need minutes or days: calling downstream services, waiting for human approval, waiting for a second asynchronous callback, or riding out a third-party outage. Doing that work inside the request handler invites duplicate processing and lost events.

The standard pattern has three parts: verify the signature, acknowledge immediately, and hand the work to something durable. Durable execution engines are one strong option for that last part.

Integrating Durable Execution Engines

Engines such as Temporal, Inngest, and AWS Step Functions persist workflow state so that long-running processes can pause and resume. Temporal does this by recording an event history that it replays after failures. Step Functions provides managed state machines with callback support. Inngest offers hosted webhook sources that turn incoming requests into events that trigger durable functions.

[External Webhook Sender]
        │ (HTTP POST)
        ▼
[Ingress Gateway / Edge Proxy]
        │ (Verifies signature, starts or signals workflow, returns 2xx)
        ▼
[Durable Engine (Temporal / Inngest / Step Functions)]
        │
        ├─────────────────────────────┐
        ▼                             ▼
[Downstream Service A]     [Waits for External Callback]

1. Verify first, then dispatch. Compute the HMAC over the raw request body and compare it in constant time before parsing anything. Providers differ in the details: GitHub prefixes its signature header with sha256=, Stripe signs a timestamped payload, and Shopify sends a base64-encoded HMAC-SHA256 header. After verification, start or signal a workflow and return a 2xx without running business logic inline.
2. Derive idempotency from the provider’s event ID. Most providers include one: Stripe sends an event id, GitHub sends an X-GitHub-Delivery header, and Shopify sends X-Shopify-Webhook-Id. Using that value as the workflow ID makes retried deliveries safe to accept. In Temporal, Signal-With-Start atomically starts a workflow if none is running and delivers a signal either way. Workflow ID reuse policies control whether a closed workflow’s ID can be used again. The default allows it.
3. Survive downstream failures. The engine records progress durably, so a crashed worker or an unavailable dependency does not lose the webhook. The workflow resumes where it left off, with retries governed by policies you configure.
4. Handle asynchronous callbacks natively. Temporal workflows can block on a condition until a Signal changes workflow state. In Python this is await workflow.wait_condition(...), and in the TypeScript SDK it is await condition(...). Step Functions offers the callback pattern: a task invoked with the .waitForTaskToken suffix pauses until something calls SendTaskSuccess or SendTaskFailure with that token. A second webhook arriving at a dedicated ingress endpoint can supply that call.

Deduplication has limits. Temporal documents that Signal deduplication is not guaranteed in every case, such as during shard migration. Its built-in deduplication also does not span Continue-As-New boundaries. Workflows that must be strictly idempotent should carry their own idempotency key in the signal payload and check it in workflow code.

2. Payload Compression for High-Volume Webhooks: Brotli, Zstandard, and Edge Offloading

The Payload Bloat Problem

Most webhook payloads are small, but bulk exports and batched telemetry can be much larger. Large uncompressed JSON consumes bandwidth, raises egress costs, and increases parsing and queueing load. Compression helps, with one caveat: only the sender decides whether to compress a webhook body. Many providers send uncompressed JSON. Compression therefore applies most clearly to senders you control, such as internal producers, partner integrations built to your spec, and hops between your own services.

Brotli vs. Zstandard

• Brotli was developed by Google for HTTP content. It ships with a built-in static dictionary tuned for web content, and at high quality levels it usually achieves better compression ratios on text than zstd. Its compression speed drops sharply at the top levels, so very high quality settings suit pre-compressed static assets better than live traffic.
• Zstandard (zstd) was created by Yann Collet at Facebook (now Meta) and is specified in RFC 8878. Its design goal is a wide speed and ratio range with fast decompression, and decompression speed stays roughly constant across compression levels. Its level 3 is the default. Zstd also supports trained dictionaries, which can improve ratios on many small, similarly shaped messages. That is a common situation for webhook streams.

Independent comparisons show a consistent trade-off rather than a single winner. In one 2024 test on an 8.5 MB JavaScript file, brotli compressed 4–22% better than the zstd levels tested, while zstd decompressed 50–80% faster. Another comparison found zstd far faster than brotli at compression with a somewhat lower ratio. Results vary with content, levels, and hardware, so benchmark your own payloads with your own tooling rather than relying on a generic table.

A reasonable rule of thumb follows from this. Prefer zstd for dynamic, latency-sensitive streams where CPU time matters. Consider brotli for content compressed once and read many times.

HTTP Negotiation and Edge Offloading

Accept-Encoding is a response negotiation header. It tells a server which encodings the client can decode in the response. For request bodies, the sender declares the encoding with Content-Encoding, and a server that does not support it can respond with 415 Unsupported Media Type, advertising the encodings it does accept in an Accept-Encoding header. Webhook receivers should therefore:

• Document which Content-Encoding values they accept, such as gzip and zstd.
• Enforce a maximum decompressed size to defend against decompression bombs.
• Verify signatures against the exact bytes the provider specifies. Check the provider’s documentation, because some sign the raw body and others sign a canonical form.

Browser support for zstd content encoding arrived in 2024, with Chrome 123 and Firefox 126 as the first versions. Safari lagged at the time of the sources reviewed, so check current support when serving zstd to browsers. This does not affect server-to-server webhook traffic.

Proxies such as Envoy include request and response compression and decompression filters, and NGINX supports response compression through modules. Confirm which algorithms your specific version supports before designing around them. Keeping compressed data compressed through your internal brokers is also a design choice, not a default. Kafka, for example, supports producer-side compression with codecs including gzip, snappy, lz4, and zstd, and RabbitMQ leaves payload compression to the application.

3. Isolating Noisy Neighbors in Multi-Tenant Webhook Ingress: Token Bucket vs. Concurrency Limiting

The Multi-Tenant Challenge

A shared webhook gateway serves many tenants through common queues, connection pools, and workers. One tenant that suddenly floods the system with events, whether through a misconfiguration, a backfill, or a retry storm, can degrade service for everyone else. Providers commonly see exactly this: a burst of activity or a backfill can deliver thousands of events in seconds, and inline processing ties each one to an open HTTP request.

Why IP-Based Limits Fall Short

Many organizations egress through shared NAT gateways or cloud address pools, so one IP address can represent several unrelated tenants. Where possible, identify the tenant from something authenticated or verifiable, such as a per-tenant endpoint URL, an API key, or a verified signature secret. Rate-limit on that identity. Keep coarse IP-based limits only as an outer layer of defense.

Token Bucket

A token bucket refills at a fixed rate up to a maximum capacity. Each request consumes tokens, which allows short bursts while enforcing an average rate.

• Strengths: Simple, predictable, and burst-tolerant. In distributed deployments it is commonly implemented with atomic Redis Lua scripts or similar atomic operations.
• Weaknesses: It counts requests, not how long they take. A tenant sending few but expensive requests can still tie up workers.

Concurrency Limiting

Concurrency limits cap the number of in-flight requests or jobs per tenant.

• Strengths: It protects workers and database connections directly, and it self-adjusts when downstream systems slow down, because slow requests hold their slots longer.
• Weaknesses: It needs shared state, such as a distributed counter or lease, so leaked slots from crashed workers must be reclaimed with timeouts.

A Two-Tier Approach

A common and defensible design combines both:

1. Edge rate limit (token bucket): reject floods early and cheaply, with a clear 429 Too Many Requests and, where appropriate, a Retry-After header.
2. Worker concurrency limit: enforce per-tenant and per-tier caps at the dispatch layer so no tenant monopolizes processing capacity.

Pairing this with a durable queue means that rejected or delayed work is retried by the sender or buffered by you, depending on your design, instead of silently dropped. Per-tenant queues, or fair scheduling across tenants, add a further layer of isolation.

4. Zero-Copy Parsing: Reducing Allocations in Go and Rust Ingress Services

Why Allocation Matters

Conventional JSON decoding allocates strings, maps, slices, and interface values for every document. At high request rates this creates work for the allocator and, in garbage-collected runtimes, for the collector.

Go’s garbage collector is concurrent, and its stop-the-world phases are typically short. The more common cost is the CPU spent on allocation and collection, along with increased tail latency under load, rather than long pauses. In Rust there is no collector, but heavy allocation still costs CPU time and can hurt cache locality.

What Zero-Copy Means

Zero-copy parsing avoids duplicating data that already exists in memory. A parser produces slices or references into the original byte buffer instead of allocating new strings. It has real limits: escaped strings, for example, cannot always be borrowed directly because they need unescaping into new memory. And the input buffer must outlive every reference into it.

Go: Targeted Extraction Instead of Full Decoding

The standard library’s encoding/json decodes through reflection and allocates freely. Two commonly used alternatives:

• simdjson-go is a Go port of simdjson that uses SIMD instructions. Its maintainers report it is about 10x faster than encoding/json on their benchmark set, and about 40–60% of the speed of the original C++ simdjson. It requires a CPU with AVX2 and CLMUL support.
• sonic (ByteDance) uses JIT compilation and SIMD, and also supports lazily reading a single path from a document without fully decoding it. Check the project’s documentation for current platform requirements.

For routing decisions, you often need only one or two fields, such as the event type. Extracting just those fields avoids building the full object graph.

package main

import (
	"github.com/bytedance/sonic"
)

// ValidateWebhookType reads a single field without decoding the whole payload
// into a struct or map. Sonic's lazy AST parses only what is needed to reach
// the requested path.
func ValidateWebhookType(payload []byte) (string, error) {
	node, err := sonic.Get(payload, "event_type")
	if err != nil {
		return "", err
	}
	return node.String()
}

This reduces allocation compared with a full decode, but it is not strictly zero-allocation. Measure with go test -bench and -benchmem on your real payloads.

Rust: Borrowing with serde

serde supports deserializing into borrowed data. A &str field borrows directly from the input buffer, tied to the buffer’s lifetime. Two practical details matter:

• A plain &str fails to deserialize if the JSON string contains escape sequences, because the unescaped result cannot be borrowed. Cow<'a, str> with #[serde(borrow)] borrows when possible and allocates only when unescaping is required.
• To keep a nested JSON object as unparsed bytes, use &serde_json::value::RawValue, which requires the raw_value feature of serde_json.

use serde::Deserialize;
use serde_json::value::RawValue;
use std::borrow::Cow;

#[derive(Deserialize, Debug)]
pub struct WebhookPayload<'a> {
    #[serde(borrow)]
    pub event_id: Cow<'a, str>,
    #[serde(borrow)]
    pub event_type: Cow<'a, str>,
    #[serde(borrow)]
    pub tenant_id: Cow<'a, str>,
    // Nested payload kept as raw, unparsed JSON text
    #[serde(borrow)]
    pub payload_data: &'a RawValue,
}

pub fn parse_webhook(raw_bytes: &[u8]) -> Result<WebhookPayload<'_>, serde_json::Error> {
    // Fields borrow from `raw_bytes` where possible, so the buffer must
    // outlive the returned struct.
    serde_json::from_slice(raw_bytes)
}

Putting It Together

Verify the signature over the raw bytes first, then extract only the fields needed for routing and idempotency, then hand the original bytes to a durable queue or workflow. The ingress tier stays light and fast, and the heavy work happens where retries, state, and long waits are handled by infrastructure designed for them. None of these techniques removes the need to measure. Compression ratios, parser speedups, and rate-limit behavior all depend on your payload shapes, traffic patterns, and hardware, so validate each choice with load tests on your own data.