Webhook Payload Compression at Scale: Evaluating Brotli, Zstandard (zstd), and Gzip for High-Volume Ingress

Most compression advice for APIs is about responses. Webhook gateways have the opposite problem. Providers push JSON into your infrastructure, often in bursts, and the request body is the part you pay to receive, buffer, verify and forward.

This article looks at when compressing inbound webhook bodies is worth it, how Gzip, Brotli and Zstandard compare for that job, and how to decompress untrusted request bodies without opening a denial-of-service hole. It uses published benchmarks and vendor documentation, and says plainly where no hard numbers exist.

1. Is ingress compression worth it?

Start with the cost model. On the major clouds, data coming in from the public internet is generally not what you are billed for, so compressing inbound webhooks will not by itself shrink a bill. Check your own provider’s pricing. The savings usually show up elsewhere:

• Bytes moved through metered hops inside your network, such as NAT, private-link or transit-style gateways, and cross-zone traffic between the gateway and your workers.
• Buffering pressure on the proxy, and queue or log storage if you persist raw bodies.
• Provider-to-you transfer time for large payloads, which matters for retries and timeouts.

Compression also moves CPU cost onto your gateway, which has to decompress every body. A sender that compresses can also shift cost onto you if you are careless about limits (see section 5).

There is also an adoption question. Mainstream providers document their webhooks as plain JSON. GitHub, for example, caps payloads at 25 MB and signs the request body with HMAC-SHA256 in X-Hub-Signature-256 (GitHub docs). Compressed webhooks do exist, but they are a feature of specific senders. Stream Chat documents Content-Encoding: gzip on its HTTP webhooks (Stream docs). QuickNode Streams may send gzip bodies and notes that signature verification has to account for them (QuickNode guide). Crawlbase’s Enterprise Crawler delivers its callbacks gzip-compressed (Crawlbase docs).

I could not find a major webhook provider that documents sending zstd or br request bodies. In practice, gzip is what you will receive from third parties today. Zstandard is most realistic for traffic between services you control, or from partners who agree to it.

2. How request-body compression works

A webhook is an HTTP POST. If the sender compresses the body, it declares the algorithm in the Content-Encoding request header, for example Content-Encoding: zstd.

The opposite direction is optional. A server can advertise which request content-codings it accepts by sending Accept-Encoding in its responses, but most webhook senders will not act on that. Treat it as documentation for integrators rather than automatic negotiation. In Envoy, the decompressor filter has an advertise_accept_encoding option for the request direction that does this advertising (API reference).

JSON is a good fit for general-purpose compressors. It is text with repeated keys, repeated punctuation, and recurring value shapes such as timestamps and IDs. Real ratios depend on your payloads, which is why section 4 shows how to measure them.

3. Gzip, Brotli and Zstandard compared

Gzip. It is supported nearly everywhere, which is its main strength. Its DEFLATE window is 32 KB, so decompression memory per stream is small and predictable.

Brotli. It ships with a static dictionary of roughly 120 KB, made of common words and snippets from natural languages and web content, which helps on small text. It is excellent at high levels for pre-compressed static assets. For on-the-fly compression, the high levels are expensive, which makes them a poor choice for a sender compressing each webhook as it is produced.

Zstandard. It was designed for real-time use, with a wide range of levels and fast decompression. It also supports dictionaries trained on sample data, which must be available on both ends. For HTTP, RFC 9659 requires decoders to support windows up to 8 MB and forbids encoders from producing frames that need more (RFC 9659). Budget for that when sizing memory under concurrency.

What published benchmarks show

I have no trustworthy JSON-webhook benchmark to cite, and an invented one would be worse than none. The most useful public reference is the Zstandard project’s own comparison, run with lzbench on the Silesia corpus (a mixed corpus, not JSON) (facebook/zstd):

At comparable fast settings, zstd compresses about five times faster than zlib and decompresses nearly four times faster, with a slightly better ratio. The project also notes that decompression speed stays roughly constant as you raise the compression level, which is why a sender can spend more CPU on ratio without slowing your gateway.

Two cautions apply. These are corpus-wide numbers at the fastest levels, so absolute figures will differ on your hardware and your JSON. Library versions and bindings also matter, and Go implementations will not match the C reference.

4. Measure your own payloads

Before changing anything, benchmark samples of your real traffic. The reference tools include benchmarking modes:

# Collect a few hundred representative webhook bodies into ./samples/
# Zstandard: benchmark levels 1-5 on one file or a concatenation
zstd -b1 -e5 samples/sample.json

# Compare against gzip and brotli at the levels a sender would plausibly use
for lvl in 1 6; do gzip -c -$lvl samples/sample.json | wc -c; done
for q in 4 5; do brotli -c -q $q samples/sample.json | wc -c; done

Measure per-payload sizes, not one large concatenation, because per-request compression of small bodies behaves differently from compressing a big file. If payloads are small, also try a trained zstd dictionary:

zstd --train samples/* -o webhook.dict
zstd -D webhook.dict -3 samples/sample.json

A dictionary has to be shared out of band and versioned carefully, since a sender and receiver with different dictionaries cannot interoperate. It is a good fit for partner or internal traffic, and a poor one for public third-party providers.

5. Security: decompression bombs and memory limits

Accepting compressed request bodies means accepting attacker-controlled expansion. A tiny body can decompress into a very large one. This is a recognised vulnerability class. Envoy shipped a fix for a decompressor zip-bomb issue (CVE-2022-29225) that capped the maximum inflation size (Envoy 1.22.1 release notes).

Defences that matter, in rough order of importance:

1. Cap the decompressed size. This is the primary control. Enforce it while streaming, and fail when it is exceeded rather than silently truncating.
2. Cap the compressed size too. Limit what you will read off the wire before any decoder runs.
3. Bound decoder memory. Zstd’s window can be up to 8 MB per stream under the HTTP rules, so many concurrent streams add up.
4. Use an expansion-ratio limit as a secondary signal only. JSON can legitimately compress by large factors, so a fixed threshold can reject valid traffic. Choose one from your observed traffic and alert before you enforce. Envoy’s gzip decompressor exposes max_inflate_ratio for this purpose.
5. Never call io.ReadAll on a decoder unless it is already wrapped in a hard cap.
6. Reject unknown encodings. RFC 9110 lets a server answer with 415 Unsupported Media Type when it does not support a request’s Content-Encoding. Use 413 when a limit is exceeded.

A bounded decompression middleware in Go

This sketch uses the standard library’s compress/gzip, github.com/andybalholm/brotli and github.com/klauspost/compress/zstd. A detail worth noting: the klauspost zstd Decoder.Close() returns nothing and must be called to release resources, so it cannot be treated as an io.Closer directly. It needs an adapter.

package ingress

import (
	"compress/gzip"
	"errors"
	"io"
	"net/http"
	"strings"

	"github.com/andybalholm/brotli"
	"github.com/klauspost/compress/zstd"
)

const (
	maxCompressedBytes   int64  = 1 << 20  // read at most 1 MiB off the wire
	maxDecompressedBytes int64  = 10 << 20 // hand at most 10 MiB to handlers
	maxZstdWindowBytes   uint64 = 8 << 20  // RFC 9659 window limit for HTTP
)

// ErrDecompressedTooLarge is returned by Read when the decoded body exceeds the cap.
var ErrDecompressedTooLarge = errors.New("decompressed body exceeds limit")

// cappedReader fails loudly (unlike io.LimitReader) when the cap is exceeded.
type cappedReader struct {
	r         io.Reader
	remaining int64
}

func (c *cappedReader) Read(p []byte) (int, error) {
	if c.remaining < 0 {
		return 0, ErrDecompressedTooLarge
	}
	// Allow one byte past the cap so overflow can be detected.
	if int64(len(p)) > c.remaining+1 {
		p = p[:c.remaining+1]
	}
	n, err := c.r.Read(p)
	c.remaining -= int64(n)
	if c.remaining < 0 {
		return 0, ErrDecompressedTooLarge
	}
	return n, err
}

type decodedBody struct {
	io.Reader
	closers []func() error
}

func (d *decodedBody) Close() error {
	var first error
	for _, c := range d.closers {
		if err := c(); err != nil && first == nil {
			first = err
		}
	}
	return first
}

// Decompress transparently decodes gzip, br and zstd request bodies with hard limits.
func Decompress(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		enc := strings.ToLower(strings.TrimSpace(r.Header.Get("Content-Encoding")))
		if enc == "" || enc == "identity" {
			r.Body = http.MaxBytesReader(w, r.Body, maxDecompressedBytes)
			next.ServeHTTP(w, r)
			return
		}

		compressed := http.MaxBytesReader(w, r.Body, maxCompressedBytes)

		var (
			rd      io.Reader
			closers []func() error
		)
		switch enc {
		case "gzip":
			zr, err := gzip.NewReader(compressed)
			if err != nil {
				http.Error(w, "invalid gzip body", http.StatusBadRequest)
				return
			}
			rd, closers = zr, append(closers, zr.Close)
		case "br":
			rd = brotli.NewReader(compressed)
		case "zstd":
			zr, err := zstd.NewReader(compressed,
				zstd.WithDecoderConcurrency(1),
				zstd.WithDecoderMaxMemory(maxZstdWindowBytes),
			)
			if err != nil {
				http.Error(w, "invalid zstd body", http.StatusBadRequest)
				return
			}
			rd = zr
			closers = append(closers, func() error { zr.Close(); return nil })
		default:
			http.Error(w, "unsupported Content-Encoding", http.StatusUnsupportedMediaType)
			return
		}

		r.Body = &decodedBody{
			Reader:  &cappedReader{r: rd, remaining: maxDecompressedBytes},
			closers: closers,
		}
		r.Header.Del("Content-Encoding")
		r.Header.Del("Content-Length")
		r.ContentLength = -1
		next.ServeHTTP(w, r)
	})
}

Downstream handlers should map ErrDecompressedTooLarge and *http.MaxBytesError to 413 Payload Too Large. A truncated or corrupt stream surfaces as a read error, which should become a 400.

Signature verification

Whether to verify the signature before or after decompression depends on what the provider signed, so read each provider’s documentation. GitHub documents its HMAC as computed over the request body (GitHub docs), and GitHub sends plain JSON. For a sender that compresses, QuickNode’s guidance shows verification being done on the decoded JSON string.

If the provider signs the bytes on the wire, hash the compressed stream with io.TeeReader as you read it. If it signs the decoded payload, hash the output of the capped reader. In both cases, hash only bytes that have already passed your limits, and compare signatures in constant time. The middleware above discards the raw compressed bytes, so it suits only the second case unless you add the tee.

Don’t sniff magic bytes

It is tempting to detect compression by inspecting leading bytes when a sender forgets the header. Gzip starts with 1F 8B and zstd frames start with 28 B5 2F FD, but Brotli has no fixed signature. Guessing also makes your parser’s behaviour depend on attacker-controlled content. A sturdier policy is to trust the declared Content-Encoding, reject mismatches with 400, and tell integrators to fix their sender.

6. Offloading decompression to Envoy

Envoy’s HTTP decompressor filter supports gzip, Brotli and Zstandard (Envoy docs). It can be enabled separately for requests and responses, and each filter instance is configured with one decompressor library. To accept several encodings, you add one filter instance per library. The docs also say a buffer filter must come after the decompressor for requests.

The following is a structural sketch of an HTTP filter chain, not a complete bootstrap (listener, TLS and cluster definitions are omitted). Check field names against the Envoy version you run.

http_filters:
- name: gzip_request_decompressor
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
    decompressor_library:
      name: gzip
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip
        max_inflate_ratio: 50   # tune from observed traffic
    request_direction_config:
      common_config:
        enabled:
          default_value: true
          runtime_key: gzip_request_decompression_enabled
      advertise_accept_encoding: true
    response_direction_config:
      common_config:
        enabled:
          default_value: false
          runtime_key: gzip_response_decompression_enabled
- name: zstd_request_decompressor
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
    decompressor_library:
      name: zstd
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.compression.zstd.decompressor.v3.Zstd
    request_direction_config:
      common_config:
        enabled:
          default_value: true
          runtime_key: zstd_request_decompression_enabled
      advertise_accept_encoding: false
    response_direction_config:
      common_config:
        enabled:
          default_value: false
          runtime_key: zstd_response_decompression_enabled
- name: envoy.filters.http.buffer
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer
    max_request_bytes: 10485760
- name: envoy.filters.http.router
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Keep Envoy current. The zip-bomb fix mentioned above landed in patch releases back in 2022, and the inflation protection is on by default.

If you terminate in NGINX, check whether your build supports request-body decompression for the encodings you need. Do not assume it from response-side compression modules.

7. Recommendations

1. Support gzip first. It is the only encoding you are likely to receive from third-party providers today.
2. Add zstd for traffic you control. It combines fast decompression with good ratios, and you can pair it with a dictionary for partner integrations. Remember the 8 MB window limit when planning memory.
3. Avoid expensive Brotli levels for senders compressing in real time. Brotli is a better fit for static assets than for per-request compression.
4. Cap everything. Compressed size, decompressed size and decoder memory should all be bounded, and failures should be explicit 413 or 400 responses.
5. Treat ratio limits as tuned guardrails, not universal constants.
6. Verify signatures over the bytes the provider actually signed, and only after limits are enforced.
7. Measure before rolling out. Use samples of your real payloads, and track decompression CPU, memory per connection and rejection rates once live.

Sources

• GitHub: Webhook events and payloads
• Zstandard project README and benchmarks
• RFC 9659: Window Sizing for Zstandard Content Encoding
• Envoy: Decompressor filter
• Envoy 1.22.1 release notes (CVE-2022-29225)
• klauspost/compress zstd package documentation
• Stream Chat webhooks overview
• QuickNode: Validating incoming Streams webhook messages
• Crawlbase: Webhook receiving