Zero-Downtime Webhook Schema Evolution: Managing Breaking Payloads Without Dropping Events

Webhooks carry a lot of B2B integration traffic: payment notifications, inventory syncs, CRM updates. If you receive them from third parties, you have probably seen the failure mode. A producer silently renames a JSON key, turns an integer ID into a prefixed string, or drops a field you depended on. Your parser throws, your endpoint starts returning 4xx or 5xx responses, and events pile up in a dead letter queue.

Webhooks have what we can call the distributed publisher problem. In an internal event stream you can often coordinate producer and consumer releases through shared CI, contract tests, or a schema registry you both control. With a third-party webhook you do not control the producer’s release cadence. Your ingestion pipeline is the part that has to absorb the change.

This guide covers how to make that absorption safe: durable capture before parsing, tolerant readers, schema validation at the edge, and versioned mapping layers backed by contract tests. One caveat first. “Zero downtime” here means you never lose an event because of a schema change. It does not mean the change costs nothing, and it does not remove the need to talk to your partners.

1. The Schema Drift Dilemma in Multi-Producer Integrations

Schema drift is the gap between the data a producer sends and the structure your consumer expects. Internally you can close that gap with contract tests and coordinated rollouts. Across dozens of external producers you mostly cannot, so you have to design for it.

Common Breaking Changes

1. Type mutation. A resource id changes from a number (123456) to a prefixed string ("usr_9f83j2"), or the reverse. A strictly typed model throws on the first payload that doesn’t match. Providers do consider this kind of change safe. Stripe’s API upgrade documentation, for example, lists adding or removing fixed prefixes on object IDs among its backward-compatible changes. Treat IDs as opaque strings unless the provider promises otherwise.
2. Key renaming and restructuring. A team refactors its domain model and renames customer_email to email_address, or promotes billing.address.zip to a top-level zip_code. Stripe’s own upgrade notes list this kind of change, such as renaming a user_id property on Connect events to account, as breaking enough to require a new API version.
3. Field removal. A producer deletes a field it assumed nobody used, and your fraud check or reconciliation job depended on it.

Not every change is breaking. Stripe treats adding new API resources, new properties on existing responses, and new event types as backward-compatible, and tells consumers to ignore unfamiliar fields and event types. A consumer that rejects unknown fields or event types will break on changes the producer considers routine.

Large Integers Are a Hidden Type Problem

Numeric IDs also have a precision problem. JavaScript’s Number type is an IEEE 754 double, so integers above 2^53 − 1 can’t be represented exactly. A consumer that parses a 64-bit ID with a naive JavaScript JSON.parse can silently corrupt it. For this reason many APIs send IDs as strings, and you should too if you produce webhooks.

The Cost of Rigid Ingestion

Fail-fast validation protects your database, but in an event-driven system it has real costs:

• Dead letter queue overflow. Valid business events pile up behind a minor schema change and need manual replay.
• Stalled workflows. Payment fulfillment, inventory locking, and similar flows wait on events that were rejected rather than processed.
• Limited provider retries. Providers retry for a bounded time. Stripe, for instance, retries live-mode deliveries for up to three days with exponential backoff, then disables an endpoint that keeps failing. If your fix takes longer than the provider’s retry window, you can lose events permanently.

That last point shapes the rest of the design. The goal is to accept and persist first, interpret later.

2. Capture First: Acknowledge Fast, Parse Later

Before any schema handling, decide what your HTTP handler does. The pattern most providers recommend is:

1. Read the raw request body and verify the signature against it.
2. Write the raw payload and headers to durable storage or a queue.
3. Return a 2xx immediately.
4. Parse, validate, and map asynchronously.

Stripe’s documentation tells receivers to defer complex logic and return a successful response quickly. If you parse synchronously and fail, you depend on the provider’s retry policy. If you persist first and fail later, you can fix the parser and replay from your own store at any time.

Verify signatures against the raw bytes. Frameworks that auto-parse JSON and re-serialize it can change whitespace or key order and break the signature check. The open Standard Webhooks specification, used by providers such as Svix, Resend, and Clerk, defines webhook-id, webhook-timestamp, and webhook-signature headers. The signature is an HMAC over <id>.<timestamp>.<body>, and receivers are advised to reject messages whose timestamp falls outside a tolerance of about five minutes. The webhook-id stays the same across retries of a delivery, so it works as a deduplication key. Because providers retry, your handlers must tolerate duplicates regardless of which spec the provider follows.

3. Writing Resilient Parsers: The Tolerant Reader

Martin Fowler’s 2011 bliki entry on the Tolerant Reader builds on Postel’s Law, the robustness principle from early TCP specifications: be conservative in what you do, be liberal in what you accept from others. His recommendation is to be as tolerant as possible when reading data from a service. In practice that means extracting only the fields you need, ignoring the rest, and not making assumptions about structure you don’t depend on. He also suggests that the consumer can send its reader and tests to the provider to run in their build, so they can catch breakages early.

A caveat is worth keeping in mind. Postel’s Law has critics, including a 2023 argument by Martin Thomson and David Schinazi that unconditional liberal acceptance can undermine robustness and security over time. Tolerance without feedback hides problems. The approach in this article is therefore tolerant to keep processing, loud to keep producers honest. Accept what you can, but log and count every deviation.

Tolerant Parsing in Go

Go’s encoding/json ignores unknown object keys by default. Decoder.DisallowUnknownFields() is an opt-in method that makes the decoder strict. It takes no arguments, so tolerance is the default and you just don’t call it.

Type mismatches are the problem. Decoding a JSON string into an integer field produces an UnmarshalTypeError. The simplest tolerant approach for IDs is to store them as strings and accept either JSON form:

package main

import (
	"encoding/json"
	"fmt"
)

// FlexibleID accepts a JSON string or number and stores it as a string,
// so integer IDs, prefixed IDs and UUIDs all survive without loss.
type FlexibleID string

func (f *FlexibleID) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err == nil {
		*f = FlexibleID(s)
		return nil
	}
	var n json.Number
	if err := json.Unmarshal(data, &n); err == nil {
		*f = FlexibleID(n.String())
		return nil
	}
	return fmt.Errorf("flexible id: expected string or number, got %s", data)
}

type WebhookPayload struct {
	EventID string     `json:"event_id"`
	UserID  FlexibleID `json:"user_id"`
}

func main() {
	payloads := []string{
		`{"event_id": "evt_123", "user_id": "usr_9f83j2"}`, // new prefixed string ID
		`{"event_id": "evt_124", "user_id": 123456}`,       // original integer ID
		`{"event_id": "evt_125", "user_id": 5, "plan": "pro"}`, // unknown field is ignored
	}
	for _, p := range payloads {
		var wp WebhookPayload
		if err := json.Unmarshal([]byte(p), &wp); err != nil {
			fmt.Println("parse error:", err)
			continue
		}
		fmt.Printf("event=%s user=%s\n", wp.EventID, wp.UserID)
	}
}

The earlier common approach of coercing a non-numeric string to 0 is dangerous. Many IDs would collapse into the same value and you would corrupt data silently. Keeping the original string avoids that. Using json.Number also avoids float64 precision loss for large integers.

Key Strategies

• Ignore unknown fields and unknown event types. Route unrecognized event types to a “seen but unhandled” counter rather than an error path.
• Use union types for fields prone to change: IDs, timestamps (epoch seconds versus ISO 8601), and booleans sent as "true" versus true.
• Separate required from nice-to-have. Fail only when a field your logic cannot work without is missing. For everything else, use a default and emit a metric.
• Keep a thin boundary. Parse the payload into a small internal model at the edge, as Fowler suggests with a Data Transfer Object, so the rest of your system never touches raw partner JSON.

4. Schema Validation at the Ingress Gateway

Tolerant parsing keeps you running, but it does not tell you that a producer changed. For that you need validation and telemetry at the edge.

[Webhook producer]
       │  HTTP POST
       ▼
[Ingress: verify signature → persist raw → return 2xx]
       │
       ▼
[Validator] ──(fetch schema by partner + event type + version)──► [Schema store]
       │
       ├─► conforms   ──► [Mapper] ──► [Event bus / queue]
       └─► deviates   ──► metric + alert, still mapped if tolerable, else quarantine

Choosing a Schema Format and Registry

JSON Schema is the most widely used way to describe JSON documents. The current version is 2020-12, which the project’s site lists as the latest specification. You can store versioned schemas per partner and event type in a registry or in a simple database table.

Check what your tooling supports before standardizing on a draft. AWS Glue Schema Registry’s JSON support covers Draft-04, Draft-06, and Draft-07, with validation done by the Everit library, so it does not accept 2020-12 schemas. Glue also defines eight compatibility modes: NONE, DISABLED, BACKWARD, BACKWARD_ALL, FORWARD, FORWARD_ALL, FULL, and FULL_ALL. Other registries such as Confluent Schema Registry and Apicurio Registry also support JSON Schema, with their own draft coverage, so verify the draft against the version you run.

Tiered Validation Modes

Strict edge validation defeats the point if every producer change causes rejections. Use tiers:

1. Shadow mode. Validate and record deviations, but never block. This is where you discover drift.
2. Enforcement for critical fields. Once telemetry shows a schema is stable, reject or quarantine payloads that break the few invariants your business logic requires.
3. Compatibility checks on your own schema versions. Registries enforce compatibility modes such as BACKWARD or FULL between versions of a schema you register. This governs your view of a partner’s payload. You cannot enforce it on the partner. If a partner publishes a changelog or an OpenAPI/JSON Schema description, diff against it in CI.

Record the deviation rate per partner and event type. A sudden rise after a partner’s release date is often the first sign of drift, usually before any customer complains.

5. Dual-Schema Mapping Layers

Sometimes a breaking change is unavoidable, such as a restructured order object, and you will see old and new shapes at the same time. A mapping layer based on the Adapter pattern translates each variant into one canonical internal event, so only the adapter knows about partner formats.

• Version detection. Identify the variant from whatever the producer gives you. Stripe, for example, ties the payload shape to the API version configured on each webhook endpoint, and its documentation recommends updating webhook code to handle both the old and new version of each object before upgrading. If a provider offers no version marker, detect by shape (presence of keys) as a fallback.
• Transformation. Use declarative mappings where they fit, such as jq or JSONata, or plain functions for logic that needs types and tests.
• Canonical event. Emit one internal structure that downstream services consume. Version the canonical model too, so internal changes follow the same discipline.

Keep the raw payload alongside the canonical event so you can re-map historical events when a mapping bug is found.

6. Contract Tests for Producers You Don’t Control

Consumer-driven contract testing tools such as Pact were built around services where both sides can run the tests. External webhook producers won’t run your suite, so treat recorded payloads as the contract instead:

• Keep a fixture library of real payloads per partner, event type, and version, with secrets and personal data removed.
• Run every fixture through the parser and mapper in CI.
• When a partner announces or ships a change, add the new payload shape as a fixture before it reaches production.
• Where a partner is willing, share your fixtures or schemas with them, as Fowler suggests for the Tolerant Reader.

Here is a pytest example. Parsing monetary values from floats is a classic trap: int(19.99 * 100) evaluates to 1998 in Python because 19.99 * 100 is slightly under 1999. The example converts through Decimal instead.

from dataclasses import dataclass
from decimal import Decimal

import pytest


@dataclass(frozen=True)
class NormalizedOrder:
    order_id: str
    total_cents: int
    customer_id: str


def parse_incoming_webhook(payload: dict, schema_version: str) -> NormalizedOrder:
    if schema_version == "v1_legacy":
        # Legacy: integer id, amount in decimal dollars, "client_id"
        cents = int(Decimal(str(payload["amount"])) * 100)
        return NormalizedOrder(
            order_id=str(payload["id"]),
            total_cents=cents,
            customer_id=str(payload["client_id"]),
        )
    if schema_version == "v2_current":
        # Current: string order id, integer cents, "customer_id"
        return NormalizedOrder(
            order_id=str(payload["order_id"]),
            total_cents=int(payload["total_amount_cents"]),
            customer_id=str(payload["customer_id"]),
        )
    raise ValueError(f"Unsupported schema version: {schema_version}")


@pytest.mark.parametrize(
    "payload,version,expected",
    [
        (
            {"id": 9988, "amount": 49.99, "client_id": 42},
            "v1_legacy",
            NormalizedOrder("9988", 4999, "42"),
        ),
        (
            {"id": 9989, "amount": 19.99, "client_id": 42},
            "v1_legacy",
            NormalizedOrder("9989", 1999, "42"),
        ),
        (
            {"order_id": "ord_abc123", "total_amount_cents": 5000, "customer_id": "usr_789"},
            "v2_current",
            NormalizedOrder("ord_abc123", 5000, "usr_789"),
        ),
        (
            # Unknown extra field must not break parsing
            {"order_id": "ord_abc124", "total_amount_cents": 100, "customer_id": "usr_1", "new_flag": True},
            "v2_current",
            NormalizedOrder("ord_abc124", 100, "usr_1"),
        ),
    ],
)
def test_webhook_schema_compatibility(payload, version, expected):
    assert parse_incoming_webhook(payload, version) == expected


def test_unknown_version_is_rejected_explicitly():
    with pytest.raises(ValueError):
        parse_incoming_webhook({}, "v9_future")

Operational Checklist

1. Persist before you interpret. Verify the signature on the raw body, store the payload, and return 2xx quickly.
2. Be tolerant in parsing. Ignore unknown fields and event types, accept multiple representations of IDs and timestamps, and avoid lossy coercions.
3. Be loud in monitoring. Validate against versioned schemas in shadow mode, track deviation rates per partner, and alert on spikes.
4. Isolate with a canonical model. Map every partner variant through versioned adapters and keep raw payloads for replay.
5. Test against recorded reality. Maintain a fixture matrix of past, current, and announced payloads in CI.
6. Make handlers idempotent. Deduplicate on the provider’s event or message ID, because retries and replays will deliver the same event more than once.
7. Talk to your partners. Subscribe to their changelogs, pin API versions where they offer it, and upgrade deliberately. Stripe, for example, ships monthly releases with no breaking changes and issues major releases with breaking changes twice a year.

Treat webhook schemas as contracts that evolve and you can keep processing events through producer changes. The unavoidable work is noticing a change quickly and mapping it correctly.

Further Reading

• Stripe: Receive events in your webhook endpoint, API versioning, API upgrades
• Martin Fowler: Tolerant Reader
• Standard Webhooks specification
• JSON Schema specification
• AWS Glue Schema Registry
• Go: encoding/json package documentation