Handling Out-of-Order Webhooks: Event Sequencing in Distributed Systems

In event-driven integrations, webhooks are the standard way platforms propagate state changes across service boundaries. When a customer buys a product, updates a subscription, or cancels an invoice, the upstream platform fires an HTTP POST to every downstream consumer that's listening.

Under ideal network conditions, webhooks would arrive in the exact order the underlying events occurred. Production systems don't run under ideal conditions. Retry queues, parallel delivery workers, and multi-region infrastructure all mean events can and do arrive out of sequence — and this isn't an edge case, it's the documented, expected behavior of essentially every major webhook provider.

Consider a common e-commerce scenario: a customer creates an order and cancels it a second later. If network jitter delays the order.created webhook while order.canceled sails through, your consumer processes the cancellation first. When order.created finally lands, a naive handler overwrites the record and resurrects a dead order.

This guide covers why webhooks lose their ordering in transit, why the instinctive fixes don't hold up, and the architectural patterns — sequence numbers, finite state machines, full-state events, and canonical re-fetching — that make a consumer correct regardless of arrival order.

1. Why webhooks arrive out of order

       Upstream Provider                           Downstream Consumer
   +-----------------------+                    +------------------------+
   |  Event 1: Created     |                    |                        |
   |  (Timestamp: 10:00:00)|---[ Retried / ]--->|                        |
   |                       |   [ Delayed   ]    |                        |
   |                       |                    | (10:00:02) Receives:   |
   |  Event 2: Canceled    |                    | "Order Canceled"       |
   |  (Timestamp: 10:00:01)|------------------->| Status set to CANCELED |
   |                       |                    |                        |
   |                       |                    | (10:00:05) Receives:   |
   |                       |------------------->| "Order Created"        |
   |                       |   (Delayed Event)  | Status set to CREATED  |
   +-----------------------+                    | BAD STATE: Resurrected!|
                                                +------------------------+

Three mechanisms account for most reordering in practice:

1. Retries with exponential backoff. If a provider gets a transient error (a 503, a timeout) sending Event 1, it queues that event for retry. Event 2, which happened afterward, gets sent immediately and succeeds. When Event 1's retry finally lands, it arrives after Event 2.

2. Parallel delivery workers. High-throughput providers dispatch outgoing webhooks across many concurrent workers. If Event 1's worker hits a slow network path and Event 2's worker doesn't, Event 2 arrives first even though it happened second.

3. Multi-region replication lag. When events originate from geographically distributed databases, replication delays between regions can push an earlier event onto the outbound queue after a later one generated in a faster-replicating region.

This isn't theoretical — it's explicitly documented behavior. Stripe's webhook documentation states plainly that it doesn't guarantee events are delivered in the order they're generated, and instructs integrators to design endpoints that don't depend on a specific order. Shopify's webhook documentation says the same: ordering isn't guaranteed within a topic or across topics for the same resource, and gives the exact example this article opened with — a products/update webhook can arrive before the products/create webhook for the same product.

2. Why the instinctive fixes don't work

Anti-pattern 1: Sorting by the payload's created_at timestamp

-- DANGEROUS: susceptible to clock skew and same-millisecond collisions
UPDATE orders
SET status = 'created', updated_at = '2026-09-17T10:00:00Z'
WHERE id = 'ord_123'
  AND updated_at < '2026-09-17T10:00:00Z';

Stripe's own guidance is direct on this point: snapshot events are timestamped to the second, so distinct events can share a created value, and the docs explicitly warn against using created to determine order or to detect duplicates — event IDs are what should be tracked instead. Two problems drive this:

• Clock and granularity limits. If two events land within the same timestamp resolution window, their timestamps are identical and give you no ordering information at all.
• The timestamp reflects when the event was generated, not when it's safe to apply. A delayed retry keeps its original generation time, so sorting by it doesn't fix the resurrection problem — the delayed order.created event still looks "older" than the cancellation on paper, but it's arriving and would be applied later in wall-clock time if you don't guard against it.

Anti-pattern 2: In-memory locks across instances

Using a local mutex to serialize processing by order_id only works if exactly one instance of your receiver is running. The moment you scale horizontally — multiple pods, multiple serverless invocations — an in-memory lock on one instance offers zero protection against a concurrent event landing on another.

3. Core patterns for webhook event sequencing

Pattern A: Idempotency keys (table stakes, not optional)

Before ordering, solve for duplicates — at-least-once delivery is the default guarantee for almost every major provider (Stripe, GitHub, Shopify among them), which means your endpoint will see the same event more than once. The fix is a stable, per-event identifier: Stripe sends event.id, Shopify sends an X-Shopify-Webhook-Id header, and a growing number of providers follow the open Standard Webhooks specification, which defines a webhook-id header specifically as an idempotency key that stays constant across retries. Reporting suggests adopters of that spec now include Zapier, Twilio, Supabase, PagerDuty, and several AI providers.

The pattern is simple: record the ID in a unique-constrained table (or a short-lived Redis key) before or during processing, and skip anything you've already recorded.

CREATE TABLE processed_webhooks (
  event_id    TEXT PRIMARY KEY,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- INSERT fails on a duplicate event_id -> you know it's already been handled

Idempotency solves duplication. It does nothing for ordering on its own — that's what the remaining patterns address.

Pattern B: Monotonic sequence numbers with optimistic concurrency control

Where the upstream system exposes a per-resource sequence or version number, the consumer can reject any event that doesn't move the version forward.

{
  "id": "evt_987654321",
  "type": "order.updated",
  "resource_id": "ord_1001",
  "sequence": 3,
  "timestamp": "2026-09-17T10:00:01.452Z",
  "data": { "status": "processing", "total_cents": 4999 }
}

-- PostgreSQL atomic update, guarded by version
UPDATE orders
SET status = $1, version = $2, updated_at = NOW()
WHERE id = $3
  AND version < $2;

A 0 row count means one of two things: the event is stale (a higher sequence was already applied), or there's a gap (you received sequence 4 while sitting at sequence 1). Distinguishing the two determines whether you silently drop the event or escalate to a recovery strategy (Section 4).

This pattern only works if the upstream provider actually issues sequence numbers — most mainstream SaaS webhook systems (Stripe, Shopify, GitHub) don't expose one on standard events, which is why the next two patterns exist for providers that don't.

Pattern C: Finite state machines

Without a sequence number, you can still enforce domain-level invariants by restricting which state transitions are valid, independent of what order events arrive in.

                  +----------------------------------+
                  |                                  |
                  v                                  |
            +-----------+      +------------+      +-+----------+
[Start] --->|  CREATED  |----->| PROCESSING |----->| COMPLETED  |
            +-----------+      +------------+      +------------+
                  |                                  ^
                  |                                  |
                  +----------------------------------+
                  |
                  v
            +-----------+
            | CANCELED  | (Terminal State)
            +-----------+

import { Pool } from 'pg';

const dbPool = new Pool({ connectionString: process.env.DATABASE_URL });

const VALID_TRANSITIONS: Record<string, string[]> = {
  NONE: ['CREATED', 'CANCELED'],
  CREATED: ['PROCESSING', 'CANCELED', 'COMPLETED'],
  PROCESSING: ['COMPLETED', 'CANCELED'],
  COMPLETED: [],
  CANCELED: [],
};

interface WebhookPayload {
  eventId: string;
  orderId: string;
  targetState: string;
  sequence: number;
}

export async function handleOrderWebhook(payload: WebhookPayload): Promise<void> {
  const client = await dbPool.connect();
  try {
    await client.query('BEGIN');

    const res = await client.query(
      `SELECT status, version FROM orders WHERE id = $1 FOR UPDATE`,
      [payload.orderId]
    );
    const existing = res.rows[0];
    const currentState = existing ? existing.status : 'NONE';
    const currentVersion = existing ? existing.version : 0;

    if (existing && payload.sequence <= currentVersion) {
      await client.query('ROLLBACK');
      return; // stale/duplicate
    }

    const allowed = VALID_TRANSITIONS[currentState] || [];
    if (!allowed.includes(payload.targetState)) {
      // Special case: cancellation arriving before creation is still valid data
      if (currentState === 'NONE' && payload.targetState === 'CANCELED') {
        await client.query(
          `INSERT INTO orders (id, status, version, created_at) VALUES ($1, $2, $3, NOW())`,
          [payload.orderId, 'CANCELED', payload.sequence]
        );
        await client.query('COMMIT');
        return;
      }
      await client.query('ROLLBACK'); // invalid transition, reject
      return;
    }

    if (!existing) {
      await client.query(
        `INSERT INTO orders (id, status, version, created_at) VALUES ($1, $2, $3, NOW())`,
        [payload.orderId, payload.targetState, payload.sequence]
      );
    } else {
      await client.query(
        `UPDATE orders SET status = $1, version = $2, updated_at = NOW() WHERE id = $3`,
        [payload.targetState, payload.sequence, payload.orderId]
      );
    }
    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

Pattern D: Emit full-state or terminal events instead of deltas

The most effective fix is often on the producer side, and it's the approach several providers have converged on rather than trying to guarantee delivery order at all. Convoy's guidance for webhook providers makes the case directly: requesting strict ordering from a webhook system is an anti-pattern because it adds significant complexity, and the better fix is to design payloads so ordering doesn't matter. One technique is to emit a distinct event per state reached (invoice.paid, invoice.voided) rather than a generic invoice.updated delta — a paid invoice can't become unpaid, so a consumer applying a terminal event doesn't need to know what came before it.

Shopify's newer order-webhook design for agent integrations takes this further: every delivery carries the full current state of the order, identical to what a direct API read would return, and the documentation is explicit that consumers should "treat the latest payload as the source of truth" rather than replaying deltas to reconstruct state. If you can influence the payload shape (an internal event bus, a partner API you also control) or if the upstream API already works this way, this pattern removes the ordering problem for the receiver almost entirely — the newest payload always wins, no version tracking required.

Pattern E: Canonical state re-fetch (the "claim check" pattern)

Where you don't control the payload shape, use the webhook only as a notification and treat the API as the source of truth:

1. Receive the webhook as a signal that something changed.
2. Make a synchronous GET back to the canonical resource endpoint.
3. Overwrite local state with whatever the API returns right now.

This is exactly what both Stripe and Shopify recommend when payload data might be stale or incomplete — Stripe's docs point out you can retrieve missing objects via the API, and Shopify explicitly recommends periodic reconciliation jobs against its API because delivery (and by extension, order) isn't guaranteed. The tradeoff is an extra HTTP round trip and exposure to the upstream API's rate limits, but it's self-healing: even a dropped event gets corrected on the next re-fetch.

Pattern F: Buffering with a TTL

When you must apply deltas and can't fall back to an API re-fetch, hold out-of-sequence events in a short-lived buffer keyed by sequence number:

async function processOrBufferEvent(event: WebhookPayload) {
  const currentVer = await getCurrentDbVersion(event.orderId);

  if (event.sequence === currentVer + 1) {
    await applyEventToDatabase(event);
    await drainBufferedEvents(event.orderId, event.sequence + 1);
  } else if (event.sequence > currentVer + 1) {
    // Gap: park it and set a safety TTL in case the missing event never arrives
    await redis.zadd(`buffer:${event.orderId}`, event.sequence, JSON.stringify(event));
    await redis.expire(`buffer:${event.orderId}`, 300);
  } else {
    // Stale/duplicate
  }
}

If the upstream provider itself offers ordered delivery, that's worth using instead of building this yourself. Svix, an open-source webhook-sending platform, publishes a good explanation of why it doesn't guarantee order on regular endpoints, but it does offer opt-in FIFO endpoints for consumers who need strict ordering — at the cost of throughput, since each delivery blocks until the previous one is acknowledged.

4. What major providers actually do (a quick reference)

The pattern across all of them is consistent: no mainstream provider promises ordered delivery by default, and all of them point integrators toward the same two tools — a stable ID for deduplication, and either a state machine/version check or a live API call for correctness.

5. Comparison of consumer-side patterns

6. Debugging and testing tools

A few widely used, genuinely independent options for inspecting and replaying webhook traffic during development:

• Provider-native tooling — the Stripe CLI can trigger and forward test events locally, and GitHub lets you redeliver any webhook from the past retention window straight from the repository settings UI or the REST API.
• ngrok / webhook.site — quick, low-friction ways to expose a local endpoint or inspect raw payloads without deploying anything.
• Hookdeck and Svix — hosted webhook gateways that sit between the provider and your app, adding queuing, delivery logs, and replay so you can see exactly what was sent and when, independent of your own server logs.

None of these solve ordering for you — that logic still lives in your consumer, per the patterns above — but they make it far easier to see when out-of-order delivery is actually happening versus a bug in your own handler.

7. Resilience checklist

• Idempotency keys — store the provider's event/delivery ID and skip anything already processed.
• No timestamp-based ordering — providers explicitly warn that created/updated_at fields can collide or reflect generation time, not safe-to-apply time.
• Sequence numbers where available — enforce WHERE version < incoming_version on writes.
• State machine validation — block transitions like CANCELED → CREATED regardless of arrival order.
• Prefer full-state or terminal-event payloads where you control the producer, so the newest delivery is always correct on its own.
• Canonical re-fetch fallback — call the source API when a payload might be stale or a sequence gap is detected.
• Reconciliation jobs — periodically re-sync against the API, since delivery itself (not just order) usually isn't guaranteed either.
• Dead-letter queue — route events that fail validation after retries to a DLQ for manual inspection rather than silently dropping them.

Conclusion

Out-of-order delivery isn't a bug in any particular webhook provider — it's a documented, structural property of asynchronous HTTP delivery, and every major platform tells integrators so directly. Trying to force strict ordering at the transport layer is fragile; the durable fix is a consumer (and, where possible, a producer) designed so that arrival order simply doesn't matter — through idempotency keys, version-checked writes, state machine validation, full-state payloads, or a canonical re-fetch when in doubt.

A note on this piece: an earlier draft of this article contained a section promoting a specific "webhook debugging" product as though it were an established, neutral tool. That product's own content turned out to be SEO-style marketing material rather than documentation, so it's been replaced above with real, independently verifiable tools and vendor documentation — the tone throughout is intentionally point at primary sources you can check yourself.