InstaWebhook
November 20, 2026By InstaWebhook TeamRetries and Replay

Webhook Engineering: Retries, Dynamic Backoff, and Multi-Tenant Failure Recovery

Master webhook retry strategies and failure recovery. Learn to parse Retry-After headers, isolate partial batch failures, prevent retry storms, and classify errors.

Webhook Engineering: Retries, Dynamic Backoff, and Multi-Tenant Failure Recovery

Webhook Engineering: Retries, Dynamic Backoff, and Multi-Tenant Failure Recovery

Building a high-throughput webhook delivery engine means handling remote failures gracefully. When you deliver millions of webhooks a day to thousands of customer endpoints, naive retry loops, such as one exponential backoff schedule applied to every HTTP error, lead to worker starvation, retry storms and duplicated side effects.

There is no single retry standard to copy. Stripe retries a live-mode event for up to three days with exponential backoff, and test-mode events are retried three times over a few hours. GitHub does not automatically redeliver failed deliveries at all. Shopify documents up to eight retries over roughly four hours and may remove the subscription if failures persist. Provider timeouts also differ: GitHub expects a 2xx within 10 seconds and Shopify within 5. These figures come from provider documentation and may change, so check the current docs before relying on them.

The community Standard Webhooks specification offers non-binding recommendations for senders: treat 2xx as success, retry with exponential backoff and jitter, treat redirects as failures, and stop on 410 Gone. It is not an IETF standard.

This guide covers parsing rate-limit signals, isolating bad events in batches, containing multi-tenant outages, and classifying transient versus permanent failures.


1. Stop Retrying 429s Blindly: Honor Retry-After and Rate-Limit Headers

The Flaw of Blind Exponential Backoff

Standard exponential backoff with a cap calculates delay like this:

Tdelay=min⁡(Tmax,Tbase×2attempt)T_{\text{delay}} = \min\left(T_{\text{max}}, T_{\text{base}} \times 2^{\text{attempt}}\right)Tdelay​=min(Tmax​,Tbase​×2attempt)

The widely used “full jitter” variant randomizes the delay so that many failing deliveries do not retry in lockstep:

Tdelay=random(0,min⁡(Tmax,Tbase×2attempt))T_{\text{delay}} = \text{random}\left(0, \min\left(T_{\text{max}}, T_{\text{base}} \times 2^{\text{attempt}}\right)\right)Tdelay​=random(0,min(Tmax​,Tbase​×2attempt))

This works well for transient network glitches. Applied blindly to 429 Too Many Requests, it ignores explicit feedback from the receiver. If a receiver says “come back in 120 seconds” and your sender retries at 2s, 4s, 8s and 16s, every one of those attempts is rejected. Depending on the receiver’s limiter, rejected requests may also count against its quota, and they consume your own retry budget either way.

Code example
+-------------------+             +-----------------------+
| Webhook Dispatch  |             | Downstream Receiver   |
+-------------------+             +-----------------------+
          |                                   |
          |-------- Delivery Attempt -------->|
          |                                   | (Rate Limit Exceeded)
          |<-- 429 [Retry-After: 120] --------|
          |                                   |
          |-- Naive Retry in 2s (BREACH) ---->| (Rejected: 429)
          |-- Naive Retry in 4s (BREACH) ---->| (Rejected: 429)
          |                                   |
          |-- Header-Aware Retry (Wait 120s)->| (SUCCESS: 200 OK)

Deconstructing Rate-Limit Headers

Receivers signal backoff through standard and vendor-specific headers. Check them in this order:

  1. Retry-After (RFC 9110, Section 10.2.3). The 429 status code itself comes from RFC 6585. The value is either a non-negative integer of seconds or an HTTP-date.
    • Seconds: Retry-After: 120
    • HTTP-date: Retry-After: Fri, 27 Nov 2026 07:28:00 GMT
    • RFC 9110 also allows Retry-After on 503 Service Unavailable responses, so check it there too.
  2. IETF RateLimit and RateLimit-Policy fields. These are still an Internet-Draft, not an RFC. As of draft-ietf-httpapi-ratelimit-headers-11 (May 2026) the working group’s draft defines two structured-field headers:
    • RateLimit: "default";r=0;t=30, where r is the remaining quota and t is the number of seconds in the effective window.
    • RateLimit-Policy: "default";q=100;w=60, which describes the quota policy: q is the quota and w is the window in seconds.
    • Older drafts (and many blog posts) describe separate RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset headers, and a RateLimit-Policy value such as 100, 100;w=60. The current draft has replaced that syntax, so parse both forms only if you must support legacy senders.
    • The draft says that if both RateLimit and Retry-After are present, Retry-After takes precedence.
  3. Legacy vendor headers. X-RateLimit-Reset is common but its meaning is not consistent. The draft’s own survey notes implementations that send seconds until reset, milliseconds, a Unix timestamp in seconds, or a datetime. Prefer per-provider configuration over guessing. Where you must guess, a magnitude heuristic is a reasonable fallback.

Priority Resolution and Header Parsing

A webhook dispatcher that gets a 429 (or a 503) should run a deterministic header pipeline and clamp the result. Clamping matters because the draft’s security considerations warn that a misconfigured or malicious server can send an enormous value, for example Retry-After: 1000000. Providers cap it in practice. Kit clamps Retry-After between 1 second and 24 hours, and Webhook Relay caps it at one hour.

The code below assumes lowercase header names, as Node.js’s http module provides.

Code example
interface RateLimitDelay {
  delayMs: number;
  source:
    | 'Retry-After-Seconds'
    | 'Retry-After-Date'
    | 'RateLimit'
    | 'X-RateLimit-Reset'
    | 'Default-Backoff';
}

type Headers = Record<string, string | string[] | undefined>;

/**
 * Parses an IETF-draft `RateLimit` field (simplified Structured Fields parsing),
 * e.g. `"default";r=0;t=30`. Returns the longest wait among exhausted policies (r=0).
 */
function parseRateLimitWaitSeconds(value: string): number | null {
  let wait: number | null = null;
  const items = value.matchAll(
    /"[^"]*"((?:\s*;\s*[a-z*][a-z0-9_.*-]*(?:=[^;,\s]+)?)*)/g
  );
  for (const item of items) {
    const params = item[1];
    const r = /;\s*r=(\d+)/.exec(params);
    const t = /;\s*t=(\d+)/.exec(params);
    if (r && t && Number(r[1]) === 0) {
      wait = Math.max(wait ?? 0, Number(t[1]));
    }
  }
  return wait;
}

export function calculateRateLimitDelay(
  headers: Headers,
  defaultBackoffMs: number,
  nowMs: number = Date.now(),
  maxDelayMs: number = 60 * 60 * 1000 // cap untrusted values (1 hour)
): RateLimitDelay {
  const getHeader = (name: string): string | undefined => {
    const val = headers[name];
    return Array.isArray(val) ? val[0] : val;
  };
  const clamp = (ms: number) => Math.min(Math.max(ms, 0), maxDelayMs);

  // 1. Retry-After (RFC 9110): delay-seconds or HTTP-date. Takes precedence.
  const retryAfter = getHeader('retry-after')?.trim();
  if (retryAfter) {
    if (/^\d+$/.test(retryAfter)) {
      return { delayMs: clamp(Number(retryAfter) * 1000), source: 'Retry-After-Seconds' };
    }
    const dateMs = Date.parse(retryAfter);
    if (!Number.isNaN(dateMs)) {
      return { delayMs: clamp(dateMs - nowMs), source: 'Retry-After-Date' };
    }
  }

  // 2. IETF draft RateLimit field (r = remaining quota, t = seconds in window)
  const rateLimit = getHeader('ratelimit');
  if (rateLimit) {
    const seconds = parseRateLimitWaitSeconds(rateLimit);
    if (seconds !== null) {
      return { delayMs: clamp(seconds * 1000), source: 'RateLimit' };
    }
  }

  // 3. Legacy X-RateLimit-Reset: epoch ms, epoch seconds, or delta seconds
  const xReset = getHeader('x-ratelimit-reset')?.trim();
  if (xReset && /^\d+$/.test(xReset)) {
    const n = Number(xReset);
    if (n > 1e12) return { delayMs: clamp(n - nowMs), source: 'X-RateLimit-Reset' };
    if (n > 1e9) return { delayMs: clamp(n * 1000 - nowMs), source: 'X-RateLimit-Reset' };
    return { delayMs: clamp(n * 1000), source: 'X-RateLimit-Reset' };
  }

  return { delayMs: defaultBackoffMs, source: 'Default-Backoff' };
}

When the server tells every client to return at the same instant, you get a thundering herd. The IETF draft warns that this applies to Retry-After too and suggests adding jitter to the window. A good rule is to add random positive jitter on top of the server’s value, never subtract from it.

Integrating Dynamic Cooldowns into Scheduling Queues

Once you have the delay, the task should not occupy a worker. Requeue it in a delayed-scheduling structure:

  • Redis sorted sets. Store each task with a score of now_ms + delayMs. Workers poll for due tasks with ZRANGE retries -inf <now_ms> BYSCORE LIMIT 0 100. The older ZRANGEBYSCORE command has been deprecated since Redis 6.2 in favor of ZRANGE ... BYSCORE. Claim each member with ZREM and process it only if ZREM returns 1, so two workers never take the same task. A worker that crashes after claiming a task loses it unless you add a lease or visibility-timeout pattern.
  • Amazon SQS. The delay is limited to 15 minutes (900 seconds), set with DelaySeconds on the queue or per message. FIFO queues do not support per-message timers, and MessageGroupId is a FIFO concept unrelated to delay. AWS recommends EventBridge Scheduler for delays beyond 15 minutes.
  • RabbitMQ. The rabbitmq_delayed_message_exchange plugin (exchange type x-delayed-message, delay in the x-delay header, in milliseconds) is a community plugin that does not ship with RabbitMQ. Its repository says it is no longer actively maintained and describes limitations, including delayed messages being held in a Mnesia table with a single disk replica. Evaluate it carefully for new deployments. A common alternative is a set of TTL queues, one per delay tier, that dead-letter into the work queue.

2. Partial Batch Failures: Isolating Bad Events Inside Multi-Event Payloads

To lower connection overhead and raise throughput, dispatchers often combine multiple events for the same endpoint into one request:

Code example
{
  "batch_id": "btc_9081237",
  "events": [
    { "id": "evt_001", "type": "order.created", "payload": { "order_id": 101 } },
    { "id": "evt_002", "type": "order.updated", "payload": { "order_id": 102 } },
    { "id": "evt_003", "type": "user.deleted",  "payload": { "user_id": null } }
  ]
}

The All-or-Nothing Problem

Suppose evt_003 carries data that triggers an unhandled exception on the customer’s server, and the server answers the whole batch with HTTP 500. If the dispatcher treats that as a single binary outcome and retries the entire payload:

  1. evt_001 and evt_002 are delivered again, producing duplicate side effects unless the receiver deduplicates.
  2. evt_003 keeps failing, so its healthy siblings are stuck in the retry loop until the batch exhausts its retry limit.

Real systems already behave this way. Kit’s documentation states that a retry re-sends the delivery’s entire event array under the same delivery_id, and it tells receivers to deduplicate on each event’s UUID id.

Code example
              UNHANDLED BATCH RETRY (BAD)
+-------------------------------------------------------+
|  Batch [evt_001 (OK), evt_002 (OK), evt_003 (FAIL)]   |
+-------------------------------------------------------+
                           |
                     Delivery Try #1 -> HTTP 500
                           |
                     Delivery Try #2 -> HTTP 500
  (evt_001 and evt_002 re-executed, causing duplication!)

Mitigation Strategies

Pattern A: Receiver-Side Inbox and Deduplication Keys

Every event carries a stable unique id, and the receiver records it in an inbox table with a unique constraint. Processing each event in its own transaction (or savepoint) lets one failure leave the others committed.

Code example
-- PostgreSQL / SQLite syntax
BEGIN;

INSERT INTO inbox_events (event_id, received_at)
VALUES ('evt_001', NOW())
ON CONFLICT (event_id) DO NOTHING;

-- If 0 rows were inserted, the event was already processed: skip it.
-- Otherwise apply the business logic here, in the SAME transaction.

COMMIT;

Recording the ID and applying the side effect in one transaction is what makes the pattern safe. If you commit the inbox row first and the side effect fails afterward, the event is silently lost on retry.

The sender’s job is to keep the same event ID across every attempt and to treat delivery as at-least-once.

Pattern B: Itemized Partial-Success Responses

Some APIs return a per-event result map in the response body. HTTP 207 Multi-Status is sometimes borrowed for this. It originates in WebDAV (RFC 4918), and no webhook standard defines it for batches, so this is a contract you agree with your receivers rather than something to assume. A 207 is a 2xx, which most senders treat as plain success, so your dispatcher must explicitly parse the body or failed items will be silently counted as delivered.

Code example
HTTP/1.1 207 Multi-Status
Content-Type: application/json

{
  "batch_id": "btc_9081237",
  "summary": { "succeeded": 2, "failed": 1 },
  "results": [
    { "id": "evt_001", "status": 200 },
    { "id": "evt_002", "status": 200 },
    { "id": "evt_003", "status": 422, "error": "Invalid user_id null payload" }
  ]
}

When the dispatcher receives an itemized response:

  1. It marks evt_001 and evt_002 as DELIVERED.
  2. It retries evt_003 alone if its status is retryable, or sends it straight to the dead-letter queue (DLQ) if it is not (such as 422).

Pattern C: Dispatcher-Side Atomization

For endpoints that return no itemized results, fall back to splitting the batch:

Code example
interface WebhookEvent {
  id: string;
  type: string;
  payload: Record<string, unknown>;
}

interface BatchDeliveryPayload {
  batchId: string;
  events: WebhookEvent[];
}

interface ErrorResponse {
  statusCode: number;
  body: string;
}

async function handleBatchDeliveryFailure(
  batch: BatchDeliveryPayload,
  error: ErrorResponse,
  enqueueSingleEvent: (event: WebhookEvent, delayMs: number) => Promise<void>,
  processSingleEventFailure: (event: WebhookEvent, error: ErrorResponse) => Promise<void>
): Promise<void> {
  if (batch.events.length > 1) {
    console.warn(
      `Batch ${batch.batchId} failed with HTTP ${error.statusCode}. ` +
      `Splitting ${batch.events.length} events into single-event deliveries.`
    );
    for (const event of batch.events) {
      const jitterMs = Math.floor(Math.random() * 500);
      await enqueueSingleEvent(event, jitterMs);
    }
  } else {
    // A single event failed: run the normal classification path (see Section 4).
    await processSingleEventFailure(batch.events[0], error);
  }
}

Atomization multiplies request volume. If the original failure was an outage rather than a poisoned event, splitting a 100-event batch sends 100 requests to an endpoint that is already struggling. Only atomize on failures that plausibly point to payload content. Let 429, 502, 503 and 504 follow the normal retry path, and route the split single-event deliveries through the per-endpoint rate limiter described in Section 3. A bisection strategy (splitting into halves repeatedly) is a lower-amplification variant.


3. Multi-Tenant Retry Storm Containment: Token Buckets vs. Circuit Breakers

In multi-tenant systems, thousands of customer webhooks share queue infrastructure and worker pools. If Tenant A’s receiver goes down, its backlog generates a flood of failed attempts. Without isolation, those retries saturate the shared consumers and delay deliveries for healthy tenants. This is a form of head-of-line blocking, often called worker starvation.

Code example
SHARED WORKER POOL (WITHOUT CONTAINMENT)
[ Tenant A Retry ] [ Tenant A Retry ] [ Tenant A Retry ] [ Tenant A Retry ]
                      ^ (pool saturated by one tenant)
[ Tenant B Normal ] -- STARVED IN QUEUE

Containment Mechanics

MechanismOperating LayerPrimary ObjectiveProsCons
Circuit breakerPer endpoint or domainStop sending to a destination that is clearly down.Saves worker time and network connections.Thresholds can misfire; breaking per domain can block healthy endpoints that share a host.
Per-endpoint outbound token bucketDispatcher rate limiterCap delivery velocity per destination.Prevents overwhelming a receiver, including during recovery.Needs shared state (e.g., Redis) across workers.
Separate retry and real-time queuesQueue infrastructureKeep retry traffic from delaying first-attempt deliveries.Protects fresh-traffic latency regardless of retry volume.More queues to operate and monitor.

Per-Endpoint Token Buckets

Before a worker calls https://api.customer-a.com/webhook, it must take a token from the bucket for that destination. Tokens refill continuously over the elapsed interval Δt\Delta t Δt:

Tavailable=min⁡(Bcapacity,Tcurrent+Δt×Rfill)T_{\text{available}} = \min\left(B_{\text{capacity}}, T_{\text{current}} + \Delta t \times R_{\text{fill}}\right)Tavailable​=min(Bcapacity​,Tcurrent​+Δt×Rfill​)

  • BcapacityB_{\text{capacity}} Bcapacity​ is the maximum burst size.
  • RfillR_{\text{fill}} Rfill​ is the refill rate in tokens per second.

An atomic Lua script keeps the read-modify-write safe across many concurrent workers. This version reads the clock from Redis itself (TIME, safe in scripts on Redis 5.0+) so worker clock skew cannot corrupt the bucket. It also uses HSET, because HMSET has been deprecated since Redis 4.0. On denial it returns how long to wait, so the task can be rescheduled precisely instead of spinning.

Code example
-- KEYS[1]: bucket key, e.g. "ratelimit:endpoint:api.customer-a.com"
-- ARGV[1]: capacity (max burst), e.g. 20
-- ARGV[2]: refill rate in tokens per second, e.g. 5
-- ARGV[3]: tokens requested, e.g. 1
-- Returns: { allowed (1 or 0), retry_after_ms }

local capacity  = tonumber(ARGV[1])
local fill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])

local t = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local bucket = redis.call("HMGET", KEYS[1], "tokens", "last_updated")
local tokens = tonumber(bucket[1])
local last_updated = tonumber(bucket[2])

if tokens == nil then
  tokens = capacity
else
  tokens = math.min(capacity, tokens + math.max(0, now - last_updated) * fill_rate)
end

local allowed = 0
local retry_after_ms = 0
if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
else
  retry_after_ms = math.ceil((requested - tokens) / fill_rate * 1000)
end

redis.call("HSET", KEYS[1], "tokens", tokens, "last_updated", now)
redis.call("EXPIRE", KEYS[1], math.max(1, math.ceil(capacity / fill_rate) * 2))

return { allowed, retry_after_ms }

If a receiver publishes its own limits through Retry-After or the IETF RateLimit fields, feed that information back into the bucket so your outbound rate tracks what the receiver says it can handle.

Circuit Breakers for Endpoint Cooldowns

When an endpoint fails many consecutive requests, or returns repeated 5xx responses or timeouts within a window, a circuit breaker trips to OPEN.

Code example
                 failure threshold reached
  +----------+ ---------------------------> +----------+
  |  CLOSED  |                              |   OPEN   |
  +----------+ <--+                         +----------+
       ^         |                               |
       |         | probe fails                   | cooldown elapsed
       |         |      +-------------+          |
       |         +----- |  HALF-OPEN  | <--------+
       |                +-------------+
       |   probe succeeds     |
       +----------------------+
  1. CLOSED. Deliveries flow normally while failures are counted over a rolling window.
  2. OPEN. Live requests to the endpoint are suppressed. Events are not dropped; they are parked in a delayed queue so no worker time or connections are spent on a dead destination.
  3. HALF-OPEN. After a cooldown (for example, several minutes), a limited number of probe requests are allowed. If they succeed, the breaker returns to CLOSED. If they fail, it returns to OPEN and the cooldown restarts.

Pair the breaker with a longer-term health policy. Stripe, for example, emails account owners when an endpoint has not returned a 2xx for multiple days and discloses when it will be automatically disabled. Always pair auto-disabling with notification and a way to re-enable and replay.


4. Deterministic Error Classification: Transient vs. Permanent

A common weakness in failure handling is putting every non-2xx response and network error on the same retry schedule.

Retrying a 401 Unauthorized response ten times over a day consumes worker capacity and fills your logs, and an invalid credential is unlikely to fix itself. But “permanent” is a policy decision, not a fact, and providers differ. Stripe retries any non-2xx response, while Standard Webhooks-style senders such as Kit stop immediately only on 410 Gone. A receiver that rolls out a fix or rotates a secret can recover within hours.

A practical middle path is to stop automatic retries early for errors that need human action, but never discard the payload. Quarantine it, notify the customer and allow replay after the fix.

Code example
                            OUTBOUND WEBHOOK RESPONSE
                                        |
                   +--------------------+--------------------+
                   |                                         |
         HTTP Status Code / Network                      Response Body
                   |                                         |
        +----------+----------+                   +----------+----------+
        |                     |                   |                     |
   Success (2xx)        Failure (non-2xx)    Structured error      Unstructured
        |                     |               (RFC 9457 JSON)         text
   [ Delivered ]    +---------+---------+            |                  |
                    |                   |      [ Match known     [ Don't guess:
              Transient           Permanent       error codes ]    use status ]
        (408/429/5xx/network)   (most other 4xx)
                    |                   |
             [ Retry queue ]    [ Quarantine / DLQ ]

Classification Matrix

CategoryTriggerAction
Transient429 Too Many RequestsRetry, honoring Retry-After / RateLimit
Transient503 Service UnavailableRetry, honoring Retry-After if present
Transient500, 502, 504, 408Exponential backoff with jitter
TransientETIMEDOUT, ECONNRESET, EAI_AGAIN (temporary DNS failure)Backoff with jitter
AmbiguousENOTFOUND (DNS name did not resolve)Limited retries with long delays, then escalate
Ambiguous404 Not FoundLimited retries, then disable the endpoint after sustained failure
Permanent410 GoneDisable the endpoint; no retries (the receiver is explicitly asking you to stop)
Permanent400, 422, 415Quarantine
Permanent401, 403Quarantine and notify the customer
Permanent413 Content Too LargeQuarantine, or split the batch and resend
PermanentCertificate errors (expired, self-signed, hostname mismatch)Quarantine and notify

Response-Body Heuristics (RFC 9457 Problem Details)

Status codes don’t tell the whole story. Many APIs return 4xx or 5xx with a structured JSON body. The standard format is RFC 9457, Problem Details for HTTP APIs, which obsoleted RFC 7807 in 2023. Problem documents can carry extension members such as an application error code:

Code example
{
  "type": "https://api.example.com/errors/invalid-signature",
  "title": "Invalid Webhook HMAC Signature",
  "status": 400,
  "detail": "The calculated HMAC-SHA256 signature did not match the signature header.",
  "code": "ERR_HMAC_MISMATCH"
}

If a customer’s endpoint reports a signature mismatch, retrying will not help until the secret is corrected. Before blaming the customer, rule out sender-side causes such as an in-progress secret rotation. Sign each attempt freshly, with a new timestamp and the currently valid secrets, as Kit’s retry behavior does.

The IETF rate-limit draft also defines problem types registered under https://iana.org/assignments/http-problem-types: quota-exceeded and abnormal-usage-detected (both recommended with 429), and temporary-reduced-capacity (recommended with 503). Receivers that adopt them give you machine-readable transient signals.

Match on parsed, structured fields (type, code) against an explicit allowlist. Avoid substring searches over free-form text. A phrase like unauthorized appearing in a 5xx page is not a reliable permanent-failure signal.

Deterministic Failure Classifier

Code example
import { calculateRateLimitDelay } from './rate-limit'; // from Section 1

export enum FailureAction {
  RETRY_HEADER_AWARE = 'RETRY_HEADER_AWARE',
  RETRY_EXPONENTIAL = 'RETRY_EXPONENTIAL',
  QUARANTINE_PERMANENT = 'QUARANTINE_PERMANENT',
  DISABLE_ENDPOINT = 'DISABLE_ENDPOINT',
}

export interface ClassificationResult {
  action: FailureAction;
  reason: string;
  suggestedDelayMs?: number;
}

type Headers = Record<string, string | string[] | undefined>;

export class WebhookErrorClassifier {
  // Codes that your integration contract defines as unrecoverable.
  private static PERMANENT_ERROR_CODES = new Set([
    'ERR_HMAC_MISMATCH',
    'ERR_UNKNOWN_EVENT_TYPE',
    'ERR_SCHEMA_VALIDATION',
  ]);
  private static PERMANENT_STATUS_CODES = new Set([400, 401, 403, 413, 415, 422]);
  private static TRANSIENT_STATUS_CODES = new Set([408, 500, 502, 503, 504]);
  private static PERMANENT_NETWORK_CODES = new Set([
    'DEPTH_ZERO_SELF_SIGNED_CERT',
    'CERT_HAS_EXPIRED',
    'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
    'ERR_TLS_CERT_ALTNAME_INVALID',
  ]);

  private static extractErrorCode(body: string | null): string | null {
    if (!body) return null;
    try {
      const parsed = JSON.parse(body);
      return typeof parsed?.code === 'string' ? parsed.code : null;
    } catch {
      return null; // not JSON: do not guess from free text
    }
  }

  public static classify(
    statusCode: number | null,
    body: string | null,
    networkErrorCode: string | null,
    headers: Headers,
    defaultBackoffMs = 30_000
  ): ClassificationResult {
    // 1. Low-level network faults
    if (networkErrorCode) {
      if (this.PERMANENT_NETWORK_CODES.has(networkErrorCode)) {
        return {
          action: FailureAction.QUARANTINE_PERMANENT,
          reason: `TLS certificate validation failed: ${networkErrorCode}`,
        };
      }
      return {
        action: FailureAction.RETRY_EXPONENTIAL,
        reason: `Network fault: ${networkErrorCode}`, // ECONNRESET, ETIMEDOUT, EAI_AGAIN, ENOTFOUND...
      };
    }

    // 2. Explicit "stop sending" signal
    if (statusCode === 410) {
      return {
        action: FailureAction.DISABLE_ENDPOINT,
        reason: 'Receiver returned 410 Gone: endpoint no longer wants webhooks.',
      };
    }

    // 3. Throttling and unavailability: honor the receiver's own timing
    if (statusCode === 429 || (statusCode === 503 && headers['retry-after'])) {
      const { delayMs, source } = calculateRateLimitDelay(headers, defaultBackoffMs);
      return {
        action: FailureAction.RETRY_HEADER_AWARE,
        reason: `HTTP ${statusCode}; delay from ${source}.`,
        suggestedDelayMs: delayMs,
      };
    }

    // 4. Structured body markers (any status): explicit unrecoverable codes
    const code = this.extractErrorCode(body);
    if (code && this.PERMANENT_ERROR_CODES.has(code)) {
      return {
        action: FailureAction.QUARANTINE_PERMANENT,
        reason: `Unrecoverable error code in response body: ${code}`,
      };
    }

    // 5. Permanent status codes
    if (statusCode !== null && this.PERMANENT_STATUS_CODES.has(statusCode)) {
      return {
        action: FailureAction.QUARANTINE_PERMANENT,
        reason: `Terminal HTTP status ${statusCode}.`,
      };
    }

    // 6. Transient server errors, and 404 (ambiguous: a route may be mid-deploy).
    //    Let endpoint-health tracking disable endpoints with sustained 404s.
    if (statusCode !== null && (this.TRANSIENT_STATUS_CODES.has(statusCode) || statusCode === 404)) {
      return {
        action: FailureAction.RETRY_EXPONENTIAL,
        reason: `Retryable status ${statusCode}.`,
      };
    }

    // Default for unmapped responses (including unexpected 3xx): bounded retry
    return {
      action: FailureAction.RETRY_EXPONENTIAL,
      reason: `Unclassified response (status: ${statusCode}). Defaulting to bounded retry.`,
    };
  }
}

Note that Standard Webhooks recommends treating redirects as failures rather than following them, since a redirect can break signatures and open SSRF paths. The default branch above handles that case.

Dead-Letter Quarantine (DLQ) and Remediation

When an event is marked QUARANTINE_PERMANENT:

  1. Payload vaulting. Store the full delivery context (request headers, raw body, response status, response body and failure trace) in a searchable store such as PostgreSQL with JSONB or Elasticsearch.
  2. Customer alerting. Send an in-app alert or email digest showing the endpoint errors, with a link to the failed deliveries.
  3. Manual and bulk replay. Offer an API and UI to re-dispatch single events or batches once the customer has fixed the problem. Replays must reuse the original event IDs so receivers can deduplicate:
Code example
POST /api/v1/webhooks/quarantine/replay
Content-Type: application/json

{
  "quarantine_ids": ["q_8801", "q_8802", "q_8803"],
  "override_endpoint_url": "https://api.customer.com/v2/webhook"
}

Stripe and GitHub both offer bounded manual resend or redelivery windows, which is the same idea. Because a replay can create duplicates, receiver-side idempotency (Section 2, Pattern A) remains essential.


Complete Architectural Checklist for Production Webhook Systems

  • Header parsing: Honor Retry-After (on 429 and 503) first, then the IETF RateLimit field (r/t parameters), then legacy vendor headers, before falling back to computed backoff. Clamp every server-supplied delay to a sane maximum.
  • Jitter: Apply full jitter to exponential backoff, and add positive jitter on top of server-specified waits, to avoid synchronized retry spikes.
  • Stable event IDs: Keep the same event ID across every attempt and replay, and re-sign each attempt with a fresh timestamp.
  • Batch isolation: Support itemized partial-success responses where receivers opt in, and fall back to rate-limited atomization or bisection when a batched request fails.
  • Tenant isolation: Enforce per-endpoint token buckets in shared state, and keep retry traffic separate from first-attempt traffic.
  • Circuit breakers: Use the full CLOSED, OPEN and HALF-OPEN cycle, park events instead of dropping them, and notify customers before auto-disabling an endpoint.
  • Deterministic classification: Combine network codes, status codes and structured error bodies. Treat 410 Gone as the explicit stop signal, and be deliberate about which 4xx codes you stop retrying.
  • Operational tooling: Provide DLQ inspection, failure alerts, and replay APIs, and document your retry policy for customers (timeouts, retry window, backoff, and what happens on exhaustion).