InstaWebhook
November 19, 2026By InstaWebhook TeamWebhook Reliability

Delivery Timelines & State Machine Forensics: Mastering Reliable Webhook Infrastructure

Debug stuck queued webhook jobs, prevent silent data loss from early 200 OK acks, and stop duplicate delivery storms with state machine forensics and worker lease heartbeats.

Delivery Timelines & State Machine Forensics: Mastering Reliable Webhook Infrastructure

Delivery Timelines & State Machine Forensics: Mastering Reliable Webhook Infrastructure

In distributed systems, webhooks look deceptively simple on the surface: Service A sends an HTTP POST request with a JSON payload to Service B, and Service B returns a 2xx. In production, however, webhook pipelines are prone to network partitions, sudden pod terminations, database deadlocks, and thundering herds. When events go missing or get processed twice, high-level HTTP metrics often show healthy green lines while silent data corruption accumulates underneath.

Solving these failure modes requires moving beyond simple “sent vs. failed” status flags. Engineering teams should treat webhook delivery as an explicit, observable delivery state machine, with clearly defined acknowledgment boundaries and lease management.

This guide covers three foundational pillars of webhook delivery forensics: diagnosing stuck jobs across granular state transitions, fixing premature 200 OK acknowledgment traps, and preventing double-execution storms during worker restarts.

What the Major Providers Actually Do

Before designing either side of a webhook pipeline, it helps to know the contracts that real senders impose. These differ widely, and none of them is a universal standard.

ProviderResponse deadlineAutomatic retriesDuplicate-detection key
StripeNo exact timeout published; docs ask you to return a 2xx quickly, before any complex logicIn live mode, exponential backoff for up to three days; in sandbox, three attempts over a few hours. Any non-2xx response triggers a retry, including 4xxEvent id in the payload
GitHub2xx within 10 seconds, otherwise the connection is terminated and the delivery is recorded as failedNone. Failed deliveries must be redelivered manually or through the REST APIX-GitHub-Delivery header, unchanged on redelivery
Shopify1 second to connect, 5 seconds total; only 2xx counts as success (a 3xx redirect is a failure)Up to 8 retries over about 4 hours, after which delivery stops and the subscription may be removedX-Shopify-Webhook-Id, kept the same across retries

Two lessons follow. First, the sender’s retry budget determines how much outage a receiver can survive. GitHub gives you no automatic safety net, and Shopify’s budget is only a few hours. Second, every sender gives you a stable delivery or event ID. Using it for deduplication is the cheapest reliability measure available.

1. Anatomy of a Webhook Delivery Timeline: Diagnosing Stuck Queued vs. Leased Jobs

Many webhook engines track delivery using a naive boolean (delivered: true/false) or a three-state enum (pending, success, failed). When latency spikes or deliveries stall, a generic pending status hides the root cause. Is the broker backlogged, or are workers crashing mid-flight?

One practical design for the sending side is a lifecycle like this, with a terminal delivered state for success:

received → queued → leased → attempted → retried → dead-lettered

(A successful attempted job moves to delivered; a failed one moves to retried, and eventually to dead-lettered.) This is a design pattern, not an industry standard, so adapt the names to your system. What matters is that waiting, claimed, and in-flight jobs are distinguishable.

The Granular State Machine

StateTrigger / Entry ConditionOwnershipKey Timestamp RecordedPrimary Failure Mode
receivedEvent accepted into the outbox or ingestion API, ideally within the producer’s own database transaction.Ingestion Layercreated_atOutbox poller / CDC lag preventing promotion to the queue.
queuedDispatcher publishes a job reference to the message broker (e.g., Kafka, SQS, Redis Streams) or the job becomes eligible in a Postgres table polled with FOR UPDATE SKIP LOCKED.Queue / Brokerenqueued_atConsumer starvation, partition skew, or per-tenant rate-limiting backoff.
leasedA worker claims the job and acquires a time-bounded lock (lease_expires_at).Delivery Workerleased_atWorker OOM kill, network hang, or DNS stall without an explicit NACK.
attemptedWorker opens the TLS connection and sends the HTTP POST to the subscriber’s endpoint.Delivery Workerattempted_atSlow subscriber responses (tarpitting), connection resets, or 5xx errors.
retriedDelivery failed with a retryable outcome (typically timeouts, 408, 429, 5xx); rescheduled with exponential backoff plus jitter.Scheduler / Queuenext_attempt_atRetry storms amplifying downstream outages when jitter is omitted.
dead-letteredExhausted max_attempts, or hit a terminal error your policy treats as non-retryable (e.g., 410 Gone).DLQ Storagedead_lettered_atPermanent schema mismatch or a dead endpoint that needs manual replay.

Which status codes count as retryable is a policy choice. Stripe, for example, retries on any non-2xx response, while many senders treat 410 Gone as a signal to stop and disable the endpoint. Document your policy and make it visible to customers.

Diagnosing Queued Latency vs. Leased Stalls

When end-to-end delivery latency climbs, comparing the deltas between enqueued_at, leased_at, and attempted_at quickly isolates the failing subsystem.

Scenario A: Jobs Stuck in Queued (leased_at - enqueued_at is growing)

When jobs accumulate in queued while leased counts stay flat, the bottleneck is dispatch capacity or concurrency throttling:

  • Worker Pool Exhaustion: All worker threads or goroutines are blocked on slow external endpoints. Suppose you run 500 workers and one customer’s endpoint takes 30 seconds to time out. Then 500 concurrent events for that tenant can occupy every worker slot and starve all other tenants (the “noisy neighbor” problem).
  • Partition Hotspotting: In ordered webhook systems (e.g., partitioning by customer_id or account_id), a burst of 100,000 events for one tenant blocks everything behind it in that partition.
  • Fix: Implement per-tenant concurrency limits (semaphores) and move consistently slow endpoints into a separate lower-priority queue so they cannot starve healthy ones.

Scenario B: Jobs Stuck in Leased (now() - leased_at exceeds the expected HTTP timeout)

A job in the leased state means a worker claimed it, yet no transition to attempted or retried has occurred. This points to worker-side execution hangs:

  • Unbounded HTTP Client Timeouts: HTTP clients without explicit dial, TLS-handshake, and response-header timeouts (or an overall request deadline) let a slow or malicious server accept a connection and then trickle bytes, holding the worker for a very long time. In Go, for instance, the default http.Client has no overall timeout.
  • Silent Worker Crashes (SIGKILL / OOM): If a container is OOM-killed while serializing a large payload, it dies without sending a NACK or updating the database row. The job stays leased until its lease expires.
  • Connection Pool Deadlocks: A worker that holds a database connection across the outbound HTTP call, then requests a second connection to record the result, can exhaust the pool and deadlock under high concurrency. Keep database transactions short and never hold them across network calls.

Forensics SQL: Spotting Stalled Queues Before Alerts Fire

If your state machine is backed by a relational job table (such as PostgreSQL), a lightweight diagnostic query run every 30 seconds can reveal stalls before queue depth triggers a page:

Code example
SELECT
    status,
    COUNT(*) AS job_count,
    MAX(EXTRACT(EPOCH FROM (NOW() - updated_at))) AS max_age_seconds,
    COUNT(*) FILTER (
        WHERE status = 'leased' AND lease_expires_at < NOW()
    ) AS expired_leases
FROM webhook_deliveries
WHERE status IN ('received', 'queued', 'leased', 'retried')
GROUP BY status;

How to read the output:

  • High max_age_seconds in received suggests the transactional outbox relay (or CDC connector) has stalled.
  • High max_age_seconds in queued suggests consumer starvation or per-tenant rate limiting.
  • Any non-zero expired_leases means workers are crashing, timing out, or pausing (for example during garbage collection) for longer than the lease window.

Metrics Worth Exporting

Alongside the query above, export these as time series so you can alert on trends rather than snapshots:

  • Queue wait time (leased_at - enqueued_at) as a histogram, not an average. Tail latency is where tenant starvation shows up first.
  • Attempt duration and outcome by status class and by tenant or endpoint.
  • Lease expirations per minute, and the share of expirations where the original worker later completed anyway (a direct measure of duplicate-delivery risk).
  • Dead-letter arrivals and replay counts.

2. Why Returning 200 OK Early Is a Silent Data-Loss Trap (And How to Fix Your Ack Boundary)

On the receiving end of a webhook pipeline, the standard advice is to respond with a 2xx quickly so the sender doesn’t time out. The advice is sound and comes straight from provider documentation: GitHub wants a response within 10 seconds, Shopify within 5, and Stripe asks you to return a 2xx before running complex logic.

The danger is reading “quickly” as “immediately, before doing anything durable.” That produces one of the most destructive anti-patterns in webhook handling: acknowledging an HTTP request before durably persisting the payload.

The Premature 200 OK Failure Mode

Consider a receiver that fires off an in-process background task and immediately returns 200 OK:

Code example
# ANTI-PATTERN: Silent Data-Loss Trap
@app.post("/webhooks/payments")
async def handle_webhook(request: Request, background_tasks: BackgroundTasks):
    raw_body = await request.body()
    verify_signature(request.headers, raw_body)  # signatures cover the raw bytes
    payload = json.loads(raw_body)

    # Queued in volatile process memory!
    background_tasks.add_task(process_payment_event, payload)

    # Sender marks the event as DELIVERED and may never retry it
    return JSONResponse(status_code=200, content={"status": "accepted"})

(Signature verification is done over the raw request body, not a re-serialized JSON object, because re-serialization can change the bytes and break the HMAC.)

This pattern creates an unrecoverable data-loss window between returning the 200 OK and completing the side effect:

  1. Pod Eviction or Rolling Deploy: Kubernetes sends SIGTERM during a deployment. The response has already been sent, so the provider marks the delivery successful and stops retrying. When the pod’s grace period expires (30 seconds by default), SIGKILL terminates the process and the in-memory task queue disappears.
  2. Downstream Transient Failure: process_payment_event writes to your primary database during a brief failover and throws. Because you already returned 200 OK, the sender has no idea the event failed. Unless you built a separate dead-letter mechanism inside the worker, the event is lost.
  3. Unbounded Memory Growth (OOM): During a traffic spike, webhooks arrive faster than background tasks execute. In-memory tasks pile up until the container hits its memory limit and crashes, taking every unprocessed event with it.

Defining the Safe Acknowledgment Boundary

The acknowledgment boundary is the point in your ingestion pipeline where responsibility for the event’s durability transfers from the sender to the receiver.

A receiver should not return a 2xx status until the raw webhook payload has been committed to durable storage, such as a relational database transaction, or a replicated message broker (for example Kafka with acks=all and a sensible min.insync.replicas).

Code example
[Sender] ──POST /webhook──▶ [Receiver API]
                                  │
                     1. Verify HMAC signature (raw body)
                     2. INSERT INTO webhook_inbox (event_id, payload)
                     3. COMMIT transaction  ◀── [SAFE ACK BOUNDARY]
                                  │
[Sender] ◀────2xx─────────────────┘
                                  │
                     4. Async worker polls inbox & executes business logic

Decoupling durable ingestion from business-logic execution gives you speed and durability together:

  • Fast response: Inserting one row into an inbox table or publishing to a durable queue is cheap compared with running business logic. It fits comfortably inside the tightest published deadlines, such as Shopify’s 5 seconds and Slack’s 3 seconds for its Events API. Measure your own p99 rather than assuming.
  • Safe failure behavior: If your database or broker is unavailable, the commit fails and your API should return a 5xx (such as 503 Service Unavailable). Whether that is safe depends on the sender. Stripe and Shopify will retry within their windows, but GitHub does not retry automatically, so a rejected delivery there must be redelivered by you, manually or via its API.

Implementation: The Transactional Inbox Pattern

Instead of executing business logic inside the HTTP handler, write the raw payload into an inbox table with a unique constraint on the sender’s event ID:

Code example
CREATE TABLE webhook_inbox (
    event_id     VARCHAR(255) PRIMARY KEY,
    provider     VARCHAR(64)  NOT NULL,
    payload      JSONB        NOT NULL,
    status       VARCHAR(32)  NOT NULL DEFAULT 'pending',
    received_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    processed_at TIMESTAMPTZ
);

In your HTTP handler, validate the signature and perform an idempotent insert:

Code example
INSERT INTO webhook_inbox (event_id, provider, payload)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING;

Only after the driver confirms the commit does the endpoint return 200 OK (or 202 Accepted; note that Shopify requires a 200-series code, and a 3xx counts as failure). Once persisted, an internal worker pool processes the row asynchronously, using your own retry and dead-letter infrastructure if downstream services fail.

Because providers can deliver the same event more than once, the primary key doubles as a first line of deduplication. For senders without automatic retries, or for events that arrived while you were down, pair the inbox with a periodic reconciliation job that lists recent events through the provider’s API (for example Stripe’s Events API) and fills any gaps.

3. Tracking Worker Heartbeats and Lease Expiration to Prevent Double-Execution Storms

Once you introduce durable queues and worker pools on either side of a webhook pipeline, a new failure mode appears: duplicate execution during worker restarts, network partitions, or garbage-collection pauses.

How Short Leases and Restarts Trigger Double-Execution Storms

To make sure crashed workers don’t hold jobs forever, queue systems rely on visibility timeouts (leases). Amazon SQS, for example, hides a received message for a visibility timeout (30 seconds by default) and makes it visible again if it isn’t deleted in time. A Postgres-backed queue can emulate this with a lease_expires_at column. In both cases, a job that outlives its lease is handed to someone else.

Consider this timeline:

  1. The Slow Endpoint: Worker A leases Event #409 with a 30-second lease and sends an HTTP POST to a subscriber whose endpoint takes 35 seconds to respond.
  2. The Premature Expiration: At t = 30s, Worker A is still waiting on the socket. The queue sees lease_expires_at < NOW() and hands Event #409 to Worker B.
  3. The Duplicate Delivery: At t = 31s, Worker B sends a second identical POST. At t = 35s, Worker A receives a 200 OK for the first request. The customer has now processed the webhook twice.

During a rolling deployment across dozens of worker pods, if in-flight requests outlast the lease or pods terminate mid-request without draining, many leases can expire at once. The resulting double-execution storm hits downstream APIs at the moment your cluster is least stable.

Mechanism 1: Fencing Tokens and Heartbeat Lease Renewal

Static lease timeouts force a trade-off:

  • Too short (e.g., 10 seconds): Any slow HTTP call or brief GC pause expires the lease and causes duplicate execution.
  • Too long (e.g., 10 minutes): If a worker is killed, the event sits stalled for 10 minutes before another worker picks it up.

A common solution is a short initial lease (e.g., 15–30 seconds) paired with a background heartbeat that extends it, and a monotonically increasing fencing token (here, lease_epoch). With SQS you would use ChangeMessageVisibility for the same purpose.

Step 1: Claiming with a Fencing Token

When a worker leases a job, it atomically increments lease_epoch and sets a short lease_expires_at:

Code example
UPDATE webhook_deliveries
SET
    status = 'leased',
    worker_id = 'worker-pod-84a',
    lease_epoch = lease_epoch + 1,
    lease_expires_at = NOW() + INTERVAL '20 seconds',
    leased_at = NOW(),
    updated_at = NOW()
WHERE id = (
    SELECT id FROM webhook_deliveries
    WHERE (status IN ('queued', 'retried') AND next_attempt_at <= NOW())
       OR (status = 'leased' AND lease_expires_at < NOW())
    ORDER BY next_attempt_at ASC
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
RETURNING id, payload, lease_epoch;

FOR UPDATE SKIP LOCKED lets concurrent workers claim different rows without blocking one another, which is why it is a popular basis for Postgres-backed queues.

Step 2: Background Heartbeat Extension

While the worker performs the outbound HTTP call, a background timer fires roughly every lease_duration / 3 (about every 7 seconds for a 20-second lease) and extends the lease, strictly conditioned on matching both worker_id and lease_epoch:

Code example
UPDATE webhook_deliveries
SET
    lease_expires_at = NOW() + INTERVAL '20 seconds',
    updated_at = NOW()
WHERE id = $1
  AND worker_id = $2
  AND lease_epoch = $3
  AND status = 'leased';

If this UPDATE affects 0 rows, the worker knows it lost its lease and should abort its in-flight work (for example via context cancellation) instead of continuing as a zombie.

Step 3: Fence the Completion Write

The same epoch check must guard the final state transition, otherwise a zombie worker can overwrite the result recorded by the new lease holder:

Code example
UPDATE webhook_deliveries
SET status = 'delivered', delivered_at = NOW(), updated_at = NOW()
WHERE id = $1
  AND lease_epoch = $2
  AND status = 'leased';

One caveat: fencing protects your own state table. It cannot recall an HTTP request that has already left the building. A zombie worker aborting early reduces wasted work, but the subscriber may still receive the event twice. That is why Mechanism 2 below is not optional.

Step 4: Graceful Shutdown Draining (SIGTERM)

When Kubernetes terminates a pod, it sends SIGTERM and, after terminationGracePeriodSeconds (30 seconds by default), SIGKILL. Workers should:

  1. Stop polling for new leases.
  2. Let in-flight HTTP requests (and their heartbeats) finish, within a grace period you have raised above your longest expected delivery timeout.
  3. If a request cannot finish in time, release the lease explicitly, with the same ownership checks, instead of leaving it to expire:
Code example
UPDATE webhook_deliveries
SET status = 'retried', next_attempt_at = NOW(), lease_expires_at = NULL, updated_at = NOW()
WHERE id = $1 AND worker_id = $2 AND lease_epoch = $3 AND status = 'leased';

Mechanism 2: Coordinating Lease Renewal with Receiver Idempotency

Even with heartbeats, fencing tokens, and graceful shutdowns, network failures mean at-least-once delivery will occasionally deliver the same event twice. A receiver might commit a transaction and send a 200 OK that is lost before reaching the sender, and the sender, seeing no response, retries.

To make duplicates harmless, both sides must cooperate on idempotency keys:

  1. Sender-Side Stability: The sender assigns a globally unique, immutable event ID (for example a UUIDv7 or ULID) when the event first enters the received state. Across every retry, lease expiry, and worker failover, the ID in the headers and payload stays identical. Real providers follow this pattern: GitHub keeps X-GitHub-Delivery unchanged on redelivery, and Shopify keeps X-Shopify-Webhook-Id the same across retries. The Standard Webhooks specification similarly defines a webhook-id header for this purpose.
  2. Receiver-Side Atomic Deduplication: Deduplication should happen inside the same database transaction as the business state change, not in a separate non-transactional lookup (such as checking Redis before writing to Postgres), which leaves a race window.
Code example
BEGIN;

-- Claim the idempotency key inside the main transaction
INSERT INTO processed_webhooks (webhook_id, processed_at)
VALUES ('evt_01J9X8K2M4N5P6Q7R8S9T0', NOW())
ON CONFLICT (webhook_id) DO NOTHING;

-- In application code: if rows_affected == 0, another transaction already
-- handled this event, so ROLLBACK and return 200 OK.

UPDATE account_balances
SET balance_cents = balance_cents + 5000
WHERE account_id = 'acc_123';

COMMIT;

If Worker A and Worker B process the same re-leased webhook at the same time:

  • Both attempt the INSERT INTO processed_webhooks.
  • PostgreSQL makes the second inserter wait on the conflicting unique-index entry until the first transaction commits or rolls back.
  • If Worker A commits, Worker B’s insert affects zero rows. It rolls back its balance update and returns 200 OK.
  • If Worker A crashes mid-transaction, PostgreSQL rolls back its uncommitted insert, and Worker B proceeds to complete the work.

Summary Checklist for Webhook Reliability Forensics

When auditing or hardening a webhook pipeline, verify these invariants:

  1. Separate queued from leased: Don’t lump waiting and in-flight jobs into one pending state. Track enqueued_at, leased_at, and attempted_at independently to distinguish queue starvation from hung workers.
  2. Monitor expired_leases as a leading indicator: A rising count signals worker OOMs, GC pauses, or missing HTTP client timeouts long before overall delivery rates drop.
  3. Don’t return 2xx before a durable commit: Emit success responses only after the payload is committed to a transactional inbox or replicated broker, and know whether your sender will retry if you return a 5xx.
  4. Use heartbeats and fencing tokens (lease_epoch): Pair short 15–30 second leases with heartbeat renewals and epoch-guarded writes so a worker that lost its lease can’t corrupt state. Remember that fencing doesn’t unsend requests.
  5. Deduplicate on the sender’s event ID inside the business transaction: Treat retries and lease expirations as inevitable, so duplicate deliveries resolve to clean no-ops.
  6. Reconcile: Periodically compare your records with the provider’s event list. Retry windows are finite, and some senders don’t retry at all.

Sources