Delivery Timelines & State Machine Forensics: Mastering Reliable Webhook Infrastructure
Debug stuck queued webhook jobs, prevent silent data loss from early 200 OK acks, and stop duplicate delivery storms with state machine forensics and worker lease heartbeats.

Delivery Timelines & State Machine Forensics: Mastering Reliable Webhook Infrastructure
In distributed systems, webhooks look deceptively simple on the surface: Service A sends an HTTP POST request with a JSON payload to Service B, and Service B returns a 2xx. In production, however, webhook pipelines are prone to network partitions, sudden pod terminations, database deadlocks, and thundering herds. When events go missing or get processed twice, high-level HTTP metrics often show healthy green lines while silent data corruption accumulates underneath.
Solving these failure modes requires moving beyond simple “sent vs. failed” status flags. Engineering teams should treat webhook delivery as an explicit, observable delivery state machine, with clearly defined acknowledgment boundaries and lease management.
This guide covers three foundational pillars of webhook delivery forensics: diagnosing stuck jobs across granular state transitions, fixing premature 200 OK acknowledgment traps, and preventing double-execution storms during worker restarts.
What the Major Providers Actually Do
Before designing either side of a webhook pipeline, it helps to know the contracts that real senders impose. These differ widely, and none of them is a universal standard.
| Provider | Response deadline | Automatic retries | Duplicate-detection key |
|---|---|---|---|
| Stripe | No exact timeout published; docs ask you to return a 2xx quickly, before any complex logic | In live mode, exponential backoff for up to three days; in sandbox, three attempts over a few hours. Any non-2xx response triggers a retry, including 4xx | Event id in the payload |
| GitHub | 2xx within 10 seconds, otherwise the connection is terminated and the delivery is recorded as failed | None. Failed deliveries must be redelivered manually or through the REST API | X-GitHub-Delivery header, unchanged on redelivery |
| Shopify | 1 second to connect, 5 seconds total; only 2xx counts as success (a 3xx redirect is a failure) | Up to 8 retries over about 4 hours, after which delivery stops and the subscription may be removed | X-Shopify-Webhook-Id, kept the same across retries |
Two lessons follow. First, the sender’s retry budget determines how much outage a receiver can survive. GitHub gives you no automatic safety net, and Shopify’s budget is only a few hours. Second, every sender gives you a stable delivery or event ID. Using it for deduplication is the cheapest reliability measure available.
1. Anatomy of a Webhook Delivery Timeline: Diagnosing Stuck Queued vs. Leased Jobs
Many webhook engines track delivery using a naive boolean (delivered: true/false) or a three-state enum (pending, success, failed). When latency spikes or deliveries stall, a generic pending status hides the root cause. Is the broker backlogged, or are workers crashing mid-flight?
One practical design for the sending side is a lifecycle like this, with a terminal delivered state for success:
received → queued → leased → attempted → retried → dead-lettered
(A successful attempted job moves to delivered; a failed one moves to retried, and eventually to dead-lettered.) This is a design pattern, not an industry standard, so adapt the names to your system. What matters is that waiting, claimed, and in-flight jobs are distinguishable.
The Granular State Machine
| State | Trigger / Entry Condition | Ownership | Key Timestamp Recorded | Primary Failure Mode |
|---|---|---|---|---|
received | Event accepted into the outbox or ingestion API, ideally within the producer’s own database transaction. | Ingestion Layer | created_at | Outbox poller / CDC lag preventing promotion to the queue. |
queued | Dispatcher publishes a job reference to the message broker (e.g., Kafka, SQS, Redis Streams) or the job becomes eligible in a Postgres table polled with FOR UPDATE SKIP LOCKED. | Queue / Broker | enqueued_at | Consumer starvation, partition skew, or per-tenant rate-limiting backoff. |
leased | A worker claims the job and acquires a time-bounded lock (lease_expires_at). | Delivery Worker | leased_at | Worker OOM kill, network hang, or DNS stall without an explicit NACK. |
attempted | Worker opens the TLS connection and sends the HTTP POST to the subscriber’s endpoint. | Delivery Worker | attempted_at | Slow subscriber responses (tarpitting), connection resets, or 5xx errors. |
retried | Delivery failed with a retryable outcome (typically timeouts, 408, 429, 5xx); rescheduled with exponential backoff plus jitter. | Scheduler / Queue | next_attempt_at | Retry storms amplifying downstream outages when jitter is omitted. |
dead-lettered | Exhausted max_attempts, or hit a terminal error your policy treats as non-retryable (e.g., 410 Gone). | DLQ Storage | dead_lettered_at | Permanent schema mismatch or a dead endpoint that needs manual replay. |
Which status codes count as retryable is a policy choice. Stripe, for example, retries on any non-2xx response, while many senders treat 410 Gone as a signal to stop and disable the endpoint. Document your policy and make it visible to customers.
Diagnosing Queued Latency vs. Leased Stalls
When end-to-end delivery latency climbs, comparing the deltas between enqueued_at, leased_at, and attempted_at quickly isolates the failing subsystem.
Scenario A: Jobs Stuck in Queued (leased_at - enqueued_at is growing)
When jobs accumulate in queued while leased counts stay flat, the bottleneck is dispatch capacity or concurrency throttling:
- Worker Pool Exhaustion: All worker threads or goroutines are blocked on slow external endpoints. Suppose you run 500 workers and one customer’s endpoint takes 30 seconds to time out. Then 500 concurrent events for that tenant can occupy every worker slot and starve all other tenants (the “noisy neighbor” problem).
- Partition Hotspotting: In ordered webhook systems (e.g., partitioning by
customer_idoraccount_id), a burst of 100,000 events for one tenant blocks everything behind it in that partition. - Fix: Implement per-tenant concurrency limits (semaphores) and move consistently slow endpoints into a separate lower-priority queue so they cannot starve healthy ones.
Scenario B: Jobs Stuck in Leased (now() - leased_at exceeds the expected HTTP timeout)
A job in the leased state means a worker claimed it, yet no transition to attempted or retried has occurred. This points to worker-side execution hangs:
- Unbounded HTTP Client Timeouts: HTTP clients without explicit dial, TLS-handshake, and response-header timeouts (or an overall request deadline) let a slow or malicious server accept a connection and then trickle bytes, holding the worker for a very long time. In Go, for instance, the default
http.Clienthas no overall timeout. - Silent Worker Crashes (SIGKILL / OOM): If a container is OOM-killed while serializing a large payload, it dies without sending a
NACKor updating the database row. The job staysleaseduntil its lease expires. - Connection Pool Deadlocks: A worker that holds a database connection across the outbound HTTP call, then requests a second connection to record the result, can exhaust the pool and deadlock under high concurrency. Keep database transactions short and never hold them across network calls.
Forensics SQL: Spotting Stalled Queues Before Alerts Fire
If your state machine is backed by a relational job table (such as PostgreSQL), a lightweight diagnostic query run every 30 seconds can reveal stalls before queue depth triggers a page:
SELECT
status,
COUNT(*) AS job_count,
MAX(EXTRACT(EPOCH FROM (NOW() - updated_at))) AS max_age_seconds,
COUNT(*) FILTER (
WHERE status = 'leased' AND lease_expires_at < NOW()
) AS expired_leases
FROM webhook_deliveries
WHERE status IN ('received', 'queued', 'leased', 'retried')
GROUP BY status;
How to read the output:
- High
max_age_secondsinreceivedsuggests the transactional outbox relay (or CDC connector) has stalled. - High
max_age_secondsinqueuedsuggests consumer starvation or per-tenant rate limiting. - Any non-zero
expired_leasesmeans workers are crashing, timing out, or pausing (for example during garbage collection) for longer than the lease window.
Metrics Worth Exporting
Alongside the query above, export these as time series so you can alert on trends rather than snapshots:
- Queue wait time (
leased_at - enqueued_at) as a histogram, not an average. Tail latency is where tenant starvation shows up first. - Attempt duration and outcome by status class and by tenant or endpoint.
- Lease expirations per minute, and the share of expirations where the original worker later completed anyway (a direct measure of duplicate-delivery risk).
- Dead-letter arrivals and replay counts.
2. Why Returning 200 OK Early Is a Silent Data-Loss Trap (And How to Fix Your Ack Boundary)
On the receiving end of a webhook pipeline, the standard advice is to respond with a 2xx quickly so the sender doesn’t time out. The advice is sound and comes straight from provider documentation: GitHub wants a response within 10 seconds, Shopify within 5, and Stripe asks you to return a 2xx before running complex logic.
The danger is reading “quickly” as “immediately, before doing anything durable.” That produces one of the most destructive anti-patterns in webhook handling: acknowledging an HTTP request before durably persisting the payload.
The Premature 200 OK Failure Mode
Consider a receiver that fires off an in-process background task and immediately returns 200 OK:
# ANTI-PATTERN: Silent Data-Loss Trap
@app.post("/webhooks/payments")
async def handle_webhook(request: Request, background_tasks: BackgroundTasks):
raw_body = await request.body()
verify_signature(request.headers, raw_body) # signatures cover the raw bytes
payload = json.loads(raw_body)
# Queued in volatile process memory!
background_tasks.add_task(process_payment_event, payload)
# Sender marks the event as DELIVERED and may never retry it
return JSONResponse(status_code=200, content={"status": "accepted"})
(Signature verification is done over the raw request body, not a re-serialized JSON object, because re-serialization can change the bytes and break the HMAC.)
This pattern creates an unrecoverable data-loss window between returning the 200 OK and completing the side effect:
- Pod Eviction or Rolling Deploy: Kubernetes sends
SIGTERMduring a deployment. The response has already been sent, so the provider marks the delivery successful and stops retrying. When the pod’s grace period expires (30 seconds by default),SIGKILLterminates the process and the in-memory task queue disappears. - Downstream Transient Failure:
process_payment_eventwrites to your primary database during a brief failover and throws. Because you already returned200 OK, the sender has no idea the event failed. Unless you built a separate dead-letter mechanism inside the worker, the event is lost. - Unbounded Memory Growth (OOM): During a traffic spike, webhooks arrive faster than background tasks execute. In-memory tasks pile up until the container hits its memory limit and crashes, taking every unprocessed event with it.
Defining the Safe Acknowledgment Boundary
The acknowledgment boundary is the point in your ingestion pipeline where responsibility for the event’s durability transfers from the sender to the receiver.
A receiver should not return a 2xx status until the raw webhook payload has been committed to durable storage, such as a relational database transaction, or a replicated message broker (for example Kafka with acks=all and a sensible min.insync.replicas).
[Sender] ──POST /webhook──▶ [Receiver API]
│
1. Verify HMAC signature (raw body)
2. INSERT INTO webhook_inbox (event_id, payload)
3. COMMIT transaction ◀── [SAFE ACK BOUNDARY]
│
[Sender] ◀────2xx─────────────────┘
│
4. Async worker polls inbox & executes business logic
Decoupling durable ingestion from business-logic execution gives you speed and durability together:
- Fast response: Inserting one row into an inbox table or publishing to a durable queue is cheap compared with running business logic. It fits comfortably inside the tightest published deadlines, such as Shopify’s 5 seconds and Slack’s 3 seconds for its Events API. Measure your own p99 rather than assuming.
- Safe failure behavior: If your database or broker is unavailable, the commit fails and your API should return a
5xx(such as503 Service Unavailable). Whether that is safe depends on the sender. Stripe and Shopify will retry within their windows, but GitHub does not retry automatically, so a rejected delivery there must be redelivered by you, manually or via its API.
Implementation: The Transactional Inbox Pattern
Instead of executing business logic inside the HTTP handler, write the raw payload into an inbox table with a unique constraint on the sender’s event ID:
CREATE TABLE webhook_inbox (
event_id VARCHAR(255) PRIMARY KEY,
provider VARCHAR(64) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'pending',
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed_at TIMESTAMPTZ
);
In your HTTP handler, validate the signature and perform an idempotent insert:
INSERT INTO webhook_inbox (event_id, provider, payload)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING;
Only after the driver confirms the commit does the endpoint return 200 OK (or 202 Accepted; note that Shopify requires a 200-series code, and a 3xx counts as failure). Once persisted, an internal worker pool processes the row asynchronously, using your own retry and dead-letter infrastructure if downstream services fail.
Because providers can deliver the same event more than once, the primary key doubles as a first line of deduplication. For senders without automatic retries, or for events that arrived while you were down, pair the inbox with a periodic reconciliation job that lists recent events through the provider’s API (for example Stripe’s Events API) and fills any gaps.
3. Tracking Worker Heartbeats and Lease Expiration to Prevent Double-Execution Storms
Once you introduce durable queues and worker pools on either side of a webhook pipeline, a new failure mode appears: duplicate execution during worker restarts, network partitions, or garbage-collection pauses.
How Short Leases and Restarts Trigger Double-Execution Storms
To make sure crashed workers don’t hold jobs forever, queue systems rely on visibility timeouts (leases). Amazon SQS, for example, hides a received message for a visibility timeout (30 seconds by default) and makes it visible again if it isn’t deleted in time. A Postgres-backed queue can emulate this with a lease_expires_at column. In both cases, a job that outlives its lease is handed to someone else.
Consider this timeline:
- The Slow Endpoint: Worker A leases Event #409 with a 30-second lease and sends an HTTP
POSTto a subscriber whose endpoint takes 35 seconds to respond. - The Premature Expiration: At
t = 30s, Worker A is still waiting on the socket. The queue seeslease_expires_at < NOW()and hands Event #409 to Worker B. - The Duplicate Delivery: At
t = 31s, Worker B sends a second identicalPOST. Att = 35s, Worker A receives a200 OKfor the first request. The customer has now processed the webhook twice.
During a rolling deployment across dozens of worker pods, if in-flight requests outlast the lease or pods terminate mid-request without draining, many leases can expire at once. The resulting double-execution storm hits downstream APIs at the moment your cluster is least stable.
Mechanism 1: Fencing Tokens and Heartbeat Lease Renewal
Static lease timeouts force a trade-off:
- Too short (e.g., 10 seconds): Any slow HTTP call or brief GC pause expires the lease and causes duplicate execution.
- Too long (e.g., 10 minutes): If a worker is killed, the event sits stalled for 10 minutes before another worker picks it up.
A common solution is a short initial lease (e.g., 15–30 seconds) paired with a background heartbeat that extends it, and a monotonically increasing fencing token (here, lease_epoch). With SQS you would use ChangeMessageVisibility for the same purpose.
Step 1: Claiming with a Fencing Token
When a worker leases a job, it atomically increments lease_epoch and sets a short lease_expires_at:
UPDATE webhook_deliveries
SET
status = 'leased',
worker_id = 'worker-pod-84a',
lease_epoch = lease_epoch + 1,
lease_expires_at = NOW() + INTERVAL '20 seconds',
leased_at = NOW(),
updated_at = NOW()
WHERE id = (
SELECT id FROM webhook_deliveries
WHERE (status IN ('queued', 'retried') AND next_attempt_at <= NOW())
OR (status = 'leased' AND lease_expires_at < NOW())
ORDER BY next_attempt_at ASC
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING id, payload, lease_epoch;
FOR UPDATE SKIP LOCKED lets concurrent workers claim different rows without blocking one another, which is why it is a popular basis for Postgres-backed queues.
Step 2: Background Heartbeat Extension
While the worker performs the outbound HTTP call, a background timer fires roughly every lease_duration / 3 (about every 7 seconds for a 20-second lease) and extends the lease, strictly conditioned on matching both worker_id and lease_epoch:
UPDATE webhook_deliveries
SET
lease_expires_at = NOW() + INTERVAL '20 seconds',
updated_at = NOW()
WHERE id = $1
AND worker_id = $2
AND lease_epoch = $3
AND status = 'leased';
If this UPDATE affects 0 rows, the worker knows it lost its lease and should abort its in-flight work (for example via context cancellation) instead of continuing as a zombie.
Step 3: Fence the Completion Write
The same epoch check must guard the final state transition, otherwise a zombie worker can overwrite the result recorded by the new lease holder:
UPDATE webhook_deliveries
SET status = 'delivered', delivered_at = NOW(), updated_at = NOW()
WHERE id = $1
AND lease_epoch = $2
AND status = 'leased';
One caveat: fencing protects your own state table. It cannot recall an HTTP request that has already left the building. A zombie worker aborting early reduces wasted work, but the subscriber may still receive the event twice. That is why Mechanism 2 below is not optional.
Step 4: Graceful Shutdown Draining (SIGTERM)
When Kubernetes terminates a pod, it sends SIGTERM and, after terminationGracePeriodSeconds (30 seconds by default), SIGKILL. Workers should:
- Stop polling for new leases.
- Let in-flight HTTP requests (and their heartbeats) finish, within a grace period you have raised above your longest expected delivery timeout.
- If a request cannot finish in time, release the lease explicitly, with the same ownership checks, instead of leaving it to expire:
UPDATE webhook_deliveries
SET status = 'retried', next_attempt_at = NOW(), lease_expires_at = NULL, updated_at = NOW()
WHERE id = $1 AND worker_id = $2 AND lease_epoch = $3 AND status = 'leased';
Mechanism 2: Coordinating Lease Renewal with Receiver Idempotency
Even with heartbeats, fencing tokens, and graceful shutdowns, network failures mean at-least-once delivery will occasionally deliver the same event twice. A receiver might commit a transaction and send a 200 OK that is lost before reaching the sender, and the sender, seeing no response, retries.
To make duplicates harmless, both sides must cooperate on idempotency keys:
- Sender-Side Stability: The sender assigns a globally unique, immutable event ID (for example a UUIDv7 or ULID) when the event first enters the
receivedstate. Across every retry, lease expiry, and worker failover, the ID in the headers and payload stays identical. Real providers follow this pattern: GitHub keepsX-GitHub-Deliveryunchanged on redelivery, and Shopify keepsX-Shopify-Webhook-Idthe same across retries. The Standard Webhooks specification similarly defines awebhook-idheader for this purpose. - Receiver-Side Atomic Deduplication: Deduplication should happen inside the same database transaction as the business state change, not in a separate non-transactional lookup (such as checking Redis before writing to Postgres), which leaves a race window.
BEGIN;
-- Claim the idempotency key inside the main transaction
INSERT INTO processed_webhooks (webhook_id, processed_at)
VALUES ('evt_01J9X8K2M4N5P6Q7R8S9T0', NOW())
ON CONFLICT (webhook_id) DO NOTHING;
-- In application code: if rows_affected == 0, another transaction already
-- handled this event, so ROLLBACK and return 200 OK.
UPDATE account_balances
SET balance_cents = balance_cents + 5000
WHERE account_id = 'acc_123';
COMMIT;
If Worker A and Worker B process the same re-leased webhook at the same time:
- Both attempt the
INSERT INTO processed_webhooks. - PostgreSQL makes the second inserter wait on the conflicting unique-index entry until the first transaction commits or rolls back.
- If Worker A commits, Worker B’s insert affects zero rows. It rolls back its balance update and returns
200 OK. - If Worker A crashes mid-transaction, PostgreSQL rolls back its uncommitted insert, and Worker B proceeds to complete the work.
Summary Checklist for Webhook Reliability Forensics
When auditing or hardening a webhook pipeline, verify these invariants:
- Separate
queuedfromleased: Don’t lump waiting and in-flight jobs into onependingstate. Trackenqueued_at,leased_at, andattempted_atindependently to distinguish queue starvation from hung workers. - Monitor
expired_leasesas a leading indicator: A rising count signals worker OOMs, GC pauses, or missing HTTP client timeouts long before overall delivery rates drop. - Don’t return
2xxbefore a durable commit: Emit success responses only after the payload is committed to a transactional inbox or replicated broker, and know whether your sender will retry if you return a5xx. - Use heartbeats and fencing tokens (
lease_epoch): Pair short 15–30 second leases with heartbeat renewals and epoch-guarded writes so a worker that lost its lease can’t corrupt state. Remember that fencing doesn’t unsend requests. - Deduplicate on the sender’s event ID inside the business transaction: Treat retries and lease expirations as inevitable, so duplicate deliveries resolve to clean no-ops.
- Reconcile: Periodically compare your records with the provider’s event list. Retry windows are finite, and some senders don’t retry at all.