Silent 200s and Phantom Side Effects: Auditing “Accepted” Webhooks That Never Finished Downstream
Why returning 200 OK at webhook intake hides partial failures, orphaned states, and silent drops—and how to build reconciliation checks and timeline forensics.

Silent 200s and Phantom Side Effects: Auditing “Accepted” Webhooks That Never Finished Downstream
Every distributed system has a shortcut it takes to keep latency low. In event-driven architectures, that shortcut is the immediate 200 OK on a webhook intake endpoint.
Webhook providers enforce short delivery deadlines. Stripe’s documentation tells you to return a 2xx before any complex logic that might cause a timeout, but it doesn’t publish an exact number. GitHub records a delivery as failed if your server takes longer than 10 seconds to respond. Shopify requires a response within five seconds. Twilio waits 5 seconds to connect and up to 15 seconds for a response by default, and imposes a hard 15-second cap on call-related requests.
If your ingestion server runs business logic synchronously and a database lock or downstream API stalls, you miss the deadline. Depending on the provider, that means retries and duplicate deliveries, or a dropped event.
The standard advice is to decouple ingestion from execution. Validate the signature, persist the raw payload somewhere durable, and acknowledge within milliseconds.
But a 2xx response only tells the provider that you received the event. It does not mean the event’s business effect happened. Providers treat a 2xx as success and stop caring, so a webhook that was accepted and then lost downstream looks, from the provider’s side, like a perfectly healthy delivery. Meanwhile customers see paid invoices stuck as pending, accounts missing permissions, or inventory counts drifting out of sync.
This article calls that failure mode silent 200s and phantom side effects. Fixing it means moving from perimeter observability (watching HTTP status codes at intake) to end-to-end state reconciliation and timeline forensics.
Provider behaviors cited below were checked against official documentation in November 2026. Limits change, so re-verify the numbers that matter to your design.
What Providers Actually Promise (and Where Their Promises End)
Before auditing your own pipeline, know what the sender will and won’t do for you. The differences are large, and they determine how long you have to detect and repair a gap.
| Provider | Response deadline | Automatic retries | Recovery window |
|---|---|---|---|
| Stripe | Not published; docs require a quick 2xx before complex logic | Live mode: up to three days with exponential backoff | Events API lists events from the last 30 days; CLI resend works for up to 30 days |
| GitHub | 10 seconds | None; you must redeliver manually or via API | Deliveries from the past 3 days can be redelivered |
| Shopify | 5 seconds | Up to eight retries over four hours; persistent failures remove the subscription | After retries are exhausted, you may need to recover missing data yourself |
| Twilio | 15 seconds default read timeout; hard 15-second cap on call-related requests | Default is one retry, and only on TCP connect or TLS handshake failure; a 5xx or read timeout doesn’t trigger it unless you configure a retry policy | Reconcile via the REST API |
Three consequences follow:
- Provider retries are not a safety net for downstream failures. They only cover the delivery hop. Once you return
2xx, no retry will ever fire, however badly your consumer fails later. - Providers like GitHub never retry at all. If your endpoint returns a
5xxbecause your database was briefly down, that event is gone unless something redelivers it. - Every recovery window is finite. Anything that slips past the window (three days for GitHub redelivery, 30 days for Stripe’s Events API) can’t be recovered from the provider. Your reconciliation cadence must be much tighter than these limits.
The Anatomy of a Silent 200 Failure
To see why acknowledged webhooks fail silently, trace an event’s lifecycle after the HTTP connection closes. A typical pipeline has an API gateway, an ingress worker, a message broker (Apache Kafka, Amazon SQS, RabbitMQ, or Redis Streams), a pool of background consumers, and several downstream databases or third-party services.
Between intake acknowledgment and terminal completion, there are four main fracture points.
1. The Ingress-to-Broker Gap (Unlogged Drops)
The most immediate failure happens when the ingress service validates a payload, hands it to a producer, and returns 200 OK without confirming durable persistence.
- Fire-and-forget publishing: Kafka’s Java producer
send()is asynchronous. If the handler responds without waiting for the result, a failed publish never reaches the HTTP response. Producer acknowledgment settings also matter.acks=0offers no delivery guarantee, andacks=1can lose data if the partition leader fails before followers replicate the record. Since Kafka 3.0 the Java client defaults toacks=all(previouslyacks=1), but older clients, other-language clients, and explicitly tuned configurations may still run weaker settings. Check what you actually deploy. Separately, anything buffered in process memory is lost if the pod is killed. - Serialization failures after the ack: Ingress accepts raw JSON, returns
200 OK, and then fails while converting the payload into an internal Avro or Protobuf envelope. A generic error handler logs a warning but never alerts or re-drives.
2. Poison Pills and Silent Dead-Letter Queues (DLQs)
Once a message reaches a durable queue, a consumer picks it up. If the payload triggers an unhandled edge case (a null on an optional field, an unexpected currency code, a schema change from the provider), the consumer throws.
- Infinite retry loops vs. silent DLQ routing: In a poorly configured consumer group, a failing message either blocks progress or is routed to a dead-letter queue after a fixed number of attempts (in SQS, the redrive policy’s
maxReceiveCount). Setting that threshold too low means a single transient failure can send a message to the DLQ. - The DLQ graveyard: Moving a message to a DLQ is a storage strategy, not a recovery strategy. Queues have retention limits. SQS dead-letter queues, for example, recommend a maximum retention period of 14 days. For SQS standard queues there’s a further trap: expiration is based on the message’s original enqueue timestamp, which doesn’t reset when it moves to the DLQ. A message that spent days in the source queue may have far less than the configured retention left. Without alerting and replay tooling, a message that ages out leaves behind a webhook that was “accepted” but whose business effect was permanently erased.
3. Partial Execution and Phantom Side Effects
The hardest failures to debug occur when a consumer runs a multi-step workflow without transactional atomicity. Consider a Stripe checkout.session.completed event that must do three things:
- Update the
orderstable in PostgreSQL fromPENDINGtoPAID. - Call an internal fulfillment service over gRPC to reserve warehouse inventory.
- Dispatch a transactional receipt email through an external API.
Suppose step 1 commits, step 2 succeeds, and the worker dies before step 3 or before acknowledging the message. Workers die for ordinary reasons, such as an out-of-memory kill, a node failure, or a rolling deploy. Kubernetes sends SIGTERM and then SIGKILL once the grace period (30 seconds by default) expires, so any in-flight work that outlasts the grace period is cut off mid-step. Two failure modes follow:
- If the offset or message wasn’t acknowledged: The broker redelivers. If the consumer lacks idempotency keys per side effect, the customer may be charged or allocated inventory twice.
- If the offset was committed before processing finished: The work is skipped. Kafka consumers have
enable.auto.commitset totrueby default, which can commit offsets on a timer independent of whether your handler finished. The order is paid and reserved, but the customer never gets a confirmation or a downstream analytics pipeline never sees the event.
The mirror-image failure is just as damaging. If step 2 throws after step 1 has committed, and nothing coordinates the two, the database says PAID while the warehouse never ships the item. That’s a classic phantom side effect.
4. Out-of-Order Execution and State Clobbering
Webhooks do not arrive in strict causal order. Stripe states that it doesn’t guarantee delivery of events in the order they’re generated, and delivery is at-least-once, so late retries and concurrent delivery can both reorder events.
Consider an illustrative case. A user updates a subscription (customer.subscription.updated) and cancels it (customer.subscription.deleted) within a few hundred milliseconds. Variable consumer speeds let the deleted worker finish first, and then the slower updated worker overwrites the terminal CANCELED status with ACTIVE. Both webhooks returned 200 OK, both consumers succeeded, and the final state is wrong.
Shopify adds a related warning: webhooks can arrive late, sometimes up to a day, so comparing the webhook’s timestamp to the current time is part of correct handling.
Why Traditional Monitoring Misses Downstream Drops
Standard observability stacks, built around RED metrics (Rate, Errors, Duration) at the gateway and consumer level, are structurally blind to silent 200s.
| Observability Layer | What It Measures | Why It Misses Silent 200s |
|---|---|---|
| API Gateway / Load Balancer | Ingress HTTP status codes (200, 4xx, 5xx) and latency. | Confirms the payload reached your perimeter, not that it entered a durable queue or changed state. |
| Message Broker Metrics | Consumer lag, throughput, queue depth. | Low lag only means messages are being consumed quickly, including ones that are immediately dropped or routed to a DLQ. |
| Application Error Tracking | Uncaught exceptions and 500 stack traces. | Swallowed exceptions, silent conditional skips (if (!record) return;), and process kills produce no useful stack trace. |
| Provider Webhook Logs | Delivery attempts and your server’s response. | Once the provider sees a 2xx, the delivery counts as successful. |
When a team relies on ingress 200 rates and consumer exception alerts alone, a silent drop looks identical to normal operation. You cannot detect the absence of an expected side effect with metrics that only count errors.
Architectural Blueprint: From Perimeter Ack to Verified Completion
Eliminating silent 200s means treating webhook ingestion as a state machine instead of a stateless pass-through. Every external event gets a durable tracking record before the 200 OK is sent, and that record stays open until every downstream side effect has verifiably completed.
Step 1: The Transactional Inbox Pattern
Before returning 200 OK, persist the raw payload, headers, provider event ID, and ingestion timestamp into a webhook_inbox table in your primary transactional database (or another datastore where a successful write is durable).
CREATE TABLE webhook_inbox (
inbox_id UUID PRIMARY KEY,
provider VARCHAR(64) NOT NULL,
provider_event_id VARCHAR(255) NOT NULL,
event_type VARCHAR(128) NOT NULL,
correlation_key VARCHAR(255), -- order_id, customer_id, subscription_id...
payload JSONB NOT NULL,
headers JSONB NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'RECEIVED',
-- RECEIVED, QUEUED, PROCESSING, COMPLETED, FAILED
attempts INT NOT NULL DEFAULT 0,
lease_expires_at TIMESTAMPTZ,
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ,
error_context JSONB,
UNIQUE (provider, provider_event_id)
);
CREATE INDEX webhook_inbox_status_received_idx ON webhook_inbox (status, received_at);
CREATE INDEX webhook_inbox_correlation_idx ON webhook_inbox (correlation_key, received_at);
Why this works:
- Atomic deduplication. The
UNIQUE (provider, provider_event_id)constraint means provider retries can’t create duplicate work records. Choose the key per provider. GitHub’sX-GitHub-DeliveryGUID, for instance, stays the same on redelivery, which makes it a natural dedupe key. Be careful about what you return on a duplicate, though. If the existing row is alreadyCOMPLETED, acknowledging is correct. If it’sFAILEDor stuck, the redelivery is a chance to re-drive it, not just discard it. - Proof of receipt. Returning
200 OKonly after thisINSERTcommits means no event is acknowledged unless it sits in a queryable ledger. - Fail loudly at the edge. If the insert fails (the database is down, say), return a
5xxrather than a2xx. Stripe and Shopify will retry. GitHub won’t, so you’ll need its redelivery API to recover. - Decoupled dispatch via outbox or CDC. Writing to PostgreSQL and a broker separately is the classic dual-write problem: if one write fails, the data is inconsistent. Use change data capture with Debezium, or a lightweight polling publisher, to push
RECEIVEDrows onto your broker and move them toQUEUED.
Step 2: Explicit Terminal State Transitions
A consumer processing an event from the inbox must never treat a broker acknowledgment alone as proof of completion. The consumer should explicitly flip the webhook_inbox row to COMPLETED as the last step of its work. Disable auto-commit (or, for SQS, delete the message only after success) so the broker’s view of progress follows the inbox’s.
When a webhook triggers several side effects, model each as an idempotent ledger entry linked to the parent inbox_id:
CREATE TABLE webhook_side_effects (
effect_id UUID PRIMARY KEY,
inbox_id UUID NOT NULL REFERENCES webhook_inbox(inbox_id),
effect_name VARCHAR(128) NOT NULL, -- 'update_ledger', 'provision_tenant', 'send_receipt'
status VARCHAR(32) NOT NULL DEFAULT 'PENDING', -- PENDING, SUCCEEDED, FAILED
external_reference_id VARCHAR(255),
executed_at TIMESTAMPTZ,
UNIQUE (inbox_id, effect_name)
);
If a worker dies mid-execution and another picks up the retry, it reads webhook_side_effects for that inbox_id, skips anything SUCCEEDED, and runs the rest. Only when every expected side effect is SUCCEEDED does the parent row become COMPLETED.
For side effects that call external APIs, pass a deterministic idempotency key derived from (provider, provider_event_id, effect_name) wherever the API supports one. Many payment and messaging APIs do. This closes the gap where a call succeeds but the worker dies before recording SUCCEEDED.
To keep two workers from claiming the same row, use a lease. In PostgreSQL, SELECT ... FOR UPDATE SKIP LOCKED is a common way to claim rows without blocking other workers. Set lease_expires_at when a worker claims a row, so a crashed worker’s lease eventually expires and the watchdog described below can reclaim the event.
Step 3: Guarding Against Out-of-Order Clobbering
The tempting fix is “newest timestamp wins.” It’s riskier than it looks. Stripe advises against using an event’s created time to decide ordering or detect duplicates, because distinct events can share a timestamp, and late retries of old events can still arrive after newer ones. Stripe’s guidance is to rely on the object’s current state instead of arrival order, and to fetch the object from the API when an event is ambiguous.
Apply the strongest mechanism the provider supports:
- Treat the webhook as a trigger, not the source of truth. For providers like Stripe, have the consumer fetch the current object state and write that. Then late or reordered events converge on the same correct answer.
- Use a monotonic version where one exists. If the payload carries a version or sequence number for the object, enforce it at the database layer:
UPDATE subscriptions
SET status = :new_status,
tier = :new_tier,
last_event_version = :event_version,
updated_at = NOW()
WHERE subscription_id = :sub_id
AND (last_event_version IS NULL OR last_event_version < :event_version);
- Use timestamps only as a last resort, and know their limits. If you must, compare against the provider’s own timestamp, treat ties explicitly, and prefer re-fetching state when a tie or ambiguity arises.
If the WHERE clause matches zero rows, the event is stale. The consumer records STALE_EVENT_IGNORED in error_context, marks the inbox row COMPLETED, and leaves current state untouched.
Building Automated Reconciliation Checks
Even with an inbox ledger and idempotent workers, distributed systems still lose events. Network partitions happen, conditional branches hide bugs, and sometimes the provider never sends the event at all. Reconciliation is the background safety net that detects divergence between what the provider believes happened and what your database committed.
1. The Internal Orphan Sweep (SLA Watchdog)
The highest-leverage check runs purely against your own webhook_inbox. Any event that entered with a 200 OK should reach COMPLETED within a defined service level objective, such as five minutes (choose a number that fits your workload).
Run a scheduled query every minute:
SELECT inbox_id, provider, provider_event_id, event_type, status, received_at, attempts
FROM webhook_inbox
WHERE status IN ('RECEIVED', 'QUEUED', 'PROCESSING')
AND received_at < NOW() - INTERVAL '5 minutes'
ORDER BY received_at ASC
LIMIT 500;
Automated remediation:
- Stuck in
RECEIVED: the outbox publisher missed it, so re-enqueue it. - Stuck in
PROCESSINGwith an expired lease: the worker likely crashed or was killed mid-flight, so reclaim the lease and re-drive. - Past a maximum age (say, 30 minutes) or retry count: page the on-call engineer with the
inbox_idanderror_context.
The same logic applies to your broker-level DLQ. Alert on any message arriving there, and build replay tooling before you need it, so that dead-lettered messages can’t quietly age out.
2. Cross-System State Reconciliation (Provider vs. Internal Ledger)
Internal sweeps catch events that entered your system and stalled. Cross-system reconciliation catches events that were dropped before persistence or silently ignored because of business-logic bugs.
Run a batch job regularly (hourly is a reasonable starting point, and it must be well inside the provider’s recovery windows) that compares a rolling lookback window:
- Fetch authoritative events from the provider’s API. For Stripe, list events through the Events API. It returns only events from the last 30 days, and supports filtering with
delivery_success=falseto find events that failed to reach your endpoint. For GitHub, list webhook deliveries through the REST API and check for statuses other thanOK, as GitHub’s own guidance suggests. Leave a buffer (for example, ignore the most recent 15 minutes) so in-flight processing isn’t flagged as drift. - Diff against
webhook_inboxand your domain tables:- Missing inbox records: The provider has event
evt_123, but it isn’t inwebhook_inbox. This indicates an ingress drop or delivery failure. Fetch the event and inject it into the inbox. - Completed inbox, mismatched domain state:
webhook_inboxsaysinvoice.paidisCOMPLETED, yetinvoices.statusis stillOPEN. This indicates a phantom side effect or a silent conditional skip in consumer code. - Orphaned internal state: A provisioned resource has no corresponding provider event or active subscription.
- Missing inbox records: The provider has event
3. Canary Webhooks (Synthetic Heartbeats)
Don’t wait for real customer traffic to discover that a deployment broke the consumer pipeline. Inject synthetic canary webhooks through your real public ingress every few minutes, signed with a test key. Track each canary’s inbox_id through to a dummy sink table. If a canary gets a 200 OK at the edge but doesn’t reach the sink within your chosen threshold (say, 15 seconds), raise a pipeline health alert.
Timeline Forensics: Reconstructing What Actually Happened
When a customer reports a discrepancy or reconciliation flags an orphaned record, engineers often burn hours grepping disconnected logs across gateways, pods, and database audit trails. Effective forensics means stitching every step of a webhook’s life into one queryable trace.
Propagating Correlation Context Across Async Boundaries
Distributed tracing breaks at queue boundaries unless context is carried explicitly. OpenTelemetry’s messaging conventions say a producer should attach a message creation context to each message, and that without it consumer traces can’t be directly correlated with producer traces. In practice that means serializing the W3C traceparent (and tracestate, if present) into message metadata, and also storing it in the webhook_inbox row so a trace can be found long after the broker has discarded the message. Consumers typically link their spans to the producer’s creation context.
Then record four immutable timestamps per event:
- T0,
provider_created_at: Taken from the payload or provider header. Shows upstream delays before the request reached you. - T1,
ingress_received_at: Recorded when your edge verifies the signature and commits the inbox row. T1 − T0 separates provider lag from internal lag. - T2,
consumer_started_at: Recorded when a worker leases the event. T2 − T1 isolates queue backlog and broker saturation. - T3,
terminal_committed_at: Recorded when the final side effect commits and the status flips toCOMPLETED. T3 − T2 isolates lock contention and slow downstream calls.
Forensic Investigation Runbook
When auditing a missing or corrupted entity (for example, order_id = ord_88412):
- Query the inbox ledger by entity. Using the indexed
correlation_key, pull every webhook received for that entity in chronological order, including ignored, stale, and failed events. - Inspect side-effect granularity. Join
webhook_inboxwithwebhook_side_effects. Seeingupdate_ledger: SUCCEEDEDnext toprovision_tenant: PENDINGnarrows a vague mystery to one service call. - Diff before-and-after state. If your domain tables have audit triggers, logical replication, or event sourcing, inspect row-level changes around T2 and T3. Often the webhook did succeed, and a separate job or race condition reverted the state minutes later.
- Replay deterministically. Never repair a failed webhook by hand-editing production tables. That bypasses downstream effects such as analytics events, outbound webhooks to your own customers, and cache invalidation. Instead, reset the inbox row from
FAILEDtoRECEIVEDand let the side-effect ledger skip completed steps and run only the missing ones. Note that manual resends from the provider side don’t cancel its automatic retries. Stripe’s documentation notes that resending an event doesn’t stop the automatic retry schedule, so your dedupe layer must handle both arriving.
Implementation Checklist for Engineering Teams
To harden an existing pipeline without a rewrite, prioritize in this order:
- Persist before you acknowledge. Never return
2xxuntil the raw payload is in a durablewebhook_inboxtable or a replicated broker withacks=all. If persistence fails, return a5xx. - Bind broker acknowledgments to terminal state. Disable auto-commit, and acknowledge or delete the message only after all database mutations and external side effects have succeeded.
- Implement per-side-effect idempotency. Guard every external call and mutation with a key derived from
(provider, provider_event_id, effect_name). - Deploy an orphan watchdog. Alert whenever any inbox row stays non-terminal (
RECEIVED,QUEUED,PROCESSING) beyond your SLO. - Alert on and replay your DLQ. Verify its retention exceeds the source queue’s, and remember that on SQS standard queues the original enqueue time governs expiry.
- Make ordering safe. Re-fetch current state where the provider recommends it, and enforce monotonic versions where available.
- Run a scheduled provider-vs-internal reconciler. Compare provider records against your state on a cadence well inside the provider’s recovery window (three days for GitHub redelivery, 30 days for Stripe’s Events API), and escalate drift automatically.
- Run canaries. Send synthetic signed webhooks through the real ingress and verify they reach a sink.
Conclusion
Returning 200 OK at the edge is a necessary optimization, not a guarantee of business correctness. The provider’s retries only cover the hop to your front door, and in GitHub’s case they don’t exist at all. Everything after the acknowledgment is your responsibility. By pairing fast perimeter acknowledgment with a transactional inbox, per-step idempotency, explicit terminal states, and automated cross-system reconciliation, you can make sure every accepted webhook finishes what it started, or at least that you find out quickly when it doesn’t.
Sources
- Stripe: Receive Stripe events in your webhook endpoint
- GitHub: Handling failed webhook deliveries
- GitHub: Redelivering webhooks
- Shopify: Troubleshoot webhooks
- Twilio: Webhooks connection overrides
- Apache Kafka 3.0 upgrade notes (producer defaults)
- Confluent: Optimizing for durability
- Amazon SQS: Dead-letter queues
- Amazon SNS: Dead-letter queues
- Kubernetes: Termination of Pods
- Red Hat Developer: Avoiding dual writes in event-driven applications
- OpenTelemetry: Messaging span semantic conventions