InstaWebhook
November 22, 2026By InstaWebhook TeamWebhook Reliability

Silent 200s and Phantom Side Effects: Auditing “Accepted” Webhooks That Never Finished Downstream

Why returning 200 OK at webhook intake hides partial failures, orphaned states, and silent drops—and how to build reconciliation checks and timeline forensics.

Silent 200s and Phantom Side Effects: Auditing “Accepted” Webhooks That Never Finished Downstream

Silent 200s and Phantom Side Effects: Auditing “Accepted” Webhooks That Never Finished Downstream

Every distributed system has a shortcut it takes to keep latency low. In event-driven architectures, that shortcut is the immediate 200 OK on a webhook intake endpoint.

Webhook providers enforce short delivery deadlines. Stripe’s documentation tells you to return a 2xx before any complex logic that might cause a timeout, but it doesn’t publish an exact number. GitHub records a delivery as failed if your server takes longer than 10 seconds to respond. Shopify requires a response within five seconds. Twilio waits 5 seconds to connect and up to 15 seconds for a response by default, and imposes a hard 15-second cap on call-related requests.

If your ingestion server runs business logic synchronously and a database lock or downstream API stalls, you miss the deadline. Depending on the provider, that means retries and duplicate deliveries, or a dropped event.

The standard advice is to decouple ingestion from execution. Validate the signature, persist the raw payload somewhere durable, and acknowledge within milliseconds.

But a 2xx response only tells the provider that you received the event. It does not mean the event’s business effect happened. Providers treat a 2xx as success and stop caring, so a webhook that was accepted and then lost downstream looks, from the provider’s side, like a perfectly healthy delivery. Meanwhile customers see paid invoices stuck as pending, accounts missing permissions, or inventory counts drifting out of sync.

This article calls that failure mode silent 200s and phantom side effects. Fixing it means moving from perimeter observability (watching HTTP status codes at intake) to end-to-end state reconciliation and timeline forensics.

Provider behaviors cited below were checked against official documentation in November 2026. Limits change, so re-verify the numbers that matter to your design.

What Providers Actually Promise (and Where Their Promises End)

Before auditing your own pipeline, know what the sender will and won’t do for you. The differences are large, and they determine how long you have to detect and repair a gap.

ProviderResponse deadlineAutomatic retriesRecovery window
StripeNot published; docs require a quick 2xx before complex logicLive mode: up to three days with exponential backoffEvents API lists events from the last 30 days; CLI resend works for up to 30 days
GitHub10 secondsNone; you must redeliver manually or via APIDeliveries from the past 3 days can be redelivered
Shopify5 secondsUp to eight retries over four hours; persistent failures remove the subscriptionAfter retries are exhausted, you may need to recover missing data yourself
Twilio15 seconds default read timeout; hard 15-second cap on call-related requestsDefault is one retry, and only on TCP connect or TLS handshake failure; a 5xx or read timeout doesn’t trigger it unless you configure a retry policyReconcile via the REST API

Three consequences follow:

  • Provider retries are not a safety net for downstream failures. They only cover the delivery hop. Once you return 2xx, no retry will ever fire, however badly your consumer fails later.
  • Providers like GitHub never retry at all. If your endpoint returns a 5xx because your database was briefly down, that event is gone unless something redelivers it.
  • Every recovery window is finite. Anything that slips past the window (three days for GitHub redelivery, 30 days for Stripe’s Events API) can’t be recovered from the provider. Your reconciliation cadence must be much tighter than these limits.

The Anatomy of a Silent 200 Failure

To see why acknowledged webhooks fail silently, trace an event’s lifecycle after the HTTP connection closes. A typical pipeline has an API gateway, an ingress worker, a message broker (Apache Kafka, Amazon SQS, RabbitMQ, or Redis Streams), a pool of background consumers, and several downstream databases or third-party services.

Between intake acknowledgment and terminal completion, there are four main fracture points.

1. The Ingress-to-Broker Gap (Unlogged Drops)

The most immediate failure happens when the ingress service validates a payload, hands it to a producer, and returns 200 OK without confirming durable persistence.

  • Fire-and-forget publishing: Kafka’s Java producer send() is asynchronous. If the handler responds without waiting for the result, a failed publish never reaches the HTTP response. Producer acknowledgment settings also matter. acks=0 offers no delivery guarantee, and acks=1 can lose data if the partition leader fails before followers replicate the record. Since Kafka 3.0 the Java client defaults to acks=all (previously acks=1), but older clients, other-language clients, and explicitly tuned configurations may still run weaker settings. Check what you actually deploy. Separately, anything buffered in process memory is lost if the pod is killed.
  • Serialization failures after the ack: Ingress accepts raw JSON, returns 200 OK, and then fails while converting the payload into an internal Avro or Protobuf envelope. A generic error handler logs a warning but never alerts or re-drives.

2. Poison Pills and Silent Dead-Letter Queues (DLQs)

Once a message reaches a durable queue, a consumer picks it up. If the payload triggers an unhandled edge case (a null on an optional field, an unexpected currency code, a schema change from the provider), the consumer throws.

3. Partial Execution and Phantom Side Effects

The hardest failures to debug occur when a consumer runs a multi-step workflow without transactional atomicity. Consider a Stripe checkout.session.completed event that must do three things:

  1. Update the orders table in PostgreSQL from PENDING to PAID.
  2. Call an internal fulfillment service over gRPC to reserve warehouse inventory.
  3. Dispatch a transactional receipt email through an external API.

Suppose step 1 commits, step 2 succeeds, and the worker dies before step 3 or before acknowledging the message. Workers die for ordinary reasons, such as an out-of-memory kill, a node failure, or a rolling deploy. Kubernetes sends SIGTERM and then SIGKILL once the grace period (30 seconds by default) expires, so any in-flight work that outlasts the grace period is cut off mid-step. Two failure modes follow:

  • If the offset or message wasn’t acknowledged: The broker redelivers. If the consumer lacks idempotency keys per side effect, the customer may be charged or allocated inventory twice.
  • If the offset was committed before processing finished: The work is skipped. Kafka consumers have enable.auto.commit set to true by default, which can commit offsets on a timer independent of whether your handler finished. The order is paid and reserved, but the customer never gets a confirmation or a downstream analytics pipeline never sees the event.

The mirror-image failure is just as damaging. If step 2 throws after step 1 has committed, and nothing coordinates the two, the database says PAID while the warehouse never ships the item. That’s a classic phantom side effect.

4. Out-of-Order Execution and State Clobbering

Webhooks do not arrive in strict causal order. Stripe states that it doesn’t guarantee delivery of events in the order they’re generated, and delivery is at-least-once, so late retries and concurrent delivery can both reorder events.

Consider an illustrative case. A user updates a subscription (customer.subscription.updated) and cancels it (customer.subscription.deleted) within a few hundred milliseconds. Variable consumer speeds let the deleted worker finish first, and then the slower updated worker overwrites the terminal CANCELED status with ACTIVE. Both webhooks returned 200 OK, both consumers succeeded, and the final state is wrong.

Shopify adds a related warning: webhooks can arrive late, sometimes up to a day, so comparing the webhook’s timestamp to the current time is part of correct handling.

Why Traditional Monitoring Misses Downstream Drops

Standard observability stacks, built around RED metrics (Rate, Errors, Duration) at the gateway and consumer level, are structurally blind to silent 200s.

Observability LayerWhat It MeasuresWhy It Misses Silent 200s
API Gateway / Load BalancerIngress HTTP status codes (200, 4xx, 5xx) and latency.Confirms the payload reached your perimeter, not that it entered a durable queue or changed state.
Message Broker MetricsConsumer lag, throughput, queue depth.Low lag only means messages are being consumed quickly, including ones that are immediately dropped or routed to a DLQ.
Application Error TrackingUncaught exceptions and 500 stack traces.Swallowed exceptions, silent conditional skips (if (!record) return;), and process kills produce no useful stack trace.
Provider Webhook LogsDelivery attempts and your server’s response.Once the provider sees a 2xx, the delivery counts as successful.

When a team relies on ingress 200 rates and consumer exception alerts alone, a silent drop looks identical to normal operation. You cannot detect the absence of an expected side effect with metrics that only count errors.

Architectural Blueprint: From Perimeter Ack to Verified Completion

Eliminating silent 200s means treating webhook ingestion as a state machine instead of a stateless pass-through. Every external event gets a durable tracking record before the 200 OK is sent, and that record stays open until every downstream side effect has verifiably completed.

Step 1: The Transactional Inbox Pattern

Before returning 200 OK, persist the raw payload, headers, provider event ID, and ingestion timestamp into a webhook_inbox table in your primary transactional database (or another datastore where a successful write is durable).

Code example
CREATE TABLE webhook_inbox (
    inbox_id          UUID PRIMARY KEY,
    provider          VARCHAR(64)  NOT NULL,
    provider_event_id VARCHAR(255) NOT NULL,
    event_type        VARCHAR(128) NOT NULL,
    correlation_key   VARCHAR(255),           -- order_id, customer_id, subscription_id...
    payload           JSONB NOT NULL,
    headers           JSONB NOT NULL,
    status            VARCHAR(32) NOT NULL DEFAULT 'RECEIVED',
                      -- RECEIVED, QUEUED, PROCESSING, COMPLETED, FAILED
    attempts          INT NOT NULL DEFAULT 0,
    lease_expires_at  TIMESTAMPTZ,
    received_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at      TIMESTAMPTZ,
    error_context     JSONB,
    UNIQUE (provider, provider_event_id)
);

CREATE INDEX webhook_inbox_status_received_idx ON webhook_inbox (status, received_at);
CREATE INDEX webhook_inbox_correlation_idx ON webhook_inbox (correlation_key, received_at);

Why this works:

  1. Atomic deduplication. The UNIQUE (provider, provider_event_id) constraint means provider retries can’t create duplicate work records. Choose the key per provider. GitHub’s X-GitHub-Delivery GUID, for instance, stays the same on redelivery, which makes it a natural dedupe key. Be careful about what you return on a duplicate, though. If the existing row is already COMPLETED, acknowledging is correct. If it’s FAILED or stuck, the redelivery is a chance to re-drive it, not just discard it.
  2. Proof of receipt. Returning 200 OK only after this INSERT commits means no event is acknowledged unless it sits in a queryable ledger.
  3. Fail loudly at the edge. If the insert fails (the database is down, say), return a 5xx rather than a 2xx. Stripe and Shopify will retry. GitHub won’t, so you’ll need its redelivery API to recover.
  4. Decoupled dispatch via outbox or CDC. Writing to PostgreSQL and a broker separately is the classic dual-write problem: if one write fails, the data is inconsistent. Use change data capture with Debezium, or a lightweight polling publisher, to push RECEIVED rows onto your broker and move them to QUEUED.

Step 2: Explicit Terminal State Transitions

A consumer processing an event from the inbox must never treat a broker acknowledgment alone as proof of completion. The consumer should explicitly flip the webhook_inbox row to COMPLETED as the last step of its work. Disable auto-commit (or, for SQS, delete the message only after success) so the broker’s view of progress follows the inbox’s.

When a webhook triggers several side effects, model each as an idempotent ledger entry linked to the parent inbox_id:

Code example
CREATE TABLE webhook_side_effects (
    effect_id             UUID PRIMARY KEY,
    inbox_id              UUID NOT NULL REFERENCES webhook_inbox(inbox_id),
    effect_name           VARCHAR(128) NOT NULL,  -- 'update_ledger', 'provision_tenant', 'send_receipt'
    status                VARCHAR(32) NOT NULL DEFAULT 'PENDING', -- PENDING, SUCCEEDED, FAILED
    external_reference_id VARCHAR(255),
    executed_at           TIMESTAMPTZ,
    UNIQUE (inbox_id, effect_name)
);

If a worker dies mid-execution and another picks up the retry, it reads webhook_side_effects for that inbox_id, skips anything SUCCEEDED, and runs the rest. Only when every expected side effect is SUCCEEDED does the parent row become COMPLETED.

For side effects that call external APIs, pass a deterministic idempotency key derived from (provider, provider_event_id, effect_name) wherever the API supports one. Many payment and messaging APIs do. This closes the gap where a call succeeds but the worker dies before recording SUCCEEDED.

To keep two workers from claiming the same row, use a lease. In PostgreSQL, SELECT ... FOR UPDATE SKIP LOCKED is a common way to claim rows without blocking other workers. Set lease_expires_at when a worker claims a row, so a crashed worker’s lease eventually expires and the watchdog described below can reclaim the event.

Step 3: Guarding Against Out-of-Order Clobbering

The tempting fix is “newest timestamp wins.” It’s riskier than it looks. Stripe advises against using an event’s created time to decide ordering or detect duplicates, because distinct events can share a timestamp, and late retries of old events can still arrive after newer ones. Stripe’s guidance is to rely on the object’s current state instead of arrival order, and to fetch the object from the API when an event is ambiguous.

Apply the strongest mechanism the provider supports:

  1. Treat the webhook as a trigger, not the source of truth. For providers like Stripe, have the consumer fetch the current object state and write that. Then late or reordered events converge on the same correct answer.
  2. Use a monotonic version where one exists. If the payload carries a version or sequence number for the object, enforce it at the database layer:
Code example
UPDATE subscriptions
SET status = :new_status,
    tier = :new_tier,
    last_event_version = :event_version,
    updated_at = NOW()
WHERE subscription_id = :sub_id
  AND (last_event_version IS NULL OR last_event_version < :event_version);
  1. Use timestamps only as a last resort, and know their limits. If you must, compare against the provider’s own timestamp, treat ties explicitly, and prefer re-fetching state when a tie or ambiguity arises.

If the WHERE clause matches zero rows, the event is stale. The consumer records STALE_EVENT_IGNORED in error_context, marks the inbox row COMPLETED, and leaves current state untouched.

Building Automated Reconciliation Checks

Even with an inbox ledger and idempotent workers, distributed systems still lose events. Network partitions happen, conditional branches hide bugs, and sometimes the provider never sends the event at all. Reconciliation is the background safety net that detects divergence between what the provider believes happened and what your database committed.

1. The Internal Orphan Sweep (SLA Watchdog)

The highest-leverage check runs purely against your own webhook_inbox. Any event that entered with a 200 OK should reach COMPLETED within a defined service level objective, such as five minutes (choose a number that fits your workload).

Run a scheduled query every minute:

Code example
SELECT inbox_id, provider, provider_event_id, event_type, status, received_at, attempts
FROM webhook_inbox
WHERE status IN ('RECEIVED', 'QUEUED', 'PROCESSING')
  AND received_at < NOW() - INTERVAL '5 minutes'
ORDER BY received_at ASC
LIMIT 500;

Automated remediation:

  • Stuck in RECEIVED: the outbox publisher missed it, so re-enqueue it.
  • Stuck in PROCESSING with an expired lease: the worker likely crashed or was killed mid-flight, so reclaim the lease and re-drive.
  • Past a maximum age (say, 30 minutes) or retry count: page the on-call engineer with the inbox_id and error_context.

The same logic applies to your broker-level DLQ. Alert on any message arriving there, and build replay tooling before you need it, so that dead-lettered messages can’t quietly age out.

2. Cross-System State Reconciliation (Provider vs. Internal Ledger)

Internal sweeps catch events that entered your system and stalled. Cross-system reconciliation catches events that were dropped before persistence or silently ignored because of business-logic bugs.

Run a batch job regularly (hourly is a reasonable starting point, and it must be well inside the provider’s recovery windows) that compares a rolling lookback window:

  1. Fetch authoritative events from the provider’s API. For Stripe, list events through the Events API. It returns only events from the last 30 days, and supports filtering with delivery_success=false to find events that failed to reach your endpoint. For GitHub, list webhook deliveries through the REST API and check for statuses other than OK, as GitHub’s own guidance suggests. Leave a buffer (for example, ignore the most recent 15 minutes) so in-flight processing isn’t flagged as drift.
  2. Diff against webhook_inbox and your domain tables:
    • Missing inbox records: The provider has event evt_123, but it isn’t in webhook_inbox. This indicates an ingress drop or delivery failure. Fetch the event and inject it into the inbox.
    • Completed inbox, mismatched domain state: webhook_inbox says invoice.paid is COMPLETED, yet invoices.status is still OPEN. This indicates a phantom side effect or a silent conditional skip in consumer code.
    • Orphaned internal state: A provisioned resource has no corresponding provider event or active subscription.

3. Canary Webhooks (Synthetic Heartbeats)

Don’t wait for real customer traffic to discover that a deployment broke the consumer pipeline. Inject synthetic canary webhooks through your real public ingress every few minutes, signed with a test key. Track each canary’s inbox_id through to a dummy sink table. If a canary gets a 200 OK at the edge but doesn’t reach the sink within your chosen threshold (say, 15 seconds), raise a pipeline health alert.

Timeline Forensics: Reconstructing What Actually Happened

When a customer reports a discrepancy or reconciliation flags an orphaned record, engineers often burn hours grepping disconnected logs across gateways, pods, and database audit trails. Effective forensics means stitching every step of a webhook’s life into one queryable trace.

Propagating Correlation Context Across Async Boundaries

Distributed tracing breaks at queue boundaries unless context is carried explicitly. OpenTelemetry’s messaging conventions say a producer should attach a message creation context to each message, and that without it consumer traces can’t be directly correlated with producer traces. In practice that means serializing the W3C traceparent (and tracestate, if present) into message metadata, and also storing it in the webhook_inbox row so a trace can be found long after the broker has discarded the message. Consumers typically link their spans to the producer’s creation context.

Then record four immutable timestamps per event:

  1. T0, provider_created_at: Taken from the payload or provider header. Shows upstream delays before the request reached you.
  2. T1, ingress_received_at: Recorded when your edge verifies the signature and commits the inbox row. T1 − T0 separates provider lag from internal lag.
  3. T2, consumer_started_at: Recorded when a worker leases the event. T2 − T1 isolates queue backlog and broker saturation.
  4. T3, terminal_committed_at: Recorded when the final side effect commits and the status flips to COMPLETED. T3 − T2 isolates lock contention and slow downstream calls.

Forensic Investigation Runbook

When auditing a missing or corrupted entity (for example, order_id = ord_88412):

  1. Query the inbox ledger by entity. Using the indexed correlation_key, pull every webhook received for that entity in chronological order, including ignored, stale, and failed events.
  2. Inspect side-effect granularity. Join webhook_inbox with webhook_side_effects. Seeing update_ledger: SUCCEEDED next to provision_tenant: PENDING narrows a vague mystery to one service call.
  3. Diff before-and-after state. If your domain tables have audit triggers, logical replication, or event sourcing, inspect row-level changes around T2 and T3. Often the webhook did succeed, and a separate job or race condition reverted the state minutes later.
  4. Replay deterministically. Never repair a failed webhook by hand-editing production tables. That bypasses downstream effects such as analytics events, outbound webhooks to your own customers, and cache invalidation. Instead, reset the inbox row from FAILED to RECEIVED and let the side-effect ledger skip completed steps and run only the missing ones. Note that manual resends from the provider side don’t cancel its automatic retries. Stripe’s documentation notes that resending an event doesn’t stop the automatic retry schedule, so your dedupe layer must handle both arriving.

Implementation Checklist for Engineering Teams

To harden an existing pipeline without a rewrite, prioritize in this order:

  • Persist before you acknowledge. Never return 2xx until the raw payload is in a durable webhook_inbox table or a replicated broker with acks=all. If persistence fails, return a 5xx.
  • Bind broker acknowledgments to terminal state. Disable auto-commit, and acknowledge or delete the message only after all database mutations and external side effects have succeeded.
  • Implement per-side-effect idempotency. Guard every external call and mutation with a key derived from (provider, provider_event_id, effect_name).
  • Deploy an orphan watchdog. Alert whenever any inbox row stays non-terminal (RECEIVED, QUEUED, PROCESSING) beyond your SLO.
  • Alert on and replay your DLQ. Verify its retention exceeds the source queue’s, and remember that on SQS standard queues the original enqueue time governs expiry.
  • Make ordering safe. Re-fetch current state where the provider recommends it, and enforce monotonic versions where available.
  • Run a scheduled provider-vs-internal reconciler. Compare provider records against your state on a cadence well inside the provider’s recovery window (three days for GitHub redelivery, 30 days for Stripe’s Events API), and escalate drift automatically.
  • Run canaries. Send synthetic signed webhooks through the real ingress and verify they reach a sink.

Conclusion

Returning 200 OK at the edge is a necessary optimization, not a guarantee of business correctness. The provider’s retries only cover the hop to your front door, and in GitHub’s case they don’t exist at all. Everything after the acknowledgment is your responsibility. By pairing fast perimeter acknowledgment with a transactional inbox, per-step idempotency, explicit terminal states, and automated cross-system reconciliation, you can make sure every accepted webhook finishes what it started, or at least that you find out quickly when it doesn’t.

Sources