Zero-Downtime Webhook Migrations: Changing Endpoints Without Data Loss

Every DevOps engineer and backend developer has felt the anxiety of migrating a live webhook consumer. Whether you're re-architecting a monolith into microservices, switching cloud providers, or restructuring your API domains, changing a webhook endpoint URL in production has historically been fraught with danger.

Unlike standard REST API calls — where your client controls the retry logic — webhooks invert control. Third-party providers like Stripe, GitHub, Shopify, and Twilio push events to your public endpoints on their own schedule, with their own (often very different) rules about what happens when delivery fails. Update that URL carelessly and you can end up with hours of silently dropped payments, signups, or inventory syncs.

This guide covers why direct endpoint switching fails, how a persistent webhook proxy/gateway pattern solves it, how to preserve signature verification in transit, and how to replay missed events after an outage — updated with current (2026) provider retry behavior and the real tools teams use to build this today.

1. Why Direct Webhook Endpoint Switching Fails

In a naive migration, an engineer opens the provider's dashboard, changes the destination URL from https://api-v1.example.com/webhooks to https://api-v2.example.com/webhooks, and clicks save.

+------------------+         Direct Endpoint Change         +-------------------------+
|                  | -------------------------------------> | Old Endpoint (Dying)    |
| Webhook Provider |    (Connection reuse / in-flight reqs) | https://api-v1.example  |
| (Stripe, GitHub) |                                        +-------------------------+
|                  |
|                  |                                        +-------------------------+
|                  | -------------------------------------> | New Endpoint (Cold)     |
|                  |      (Delayed traffic switch)          | https://api-v2.example  |
+------------------+                                        +-------------------------+

This introduces four real structural failure points.

A. HTTP Keep-Alive and Connection Reuse

The old article framed this purely as "DNS TTL caching," but the more precise (and better documented) mechanism is connection pooling. Many HTTP clients — including the ones vendor infrastructure runs on — reuse persistent keep-alive connections rather than re-resolving DNS on every request, because that avoids repeating the DNS + TCP + TLS handshake cost on each call. If a sender's dispatcher is mid-connection to your old server when you cut over, it will keep sending requests down that connection until it's forcibly closed or times out, regardless of what your DNS record now says. Node.js Agent/HttpClient-style keep-alive pools are a well-documented source of this exact problem when teams do DNS-based traffic switching.

B. Asynchronous In-Flight Payload Drops

At the instant you terminate the old server or redirect traffic, in-flight HTTP POSTs mid-flight terminate with ECONNRESET, 502 Bad Gateway, or 503 Service Unavailable. What happens next depends entirely on the provider — which is where most teams get surprised.

C. Provider Retry Policies Are Wildly Inconsistent

This is the part worth getting exactly right, because retry behavior changes over time and isn't uniform across providers. Here's the current state as of 2026:

The upshot: the original assumption that "most providers retry 3–5 times and give up" undersells the risk in both directions. Stripe is actually fairly forgiving (3 days). GitHub is the opposite extreme — it doesn't retry automatically at all, which means a few minutes of downtime during a naive cutover can permanently lose events unless you're polling its Deliveries API or redelivering manually. Don't assume uniform behavior across providers; check each one's current docs before you plan a migration around them.

D. The Idempotency Gap

Without a centralized event queue, a "dual-endpoint" cutover can let both the old and new backend process the same payload simultaneously, causing double-charges, race conditions, or corrupted state. This risk doesn't go away with a proxy — it just moves to needing explicit idempotency keys, covered in Section 4.

2. The Solution: A Webhook Proxy / Gateway in Front of Your Consumer

The fix is to decouple ingestion from processing. A permanent, boring, ultra-reliable proxy — a static gateway URL that never changes — insulates your application infrastructure from both provider configuration and your own internal re-architecture.

+------------------+         Static URL (Never Changes)        +------------------------+
| Webhook Provider | ----------------------------------------> | Webhook Proxy / Gateway|
| (Stripe/GitHub)  |   https://hooks.yourdomain.com/v1/ingest  | (Buffer / Dynamic Router)
+------------------+                                           +------------------------+
                                                                            |
                                           +--------------------------------+--------------------------------+
                                           | (Shadow phase / dynamic cutover)                                |
                                           v                                                                  v
                            +-----------------------------+                                  +-----------------------------+
                            | Legacy Consumer Endpoint    |                                  | Modern Consumer Endpoint    |
                            | https://api-v1.internal/    |                                  | https://api-v2.internal/    |
                            +-----------------------------+                                  +-----------------------------+

Build it yourself, or use an existing tool

The original version of this guide referenced a fictional "InstaWebhook" product. In practice, the 2026 landscape has real, purpose-built options, and it's worth knowing which category each one solves before picking one:

• Hookdeck Event Gateway — an inbound webhook proxy: ingestion, buffering, transformation, routing, and retries for webhooks arriving at you. This is the closest match to the "proxy shield" pattern described here.
• Svix — primarily an outbound webhook-sending platform (used by companies like Clerk, Brex, and Resend to deliver webhooks to their own customers), with a newer "Ingest" product for inbound receiving.
• Convoy — an open-source, self-hostable gateway (written in Go) that handles both inbound and outbound webhook delivery, useful if you need to keep everything on your own infrastructure.
• Hook0 — a smaller open-source option, mainly for outbound sending, with EU data residency as a differentiator.
• ngrok — useful for local development tunneling, but not a production migration tool.

If you'd rather not adopt a third-party dependency, the DIY pattern below (proxy + durable buffer + DLQ) is exactly what these tools implement internally, and is a reasonable choice if your volume is modest or you have strict infrastructure constraints.

Key Components

1. Static ingestion URL — e.g. https://hooks.yourdomain.com/v1/stripe. You register this once with the provider and never touch it again.
2. Durability buffer — the proxy writes raw incoming HTTP headers and bytes to an append-only store (Redis Streams, AWS SQS, NATS, Kafka) and immediately responds 200/202 to the provider.
3. Dynamic forwarding router — an internal worker layer that reads from the buffer and forwards payloads to your current backend target. The target can change in real time via config, without touching the provider.
4. Dead letter queue (DLQ) + replay engine — an automated store for payloads that fail at your target backend, so you can replay them without asking the vendor to resend.

3. Step-by-Step: Executing a Zero-Downtime Migration

Walking through migrating a payment webhook consumer from api-v1.company.internal to api-v2.company.internal.

Phase 1: Deploy the Proxy Layer

Below is a minimal Node.js/Express proxy demonstrating payload buffering and dynamic target forwarding. It's illustrative — in production you'd likely reach for one of the tools above, or harden this considerably (auth on the admin route, structured logging, backpressure handling).

import express, { Request, Response } from 'express';
import axios from 'axios';
import { createClient } from 'redis';

const app = express();
// Capture raw body buffer to preserve signature verification integrity
app.use(express.raw({ type: '*/*', limit: '10mb' }));

const redis = createClient({ url: process.env.REDIS_URL });
redis.connect();

// Target destination managed dynamically (via DB, Redis key, or env)
let CURRENT_TARGET_URL = process.env.INITIAL_TARGET_URL || 'https://api-v1.company.internal/webhooks';

// Dynamic route management API — protect this behind auth in production
app.post('/admin/set-target', express.json(), (req: Request, res: Response) => {
  const { newTargetUrl } = req.body;
  if (!newTargetUrl) return res.status(400).json({ error: 'Missing newTargetUrl' });

  CURRENT_TARGET_URL = newTargetUrl;
  console.log(`[ROUTE CHANGED] Forwarding traffic to: ${CURRENT_TARGET_URL}`);
  return res.status(200).json({ success: true, activeTarget: CURRENT_TARGET_URL });
});

// Primary webhook ingestion endpoint
app.all('/v1/ingest/*', async (req: Request, res: Response) => {
  const eventId = `evt_${Date.now()}_${Math.random().toString(36).substring(2, 9)}`;
  const targetUrl = CURRENT_TARGET_URL;

  const rawHeaders = { ...req.headers };
  delete rawHeaders['host'];
  delete rawHeaders['content-length'];

  const payload = {
    id: eventId,
    path: req.params[0],
    headers: rawHeaders,
    body: req.body.toString('utf-8'),
    targetUrl,
    timestamp: new Date().toISOString(),
  };

  try {
    // 1. Immediately buffer raw payload to a durable store
    await redis.lPush('webhook_audit_log', JSON.stringify(payload));

    // 2. Forward payload to the currently active consumer endpoint
    const response = await axios({
      method: req.method,
      url: `${targetUrl}/${req.params[0]}`,
      data: req.body,
      headers: { ...rawHeaders, 'X-Proxy-Event-ID': eventId },
      timeout: 5000,
      validateStatus: () => true, // don't throw on non-2xx
    });

    if (response.status >= 200 && response.status < 300) {
      return res.status(200).send({ received: true });
    } else {
      await redis.lPush('webhook_dlq', JSON.stringify({ ...payload, error: `HTTP ${response.status}` }));
      return res.status(200).send({ received: true, buffered: true });
    }
  } catch (error: any) {
    await redis.lPush('webhook_dlq', JSON.stringify({ ...payload, error: error.message }));
    // Acknowledge the vendor to avoid endpoint disablement; retain payload in DLQ
    return res.status(200).send({ received: true, buffered: true });
  }
});

app.listen(3000, () => console.log('Proxy Shield active on port 3000'));

Phase 2: Preserving Signature Verification

The most common technical hurdle when inserting a proxy is HMAC signature verification. Providers sign payloads with headers like Stripe-Signature or X-Hub-Signature-256; a growing number follow the open Standard Webhooks specification, which uses webhook-id, webhook-timestamp, and webhook-signature headers with HMAC-SHA256.

To avoid breaking verification:

• Pass raw body bytes unmodified. Don't parse req.body into a JSON object before forwarding — re-serializing changes key ordering and whitespace, which invalidates the hash.
• Preserve original headers, including the timestamp used for replay protection. The Standard Webhooks spec recommends rejecting requests with a timestamp more than 300 seconds (5 minutes) old; Stripe's libraries use a similar tolerance window.
• Keep the byte stream intact end-to-end — over mTLS or private networking if your proxy terminates HTTPS and re-encrypts to the target.

Phase 3: Shadow Traffic Testing

Before cutting over fully, run shadow mode: forward 100% of production traffic to api-v1 (primary) while duplicating identical traffic to api-v2 in the background, without letting api-v2 write to production data.

Phase 4: Atomic Cutover

Once shadow metrics show a 0% error rate on v2, flip the route with a single control-plane call:

curl -X POST https://hooks.yourdomain.com/admin/set-target \
  -H "Content-Type: application/json" \
  -d '{"newTargetUrl": "https://api-v2.company.internal/webhooks"}'

The switch happens in memory — no DNS propagation, no provider dashboard changes. Monitor proxy queue depth and response codes on v2 closely for the first hour.

4. Replaying Missed Webhooks After an Outage

Even with careful planning, target endpoints will have transient failures — connection pool exhaustion, a dependency outage, a bad deploy. You need to replay missed events without causing duplicate side effects.

                                 DEAD LETTER QUEUE (DLQ) REPLAY FLOW

+--------------------+        Failed Processing        +-----------------------+
|  Proxy Gateway     | ------------------------------> | Dead Letter Queue     |
|  Buffer Engine     |                                 | (Storage / Redis DLQ) |
+--------------------+                                 +-----------------------+
                                                                   |
                                                                   | Manual / Automated Trigger
                                                                   v
+--------------------+           Replay Payload        +-----------------------+
| Target Consumer    | <------------------------------ | Replay CLI / Worker   |
| (api-v2.internal)  |    (With Idempotency Key)       | Script                |
+--------------------+                                 +-----------------------+

Idempotent Handlers

Every payload needs a unique event ID (evt_12345, or a provider-supplied X-Event-ID/X-GitHub-Delivery/MessageSid), which your handler checks before doing any real work:

app.post('/webhooks/stripe', async (req: Request, res: Response) => {
  const eventId = req.headers['x-proxy-event-id'] || req.body.id;

  const isProcessed = await redis.sIsMember('processed_events', eventId);
  if (isProcessed) {
    console.log(`[DUPLICATE IGNORING] Event ${eventId} was already processed.`);
    return res.status(200).json({ status: 'ignored_duplicate' });
  }

  await processPaymentEvent(req.body);

  await redis.sAdd('processed_events', eventId);
  await redis.expire('processed_events', 604800); // 7-day TTL
  return res.status(200).json({ status: 'success' });
});

This matters more than it sounds like it should: providers guarantee at-least-once delivery, never exactly-once, so duplicate deliveries are expected behavior even outside a migration — not a bug specific to your proxy.

Automated DLQ Replay

// replay-dlq.ts
import axios from 'axios';
import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });

async function replayFailedWebhooks() {
  await redis.connect();
  console.log('--- STARTING WEBHOOK REPLAY PROCESS ---');
  let replayedCount = 0;

  while (true) {
    const rawItem = await redis.rPop('webhook_dlq');
    if (!rawItem) break;

    const event = JSON.parse(rawItem);
    console.log(`[REPLAYING] Event ID: ${event.id} to Target: ${event.targetUrl}`);

    try {
      const res = await axios.post(`${event.targetUrl}/${event.path}`, event.body, {
        headers: { ...event.headers, 'X-Webhook-Replay': 'true', 'X-Proxy-Event-ID': event.id },
      });

      if (res.status >= 200 && res.status < 300) {
        replayedCount++;
        console.log(`[REPLAY SUCCESS] Event ID: ${event.id}`);
      } else {
        await redis.lPush('webhook_dlq_failed_twice', JSON.stringify(event));
      }
    } catch (err: any) {
      console.error(`[REPLAY FAILED] Event ID: ${event.id} Error: ${err.message}`);
      await redis.lPush('webhook_dlq_failed_twice', JSON.stringify(event));
    }
  }

  console.log(`--- REPLAY COMPLETE: ${replayedCount} events processed successfully. ---`);
  process.exit(0);
}

replayFailedWebhooks();

5. Direct Migration vs. Proxy Architecture

6. Best Practices Checklist

• Decouple ingestion from processing. Never expose application servers directly to third-party webhooks.
• Capture raw payloads first, before parsing or executing business logic.
• Enforce idempotency using event IDs stored with a TTL in a fast key-value store.
• Keep ingestion response times low — well under each provider's timeout (Shopify: 5s per attempt; Twilio voice: 15s hard limit).
• Preserve raw bytes and headers for HMAC verification; don't re-serialize JSON in the proxy.
• Alert on DLQ growth via PagerDuty, Datadog, or Slack thresholds.
• Check each provider's current retry policy before you rely on it — these change (Shopify's 2024 policy update is a good example) and are not uniform across vendors.
• Test your replay path regularly, not just during an actual incident.

FAQ

Q: What's the core idea behind a webhook proxy/gateway? It's a static, high-availability URL between third-party dispatchers and your backend. You register the proxy's URL with providers once; internal routing changes happen in your own config, not in each vendor's dashboard.

Q: How do I keep signature verification working through a proxy? Forward the exact raw, unparsed body and original headers. Re-serializing JSON breaks the hash comparison on the receiving end.

Q: What happens if my backend goes down mid-migration? With buffering in place, incoming webhooks are acknowledged and stored in a durable buffer or DLQ. Once your backend recovers, run a replay job to resend them in order.

Q: Does every provider retry failed webhooks the same way? No — and this is the most important correction to older advice on this topic. Stripe retries for up to 3 days with exponential backoff. Shopify currently retries 8 times over 4 hours (down from 19 over 48 hours before September 2024). GitHub does not automatically retry at all — you must manually redeliver within a 3–7 day window. Always check current docs for any provider you depend on rather than assuming a shared standard.

Sources

• Stripe — Webhook retry behavior (Svix review)
• Shopify — Updates to webhook retry mechanism (official changelog)
• Shopify — Troubleshoot webhooks (official docs)
• GitHub — Redelivering webhooks (official docs)
• Twilio — Webhooks (HTTP callbacks): Connection Overrides (official docs)
• Standard Webhooks specification
• Svix — What is a webhook signature?
• Best Webhook Management APIs in 2026 — APIScout