The Webhook Fan-Out Pattern: Routing One Event to Multiple Microservices

When a customer completes a checkout on your platform, a single event occurs: a payment succeeds. But inside a modern distributed system, that one event triggers a cascade of downstream work across several microservices:

• Order Service — updates the core database (orders.status = 'paid')
• Notification Service — sends a transactional receipt
• Internal Ops — posts an alert to a Slack channel
• CRM Integration — updates customer lifetime value
• Data/Analytics — streams the transaction into a warehouse for reporting

If your backend receives that payment webhook and tries to handle all five operations synchronously, inside one HTTP request handler, your system will eventually break — usually at the worst possible moment (a traffic spike, a slow third party, a partial outage). That's the problem the webhook fan-out pattern solves.

This article covers why inline webhook handling fails at scale, how fan-out architecture fixes it, the real timeout/retry rules used by major webhook providers, the distributed-systems edge cases (idempotency, ordering, rate limits) you have to handle either way, and the actual build-vs-buy landscape as of 2026.

The Naive Approach: Why Synchronous Webhook Handling Fails

The Monolithic Handler Anti-Pattern

In a naive implementation, your API exposes a single endpoint (e.g. POST /api/webhooks/stripe). The handler receives a payload and synchronously works through every downstream dependency in sequence:

// ANTI-PATTERN: Monolithic Synchronous Webhook Handler
app.post('/api/webhooks/stripe', async (req, res) => {
  const event = req.body;

  if (event.type === 'payment_intent.succeeded') {
    try {
      await updateDatabase(event.data);        // Step 1: DB write
      await sendEmailReceipt(event.data);       // Step 2: external API call
      await postSlackNotification(event.data);  // Step 3: external API call
      await updateCRM(event.data);              // Step 4: external API call

      return res.status(200).send({ received: true });
    } catch (error) {
      // Which step failed? What should be retried, and what shouldn't?
      return res.status(500).send({ error: error.message });
    }
  }
});

Why this breaks in production

1. You're racing a real, provider-enforced clock. Webhook senders don't wait indefinitely for a response, and the limits are tighter than most people assume:

If your handler chains four sequential API calls that each take a few hundred milliseconds to a few seconds, you can blow through any of these windows — and a slow fourth step causes the entire delivery to be marked failed, even though steps 1–3 already succeeded.

2. Cascading failures cause duplicate side effects. If your CRM call throws an unhandled exception, the function returns a 500. The provider retries the whole webhook later. On retry, the database write and the email send — which already succeeded the first time — run again, causing duplicate charges, duplicate emails, or duplicate records.

3. Your uptime becomes coupled to every third party you call. If Slack, your CRM, or your email provider has a slow day, your core webhook ingestion — the thing that has to stay reliable — inherits that slowness.

4. Bursts overwhelm your worker pool. A batch of GitHub push events or a bulk product-sync from an e-commerce platform can spike inbound volume by orders of magnitude in seconds. If webhook processing shares infrastructure with your customer-facing API, that spike can degrade or take down endpoints that have nothing to do with the webhook.

What Is the Webhook Fan-Out Pattern?

The fan-out pattern (also called event broadcasting) ingests an event exactly once, acknowledges it immediately, and then distributes it asynchronously into independent delivery pipelines — one per destination.

                               ┌─────────────────────┐ ──> [Database Service]
                             │  Delivery Queue 1   │
                             └─────────────────────┘
                                       ▲
┌────────────────┐   ┌──────────┐     │
│ Webhook Source  │──>│ Ingestion│─────┼──────────────> [Email Service]
│ (e.g., Stripe)  │   │ Gateway  │     │
└────────────────┘   └──────────┘     ▼
                             ┌─────────────────────┐ ──> [Slack Notification]
                             │  Delivery Queue 3   │
                             └─────────────────────┘

Instead of looping through destinations inside a request handler, the event is written to durable storage or a message broker, which fans it out into N independent delivery tasks for N destinations. Each destination then operates in isolation:

• A failure in Service B doesn't delay delivery to Service A.
• A slow response from Service C doesn't block acknowledgment of the original event.
• Each destination gets its own retry schedule, rate limit, and dead-letter queue (DLQ).

Architectural comparison

Anatomy of a Fan-Out System

A production-grade fan-out architecture has four layers:

[Ingestion]  ──>  [Storage & Routing]  ──>  [Dispatch Workers]  ──>  [Destinations]
 (verify &            (fan-out to N            (parallel HTTP           (your
  accept fast)          matching targets)        deliveries)             microservices)

1. Ingestion layer. A stable, highly available public endpoint that:

• Validates the payload's HMAC signature (Stripe-Signature, X-Hub-Signature-256, etc.) to confirm authenticity before trusting the payload.
• Writes the raw payload to durable storage immediately.
• Returns a 2xx response — ideally in well under 100ms, and always inside whatever window the sending provider enforces.

2. Fan-out / routing engine. Once stored, the router checks which destinations are subscribed to this event type and creates one delivery job per match. Filtering and transformation rules can be applied here — e.g., route payment_intent.failed only to an on-call alerting tool, but route all payment events to a data warehouse.

3. Asynchronous worker pool. Independent workers pull jobs and execute the HTTP calls to each destination in parallel, with per-destination concurrency and rate limits — you don't want a burst of events to blow past Slack's rate limits just because your database can absorb the same burst fine.

4. Retry engine and dead-letter queue. On a transient failure (HTTP 429, 502, 503, timeout), the specific delivery job is retried with backoff — without affecting other subscribers. If retries are exhausted, the job lands in a per-endpoint DLQ for inspection and manual replay.

Overcoming the Real Distributed-Systems Challenges

Fan-out solves coupling and latency, but distributing one event to many services introduces the usual distributed-systems problems. These apply whether you build the fan-out layer yourself or use a managed gateway.

1. Idempotency

Nearly every major webhook provider — Stripe, GitHub, Shopify — delivers on an at-least-once basis. Retries, timeout ambiguity, and manual redeliveries all mean your downstream consumers will eventually see the same event more than once. Stripe's own documentation is explicit about this: it recommends checking whether an event has already been processed and returning a success response for duplicates rather than reprocessing them.

The standard fix is a unique event ID plus a fast atomic lock:

// Idempotency guard using an atomic "set if not exists"
async function handleWebhook(req, res) {
  const eventId = req.headers['x-event-id'] || req.body.id;
  const lockKey = `idempotency:${eventId}`;

  // NX = only set if the key doesn't already exist
  const isNewEvent = await redis.set(lockKey, 'processing', 'EX', 86400, 'NX');

  if (!isNewEvent) {
    console.log(`Duplicate event received: ${eventId}. Skipping.`);
    return res.status(200).json({ status: 'ignored', reason: 'duplicate_event' });
  }

  try {
    await processBusinessLogic(req.body);
    await redis.set(lockKey, 'completed', 'EX', 86400);
    return res.status(200).json({ status: 'success' });
  } catch (error) {
    // Release the lock on failure so a legitimate retry can run
    await redis.del(lockKey);
    return res.status(500).json({ error: 'Processing failed' });
  }
}

2. Out-of-order delivery

Because fan-out workers process concurrently, events can arrive at a destination out of sequence — an order.updated event might land before order.created finishes retrying. The fix is to carry a monotonic sequence number or timestamp in the payload and discard stale updates:

async function updateCustomerRecord(customerId, payloadTimestamp, newData) {
  const customer = await db.findCustomer(customerId);

  if (customer && new Date(customer.last_updated_at) > new Date(payloadTimestamp)) {
    console.warn(`Stale event for customer ${customerId}. Discarding.`);
    return;
  }

  await db.updateCustomer(customerId, { ...newData, last_updated_at: payloadTimestamp });
}

3. Rate limits and the "noisy neighbor" problem

A burst of thousands of events in a few seconds (a bulk price sync, a mass CSV import) will crash a downstream service if it's pushed through unthrottled. Fan-out systems should enforce per-destination rate limits — a high-capacity database endpoint and a Slack webhook (which enforces its own strict rate limits) need very different throttles.

4. Backoff needs jitter, not just exponentiation

Plain exponential backoff reduces retry frequency, but if many clients fail at the same moment, they stay synchronized and retry in clusters, which just moves the thundering herd to a later time instead of eliminating it. AWS's widely cited 2015 engineering write-up on this ("Exponential Backoff and Jitter," still referenced in AWS's Well-Architected Framework today) showed that adding randomized jitter on top of exponential backoff is what actually spreads retries out and prevents synchronized retry storms. Most current SDKs and retry libraries implement this combination by default now, so in practice you rarely need to hand-roll it — but it's worth knowing why plain backoff alone is not enough.

5. Signature verification on the way out, too

If you're operating a fan-out relay in front of your own microservices, sign your outgoing requests (a timestamped HMAC is the common pattern, mirroring how Stripe and GitHub sign their own webhooks) so your internal services can verify a delivery actually came from your gateway and not from an internal network compromise.

Build vs. Buy: DIY Queues vs. a Managed Webhook Gateway

Teams generally choose between two approaches:

DIY on cloud messaging infrastructure. The classic pattern on AWS is SNS (or EventBridge) publishing to multiple SQS queues, each consumed independently — you publish once, and every subscribed queue gets its own durable copy, so a slow or failing consumer never blocks the others. This is the same underlying pattern described above, built from primitives you already have if you're on AWS.

Provider ──> API Gateway ──> SNS Topic ──> SQS Queue A ──> Lambda / Worker A
                                        └─> SQS Queue B ──> Lambda / Worker B
                                        └─> SQS Queue C ──> Lambda / Worker C

This gives you full control and no vendor lock-in, at the cost of building and maintaining the retry logic, DLQ wiring, per-destination rate limiting, HMAC verification, and a debugging UI yourself.

Managed webhook gateways. A newer category of dedicated infrastructure exists specifically to remove that operational burden. As of 2026, the more established options include:

• Hookdeck Event Gateway — purpose-built for receiving inbound webhooks, with fan-out to multiple destinations, durable queueing, filtering/transformation rules, and delivery-level observability.
• Svix — split into Svix Ingest (inbound, with JS-based fan-out routing) and Svix Dispatch (outbound delivery for products that send webhooks to their customers), plus an open-source self-hosted server.
• Convoy — an open-source, self-hostable option that unifies sending and receiving.
• Smaller/regional options such as Hook0 (EU-hosted, lower-volume focus).

Before committing to any of these, check their current docs and pricing directly — this space has moved fast, feature sets and plan tiers change, and open-source maintenance status varies project to project.

Neither approach eliminates the idempotency, ordering, and rate-limiting work described above — that logic still has to live somewhere, either in your own workers or configured into the gateway you choose.

Production Checklist

• Acknowledge fast. Return a 2xx well inside the sending provider's timeout window — GitHub gives you 10 seconds, Shopify gives you 5. Don't perform external network calls before responding.
• Enforce idempotency. Every downstream consumer checks a unique event ID before taking action, because at-least-once delivery means duplicates will happen.
• Verify signatures. Validate HMAC signatures on both the inbound provider webhook and any internal fan-out deliveries.
• Isolate destinations. Give each destination its own queue/retry schedule so a slow or broken service never blocks the others.
• Configure DLQs. Route exhausted-retry payloads somewhere inspectable, with alerting.
• Use exponential backoff with jitter, not backoff alone, to avoid synchronized retry storms.
• Have a redelivery/reconciliation plan. Some providers (GitHub) don't auto-retry at all, and even the ones that do have finite windows (hours to a few days) — build a way to reconcile via the provider's API for anything that falls outside that window.

Summary

Coupling downstream microservices directly to an inbound webhook handler creates brittle systems: slow API responses, duplicate side effects on retry, and availability that's hostage to every third party you call. The fan-out pattern fixes this by ingesting an event once, acknowledging it immediately, and distributing it into independent, isolated delivery pipelines — whether you build that on top of SNS/SQS, or adopt a managed gateway to skip the undifferentiated infrastructure work. Either way, idempotency, ordering guards, and per-destination rate limiting are the pieces that make the pattern actually reliable in production, not optional extras.

Sources & further reading

• GitHub: Best practices for using webhooks — 10-second response window
• GitHub: Handling failed webhook deliveries — no automatic retry
• Shopify Engineering: Webhook best practices
• Stripe: Receive Stripe events in your webhook endpoint
• Stripe: Process undelivered webhook events — idempotent processing guidance
• Sentry: Integration platform webhook timeout change
• AWS Architecture Blog: Exponential Backoff and Jitter
• AWS Well-Architected Framework: Control and limit retry calls
• Hookdeck: Webhook platform guides
• Svix: Webhook resources