Managing High-Volume Blockchain Webhooks: Architecture, Retries, and Event Queuing at Scale

In Web2 architecture, webhooks arrive in a steady trickle. A payment processor fires one every time a customer checks out. A code host fires one every time someone pushes a commit. Traffic follows a daily curve, and a single-threaded handler can usually keep up.

Web3 doesn't work that way. Blockchains produce state changes in discrete, synchronized bundles called blocks. Roughly every 12 seconds on Ethereum mainnet — or every 1-2 seconds on rollups like Arbitrum, Base, and Optimism — hundreds or thousands of smart contract logs are emitted at once. When you configure a provider to push webhooks for wallet activity, DEX swaps, NFT mints, or liquidation monitoring, your ingestion server doesn't see a trickle. It sees a wall.

This piece covers how that traffic pattern breaks naive webhook handlers, what the current generation of providers (Alchemy, QuickNode, Moralis, and others) actually guarantee about delivery and retries, how Ethereum's post-Merge finality model changes how you should think about reorgs, and a production-grade ingestion architecture you can build on top of any of them.

Why Block-Synchronized Traffic Breaks Web2 Assumptions

Web2 Traffic Pattern:
Traffic  ▲      _.-""-._
         │    .'        '.
         └────┴──────────┴────────► Time

Web3 Block Burst Pattern:
Traffic  ▲   ||   ||   ||   ||
         │   ||   ||   ||   ||
         └───┴┴───┴┴───┴┴───┴┴────► Time (Block times: ~12s / ~1-2s)

When a block lands, your endpoint can receive dozens or hundreds of concurrent POST requests within milliseconds. If your handler does anything synchronous and slow — a relational database write, an external API call, business-logic branching — the requests queue up, sockets exhaust, and you start returning 5xx or timing out.

That triggers the second problem: provider retries. Retries exist to protect you from transient failures, but if your endpoint is already overwhelmed, incoming retries compound the original burst into a classic thundering-herd cascade. And if you stay unresponsive long enough, most providers will eventually pause or disable the webhook — which means you silently stop receiving events until someone notices.

The third problem is specific to blockchains: reorgs. A block that looked final a moment ago can be dropped from the canonical chain and replaced. Any downstream system that already acted on that block's events now needs to unwind that action.

How Ethereum Finality Actually Works (and Why "32 Confirmations" Is Outdated)

A lot of older Web3 content still describes reorg safety using proof-of-work heuristics — "wait for 32 confirmations" or "wait roughly a minute." That guidance predates Ethereum's move to proof-of-stake in September 2022 and no longer reflects how the chain works.

Since the Merge, Ethereum's Beacon Chain organizes time into slots (12 seconds each) and epochs (32 slots, so 6.4 minutes each). Blocks carry three commitment levels you can request directly from an execution client:

• latest — the current chain head. Low confidence; this is exactly the label that gets reorged.
• safe — a block the network has broadly attested to. Unlikely to be reorged under normal conditions, but not a cryptographic guarantee.
• finalized — a block protected by Casper FFG's two-round voting process. Reverting a finalized block would require at least one-third of all staked ETH to be provably slashed, which is considered economically and practically infeasible barring a coordinated attack.

Finality normally completes after two full epochs, which works out to roughly 12–13 minutes (768–960 seconds) under healthy network conditions — not 32 blocks or 64 seconds. For context, independent analysis of mainnet data from early 2026 found that the overwhelming majority of blocks are never reorged at all (well under 2% orphan rate over a 30-day sample), and the reorgs that do happen are almost always a single block deep. Deep, multi-block reorgs are rare events, not routine occurrences.

Practical implication for your pipeline:

• For low-stakes UI updates (showing a pending transfer, updating a "last seen" balance), acting on latest or a shallow k-block confirmation is usually fine.
• For anything that triggers an irreversible off-chain action — releasing fiat, shipping a physical good, crediting a custodial balance — wait for safe at minimum, and treat finalized as the actual safety bar.
• Layer 2 chains have their own, usually shorter, finality windows tied back to their L1 settlement schedule; don't assume Ethereum's numbers apply to Arbitrum, Base, or Optimism without checking each rollup's specific finalization delay.
• Bitcoin and other UTXO chains use a completely different, probabilistic confirmation-depth model (commonly 1, 3, or 6 confirmations depending on the counterparty's risk tolerance) rather than epoch-based finality.

The Current Webhook Provider Landscape

The original framing of "Alchemy vs. Infura" as two comparable webhook products is no longer (and was never quite) accurate. Infura — now operated under the MetaMask Developer / Consensys umbrella — is fundamentally a JSON-RPC and WebSocket provider. It gives you eth_subscribe log filters and pending-transaction and new-block-header streams, but it does not offer a native, hosted webhook-delivery product comparable to Alchemy Notify. If you build on Infura and want webhook-style delivery, you're responsible for running your own relayer that consumes the WebSocket stream and forwards it as HTTP callbacks — which reintroduces most of the reliability problems this article is about.

A more accurate comparison today is between the providers that actually operate managed webhook pipelines:

The practical takeaway: if you want a managed webhook product with reorg-aware delivery, Alchemy, QuickNode, and Moralis are the mainstream options as of 2026, each with slightly different guarantees around ordering and exactly-once vs. at-least-once delivery. Check the current docs before you build — retry windows, IP ranges, and plan-tier behavior are the kind of details that change between releases.

Anatomy of a Webhook Payload

The payload shape below is a simplified, representative example of an address-activity-style event (based on the general schema these providers use, not copied from any single vendor's live documentation). Build your parser defensively — treat unfamiliar fields as optional rather than assuming this exact shape.

{
  "webhookId": "wh_k9x2m1p0q8",
  "id": "evt_abc123xyz789",
  "createdAt": "2026-08-21T10:15:30.120Z",
  "type": "ADDRESS_ACTIVITY",
  "event": {
    "network": "ETH_MAINNET",
    "activity": [
      {
        "blockNum": "0x133f4a2",
        "hash": "0x8f2d9e61c52b8214a1e948f2195f00e2b96837df21463a56e9c17887e145ef2",
        "fromAddress": "0x71c7656ec7ab88b098defb751b7401b5f6d8976",
        "toAddress": "0x2170ed0880ac9a755fd29b2688956bd959f933f",
        "value": 1.5,
        "asset": "ETH",
        "category": "external",
        "log": {
          "address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "topics": [
            "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3e",
            "0x00000000000000000000000071c7656ec7ab88b098defb751b7401b5f6d8976",
            "0x0000000000000000000000002170ed0880ac9a755fd29b2688956bd959f933"
          ],
          "blockNumber": "0x133f4a2",
          "transactionHash": "0x8f2d9e61c52b8214a1e948f2195f00e2b96837df21463a56e9c17887e145ef2",
          "transactionIndex": "0x12",
          "blockHash": "0x4e8a1d7f6c3b2a1e0f9d8c7b6a5e4d3c2b1a0f9e8d7c6b5a4e3d2c1b0a9f8e7",
          "logIndex": "0x5",
          "removed": false
        }
      }
    ]
  }
}

Fields worth building your entire pipeline around:

• log.removed — the single most important flag in the payload. false means a valid, currently-canonical event. true means the block that produced it was reorged out; your pipeline needs to trigger a rollback for that specific log.
• transactionHash + logIndex — combined, these form a globally unique, deterministic identifier for a single log entry, which is exactly what you want for an idempotency key.
• blockHash + blockNumber — store both alongside anything you persist, so a later reorg check has something to compare against.

Production Architecture: Decouple Ingestion from Processing

To handle millions of events a day without data loss, split the system into two tiers:

• Ingestion tier (fast, stateless): verifies the signature, pushes the raw payload onto a queue, returns 200 OK.
• Worker tier (asynchronous, stateful): consumes the queue, deduplicates, checks for reorgs, and applies business logic.

                ┌─────────────────────────────────────────┐
                │    Webhook Provider                      │
                │    (Alchemy / QuickNode / Moralis)       │
                └────────────────────┬──────────────────────┘
                                     │ HTTP POST (block-burst traffic)
                                     ▼
                ┌─────────────────────────────────────────┐
                │   Ingestion Edge Server                  │
                │   1. Validate HMAC signature              │
                │   2. Return 200 OK (< 50ms)               │
                └────────────────────┬──────────────────────┘
                                     │ Publish payload
                                     ▼
                ┌─────────────────────────────────────────┐
                │   Message Buffer                         │
                │   (Redis / BullMQ / Kafka / SQS)         │
                └────────────────────┬──────────────────────┘
                                     │ Consume asynchronously
                                     ▼
                ┌─────────────────────────────────────────┐
                │   Worker Pool                             │
                │   1. Idempotency check                    │
                │   2. Check 'removed' flag (reorg)         │
                │   3. Execute business logic                │
                └────────────────────┬──────────────────────┘
                                     ▼
                ┌─────────────────────────────────────────┐
                │   State Storage (Postgres / DynamoDB)    │
                └─────────────────────────────────────────┘

Four Pillars of a Reliable Pipeline

Pillar 1 — Fast, Dumb Ingestion

Never touch a relational database or do heavy computation inside the HTTP handler itself. It should do exactly three things:

1. Verify the signature (X-Alchemy-Signature or the equivalent HMAC header for your provider).
2. Push the raw JSON body onto a queue.
3. Return 200 OK immediately.

Alchemy's own documentation confirms why this matters: their retry backoff on non-2xx responses runs up to roughly 10 minutes on Free/PAYG plans and up to an hour on Enterprise — but every one of those retries is additional load on an endpoint that's already struggling. Get under a few hundred milliseconds, and this entire failure mode disappears.

Pillar 2 — Idempotency and Deduplication

Assume every event can be delivered more than once — that's not a bug, it's the explicit contract most providers offer ("at-least-once delivery"). Build a deterministic key and gate on it before doing anything stateful:

IdempotencyKey = `${network}:${transactionHash}:${logIndex}`

A plain string concatenation is sufficient here — you don't need a cryptographic hash like keccak256 for this; you need uniqueness and determinism, which the raw fields already give you.

async function processWebhookEvent(event: WebhookEvent) {
  const { transactionHash, logIndex, removed } = event.log;
  const network = event.network;
  const idempotencyKey = `evt:${network}:${transactionHash}:${logIndex}`;

  // Atomic check-and-set; NX = only set if not already present
  const acquired = await redis.set(idempotencyKey, "PROCESSED", "NX", "EX", 86400);

  if (!acquired && !removed) {
    console.log(`[DUPLICATE] ${idempotencyKey} already processed. Skipping.`);
    return;
  }

  if (removed) {
    await handleChainReorg(event);
    return;
  }

  await executeBusinessLogic(event);
}

Pillar 3 — Reorg Handling, Sized to the Actual Finality Window

• Store blockHash and blockNumber next to every record you write.
• On removed: true, look up the matching transactionHash + logIndex and revert or flag the affected records (e.g., status = 'REORGD_OUT').
• For financial actions with real-world consequences, don't act on latest. Hold the event in a pending_confirmation table until it reaches safe, and treat finalized (~12-13 minutes on Ethereum mainnet under normal conditions) as your actual settlement bar — not the older, pre-Merge "32 blocks" heuristic.
• If you're on a provider like QuickNode Streams that re-delivers the corrected block on reorg rather than sending an explicit rollback flag, design your ingestion for upserts (ON CONFLICT ... DO UPDATE) instead of assuming every delivery is a first write.

Pillar 4 — Client-Side Retries and a Dead Letter Queue

Don't rely on the provider to re-send a webhook if your worker fails after ingestion succeeded. Handle that inside your own queue:

• Retry failed jobs with exponential backoff and a capped attempt count (e.g., 5 attempts over 30 minutes).
• Move anything that exhausts retries into a Dead Letter Queue.
• Give engineers a way to inspect DLQ payloads, see why processing failed, and replay events once the underlying bug is fixed.

Code Blueprint: Node.js Ingestion Engine

server.ts — fast HTTP ingestion:

import express, { Request, Response } from 'express';
import crypto from 'crypto';
import { Queue } from 'bullmq';

const app = express();
const SIGNING_KEY = process.env.WEBHOOK_SIGNING_KEY || '';

const webhookQueue = new Queue('blockchain-webhooks', {
  connection: { host: 'localhost', port: 6379 }
});

// Preserve the raw body buffer — required for an accurate HMAC comparison
app.use(express.json({
  verify: (req: any, _res, buf) => { req.rawBody = buf; }
}));

function isValidSignature(req: any): boolean {
  const signature = req.headers['x-alchemy-signature'] as string; // header name varies by provider
  if (!signature) return false;

  const hmac = crypto.createHmac('sha256', SIGNING_KEY);
  hmac.update(req.rawBody);
  const digestBuf = Buffer.from(hmac.digest('hex'), 'hex');
  const sigBuf = Buffer.from(signature, 'hex');

  // Guard the length check BEFORE calling timingSafeEqual — mismatched
  // buffer lengths throw, they don't fail gracefully.
  if (digestBuf.length !== sigBuf.length) return false;

  return crypto.timingSafeEqual(digestBuf, sigBuf);
}

app.post('/webhooks/ingest', async (req: Request, res: Response) => {
  const start = Date.now();

  if (!isValidSignature(req)) {
    return res.status(401).json({ error: 'Unauthorized signature' });
  }

  const payload = req.body;
  await webhookQueue.add('process-event', payload, {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: true,
  });

  console.log(`[INGEST] Enqueued ${payload.id} in ${Date.now() - start}ms`);
  return res.status(200).json({ status: 'queued', id: payload.id });
});

app.listen(3000, () => console.log('Ingestion engine listening on :3000'));

worker.ts — decoupled processing and reorg handling:

import { Worker, Job } from 'bullmq';
import Redis from 'ioredis';

const redis = new Redis({ host: 'localhost', port: 6379 });

const webhookWorker = new Worker(
  'blockchain-webhooks',
  async (job: Job) => {
    const activities = job.data.event?.activity || [];

    for (const activity of activities) {
      const log = activity.log;
      if (!log) continue;

      const { network } = job.data.event;
      const eventId = `${network}:${log.transactionHash}:${log.logIndex}`;
      const acquired = await redis.set(`lock:${eventId}`, 'PROCESSED', 'NX', 'EX', 86400);

      if (!acquired && !log.removed) {
        console.log(`[WORKER] Skipping duplicate: ${eventId}`);
        continue;
      }

      if (log.removed) {
        console.warn(`[REORG] Rolling back: ${eventId}`);
        await handleReorgRollback(network, log.transactionHash, log.logIndex);
        continue;
      }

      await updateDatabaseState(activity);
    }
  },
  { connection: { host: 'localhost', port: 6379 }, concurrency: 10 }
);

async function handleReorgRollback(network: string, txHash: string, logIndex: string) {
  // Revert or flag the record matching (network, txHash, logIndex) as REORGD_OUT
}

async function updateDatabaseState(activity: any) {
  // Apply business logic / persist state
}

webhookWorker.on('failed', (job, err) => {
  console.error(`[WORKER ERROR] Job ${job?.id} failed: ${err.message}`);
});

Monitoring Checklist

Track these to hold a high uptime SLA against webhook-driven ingestion:

• Ingestion latency (p95/p99) — keep this under ~50ms. Spikes usually mean something synchronous slipped into the HTTP handler.
• Queue depth — if it grows continuously after each block, scale out worker concurrency before it becomes a backlog problem.
• Dead Letter Queue count — alert on any growth; DLQ entries are silent data-loss risk until someone looks at them.
• Reorg frequency — track how often you see removed: true (or your provider's reorg-redelivery signal). A sudden spike can indicate RPC node instability upstream, not just normal chain behavior.
• Webhook health status — most providers will pause or disable a webhook definition after sustained failures; alert on that state directly rather than discovering it from a gap in your data.

Summary Checklist

• Decouple ingestion from processing — a lightweight receiver that buffers into a queue before returning 200 OK.
• Verify signatures with a constant-time comparison, and check buffer lengths before calling timingSafeEqual.
• Build idempotency keys from network + transactionHash + logIndex.
• Size your reorg-safety window to the chain's actual finality model — safe/finalized on Ethereum, not a leftover proof-of-work heuristic.
• Run your own client-side retry queue and DLQ; don't depend solely on the provider re-sending the original HTTP request.
• Pick a provider based on what it actually guarantees today — ordering, at-least-once vs. exactly-once, and chain coverage differ meaningfully between Alchemy, QuickNode, and Moralis, and Infura simply isn't in the same product category.

Sources

• Alchemy — Webhooks Quickstart & Delivery Behavior
• QuickNode — Streams Webhook Destination & Reorg Handling
• Moralis — Streams API Overview
• MetaMask/Infura — Services Documentation
• ethereum.org — Single Slot Finality
• Circle Research — Confirmation Rules for Ethereum