The Internal Webhooks Anti-Pattern: Why Service-to-Service HTTP Callbacks Don't Scale Microservices were supposed to make systems easier to change independently.

The Internal Webhooks Anti-Pattern: Why Service-to-Service HTTP Callbacks Don't Scale

Microservices were supposed to make systems easier to change independently. In practice, the thing that most often breaks that promise isn't the services themselves — it's how they talk to each other.

A pattern that shows up constantly in growing engineering orgs is the internal webhook: Service A fires an HTTP POST at a hardcoded URL owned by Service B whenever something happens. It's an easy trap to fall into, because most developers already understand webhooks intimately — they've built integrations with Stripe, GitHub, or Shopify, all of which use exactly this model to notify external systems of events.

The reasoning feels obvious: if it's good enough for Stripe to tell my app about a payment, it's good enough for my Inventory Service to tell my Shipping Service about a shipment.

It isn't — and the reason is architectural, not stylistic. Webhooks were designed to solve a specific problem: getting an event across a trust boundary, from a system you don't control to one you do, over the open internet. Internal service communication has almost the opposite set of constraints. Applying the same tool to both jobs is where the trouble starts.

Why Webhooks Are So Tempting

Webhooks earned their popularity honestly. For external, cross-organization integrations, they are:

Language and framework agnostic — any stack can send and receive an HTTP POST.
Firewall-friendly — they run over standard ports (80/443), so they route cleanly across the public internet without special infrastructure.
Conceptually simple — give me a URL, and I'll POST JSON to it when something happens.

That familiarity is exactly why teams building event-driven systems default to HTTP callbacks even inside their own network. But inside a private network — a VPC, a Kubernetes cluster, a data center — the constraints that made webhooks a good idea (crossing an untrusted boundary, reaching arbitrary tech stacks) mostly don't apply. What's left is HTTP's downsides without HTTP's upside.

Where Internal Webhooks Break Down

1. Point-to-point coupling

Point-to-point coupling means Service A now has to know Service B's address, keep a config of which downstream services care about which events, and loop through that list on every publish. Add a new consumer — an Analytics Service that wants "Order Created" events, say — and you have to go modify the producer's code or config. That's precisely the coupling microservices were meant to remove.

2. HTTP is a synchronous transport

HTTP is fundamentally request/response. Even if the receiving service returns a 202 Accepted immediately and processes the payload asynchronously, the delivery itself is still a blocking call from the sender's point of view: open a TCP connection, complete a TLS handshake, send the payload, wait on the socket. If the receiver is slow or unreachable, the sender's thread (or connection pool) is tied up. Under load, that's a direct path to resource exhaustion and cascading failures — you're using a synchronous transport to simulate an asynchronous system.

3. No persistence, retries, or replay by default

Plain HTTP webhooks have no built-in durability. If Service B is down for a 30-second deploy while Service A tries to deliver, that event is gone unless somebody built retry logic for it. Teams that go down this road often end up writing their own "pending webhook" tables, backoff schedules, and sweep jobs — in effect, reinventing a message broker, badly, inside their business logic.

Note that this problem is solvable for webhooks specifically — companies like Svix, Hookdeck, and Convoy exist because reliable webhook delivery (retries, dead-letter queues, replay) is a real, hard problem that's been productized. But that infrastructure is built for the edge case webhooks were designed for: delivering events to external endpoints you don't control. It's not a reason to route internal traffic the same way — for internal traffic, a message broker already gives you persistence and replay natively, without needing a delivery subsystem bolted on top of HTTP.

4. Weak backpressure

Webhooks are a push model: the producer decides the rate, and the consumer has to keep up or fall over. If an edge service scales from 5 pods to 100 under a traffic spike and starts firing webhooks at a downstream service that's capped at 3 pods by a database bottleneck, that downstream service is going to have a bad day. Message brokers handle this differently — whether it's a pull model like Kafka, where consumers read at their own pace, or a push model with consumer-controlled flow control like RabbitMQ's prefetch limits, the consumer — not the producer — effectively sets the rate. A traffic spike grows the queue depth, not the failure rate, buying time for autoscaling to catch up.

5. Unnecessary security surface

Internal webhooks routed through the same gateway infrastructure built for external traffic can accidentally end up exposed to the internet through a routing misconfiguration. Even kept strictly internal, every hop typically needs its own HMAC signature, JWT, or mTLS setup to be trustworthy — overhead that a network you already control shouldn't need on every single call.

What to Use Instead

The fix isn't a specific product — it's swapping point-to-point HTTP for a message broker or event streaming platform: Apache Kafka, RabbitMQ, Amazon SQS/EventBridge, Google Pub/Sub, NATS, or Solace PubSub+, among others. Rather than Service A calling Service B directly, Service A publishes an event once, and any number of interested consumers subscribe to it.

This solves the five problems above directly:

Decoupling — producers don't know or care who's listening. Adding a new consumer requires zero changes to the publisher.
Durability — if a consumer is down, the broker holds the message (in a queue or a durable log) until it comes back.
Backpressure — consumers pull or throttle at their own pace; the broker absorbs the spike instead of the consumer crashing.
Replay — log-based brokers like Kafka keep an ordered, immutable history, so you can reset a consumer group's offset and reprocess a day's worth of events after fixing a bug.

A quick note on "event mesh"

You'll often see the term event mesh used for this pattern — an architecture layer of interconnected event brokers that routes events to interested consumers regardless of where they're deployed. It's worth knowing where that term comes from: it was coined and popularized by Solace, introduced publicly around the 2018 Gartner Symposium as a way to describe Solace's own PubSub+ platform, and it's since been picked up more broadly in the industry. It's a legitimate architectural concept, but it isn't a neutral, vendor-independent standard the way "message broker" or "pub/sub" is — worth keeping in mind if you see it in a vendor's marketing.

It's also easy to confuse with a service mesh (Istio, Linkerd, Envoy), which is a different layer entirely: service meshes manage synchronous, request/response traffic between services (routing, retries, mTLS, observability), while event brokers and event meshes handle asynchronous, event-based traffic. The two are complementary, not competing — plenty of organizations run Kafka and Istio side by side, each handling a different half of the communication story.

Don't forget synchronous internal calls

Not everything internal should become an event, either. When Service A genuinely needs an immediate answer from Service B — "is this SKU in stock right now?" — that's a synchronous request/response call, and plenty of teams use gRPC for this rather than REST-over-HTTP, since it gives you strongly typed contracts (via Protocol Buffers), HTTP/2 multiplexing, and lower serialization overhead. The point isn't "always use a message broker" — it's "match the transport to whether the interaction is actually a fire-and-forget event or a request that needs an answer." Point-to-point webhooks are a poor fit for either case internally.

Standardizing the events themselves

If you do move to a broker, it's worth standardizing the shape of your events too, rather than letting every service invent its own JSON schema:

CloudEvents is a CNCF specification (it graduated to a full CNCF project in January 2024) for a common event envelope — fields like source, type, and timestamp — so tooling and consumers don't have to special-case every producer's format.
AsyncAPI is the closest thing the async world has to OpenAPI/Swagger: a protocol-agnostic spec (currently on version 3.x) for documenting channels, messages, and schemas across Kafka, AMQP, MQTT, WebSockets, and more.

Neither is mandatory, but both save you from rediscovering the same documentation and interoperability problems a few years into your event-driven architecture.

Where Webhooks Still Belong

None of this means webhooks are a bad idea — they're just the wrong tool for internal traffic. At the edge of your architecture, where you're talking to systems you don't control — third-party SaaS vendors, customer-owned endpoints, mobile clients — a shared internal broker isn't an option, and webhooks remain the right call. That's still exactly what Stripe, GitHub, and Shopify use them for.

The common hybrid pattern looks like this:

At the edge, accept inbound webhooks from external providers, and deliver outbound webhooks to customers' endpoints, over HTTP — with the retry, signature-verification, and rate-limiting logic that reliable webhook delivery requires.
Immediately translate those edge events into internal events and publish them onto your message broker for everything downstream.

That boundary is the whole point: HTTP webhooks for interoperability at the edge, a broker for reliability and decoupling at the core. As one comparison of webhook infrastructure tools put it plainly: message queues handle internal communication, webhooks handle external communication — they're complementary, not competing choices.

The Short Version

Internal webhooks feel free because the code is simple to write on day one. The cost shows up later, as point-to-point spaghetti, custom-built (and usually incomplete) retry logic, and services that fall over the first time traffic spikes. A message broker or event streaming platform solves the durability, backpressure, and decoupling problems natively — leaving webhooks to do the job they were actually designed for: getting events across the boundary between your systems and everyone else's.

Sources consulted: CloudEvents / CNCF, AsyncAPI Initiative, Solace on event mesh, and current (2026) webhook-infrastructure documentation from Hook0/Svix/Hookdeck comparing internal message queues to external webhook delivery.