InstaWebhook
November 1, 2026By InstaWebhook TeamWebhook Reliability

Building a Resilient Webhook Ingress Proxy in Go: Handling Slow Consumers Without Thread Starvation

Learn how to build a high-performance webhook ingress proxy in Go. Master goroutine management, non-blocking channels, and connection pooling to prevent thread starvation during slow downstream

Building a Resilient Webhook Ingress Proxy in Go: Handling Slow Consumers Without Thread Starvation

Building a Resilient Webhook Ingress Proxy in Go: Handling Slow Consumers Without Thread Starvation

Payment gateways, CI/CD systems, e-commerce platforms and chat tools all push events to your servers as HTTP POST requests. Accepting those requests is cheap. The hard part is what happens when something behind your endpoint slows down: a database under lock contention, a downstream microservice in a garbage-collection spiral, or a third-party API having a bad afternoon.

A naive receiver that does the downstream work inside the request path, or spawns a goroutine for every event, degrades in a predictable way. Memory grows, sockets pile up, latency climbs, and eventually the process stops accepting new webhooks.

A note on terminology: Go multiplexes goroutines over a small set of OS threads, so the failure here is not classic thread-pool starvation. It is exhaustion of memory, sockets and file descriptors, plus unbounded queueing delay. This article uses “thread starvation” loosely, as most people searching for this problem do. The fix is the same: decouple ingestion from delivery and put hard limits on everything in between.

By the end you will have a small, complete Go service that verifies signatures, acknowledges quickly, buffers into a bounded queue, delivers through a fixed worker pool with a tuned HTTP transport, sheds load explicitly, and shuts down without losing queued work. The code was compiled with Go 1.22 and smoke-tested locally.

1. The Anatomy of the Webhook Ingress Problem

Every webhook provider gives you a deadline and a retry policy, and they differ a lot:

ProviderResponse deadlineAutomatic retries
GitHub2xx within 10 secondsNone; failed deliveries must be redelivered manually or via the API
Shopify2xx within 5 secondsUp to 8 retries over about 4 hours; persistent failures can remove the subscription
Slack Events API2xx within 3 secondsUp to 3 retries with exponential backoff
StripeNo specific timeout published; return 2xx before any complex logicLive mode: up to 3 days with exponential backoff. Sandbox: 3 retries over a few hours

Three practical conclusions follow:

  1. There is no universal “200–500 ms” rule. The tightest documented deadline among these providers is Slack’s 3 seconds. Acknowledge as fast as you can, but design for the strictest sender you integrate with.
  2. Slow responses cause duplicates. If your handler misses a deadline, the provider counts the delivery as failed even if you eventually process it. Providers that retry will send the event again.
  3. Dropping an event is not equally recoverable everywhere. If you shed load with a 503, Stripe, Shopify and Slack will retry. GitHub will not, so you must redeliver missed events yourself once you recover.

The architecture that satisfies all of this looks like this:

Code example
[ Third-Party Senders ]
         │  HTTP POST (webhooks at scale)
         ▼
┌─────────────────────────────────────────┐
│         Go Webhook Ingress Proxy        │
│  - Bounded body read                    │
│  - HMAC signature verification          │
│  - Fast validation, then 202 Accepted   │
└─────────────────────────────────────────┘
         │ non-blocking enqueue (full → 503 + Retry-After)
         ▼
[ Bounded in-memory queue ]  ──(optional)──►  [ Durable broker ]
         │
         ▼
[ Fixed worker pool ] ──► tuned HTTP transport ──► [ Downstream service ]

The Naive Anti-Pattern: Unbounded Goroutines

A common trap is spawning a goroutine per request to handle downstream delivery:

Code example
// ANTI-PATTERN: DO NOT USE IN PRODUCTION
func WebhookHandler(w http.ResponseWriter, r *http.Request) {
	payload := parseBody(r)

	go func() {
		if err := sendToSlowDownstream(payload); err != nil { // can block for a long time
			// handle error
		}
	}()

	w.WriteHeader(http.StatusOK)
}

Goroutines are cheap, not free. A goroutine starts with a 2 KB stack, and since Go 1.19 the runtime adapts the initial stack size to the program’s observed average stack usage. That is why the old “2–8 KB” rule of thumb is only approximate.

The real cost is everything each goroutine holds. Little’s Law says average concurrency equals arrival rate multiplied by time in the system. At 15,000 events per second:

  • With a 50 ms downstream, you have about 750 deliveries in flight.
  • When the downstream slows to 5 seconds, you have about 75,000.

The stacks alone are roughly 150 MB at 2 KB each. On top of that, each goroutine pins its payload and typically an open connection. The failure modes:

  1. Memory growth. Stacks, payload copies and per-connection buffers accumulate with no upper bound.
  2. GC pressure. The garbage collector has to scan every live goroutine’s stack, so tens of thousands of parked goroutines make each cycle more expensive.
  3. Socket and file-descriptor exhaustion. Each in-flight call to a slow downstream holds a TCP connection, and every open connection consumes a file descriptor, which the OS caps per process (ulimit -n on Linux). Once you hit the cap, new inbound connections fail too.
  4. Silent data loss. The handler returned 200 before the event was durable. If the process dies, those events are gone and the provider believes they were delivered.

Go’s HTTP server already runs each connection in its own goroutine, so the handler is exactly where you must not block on downstream work.

2. Architecture Blueprint: Decoupling Ingress and Egress

A resilient proxy splits an event’s lifecycle into three phases:

  1. Ingress and validation. Read a size-limited body, verify the HMAC signature, check that the payload is well-formed, and return 202 Accepted.
  2. Buffer. Enqueue into a bounded structure. A buffered Go channel is the simplest option, but it lives in memory only. A channel cannot be “backed by persistent storage”; durability requires writing to something external (see section 11).
  3. Dispatch. A fixed pool of workers pulls from the buffer and delivers through a shared, tuned HTTP client, with timeouts, bounded retries and (optionally) a circuit breaker.

Every stage has a hard limit: body size, queue length, worker count, connection count, request timeout. When the downstream is slow, pressure propagates back to the one place designed to absorb it, the queue, and then to an explicit “no” at the front door.

3. Sizing the Pool: Little’s Law Again

Before writing code, decide the numbers:

  • Delivery capacity ≈ workers ÷ average delivery latency. With 100 workers and a 100 ms downstream, you can deliver about 1,000 events per second. If latency rises to 1 second, capacity falls to about 100 per second.
  • Queue runway ≈ queue size ÷ (arrival rate − delivery rate). If events arrive at 3,000 per second against a capacity of 1,000, a 5,000-slot queue fills in about 2.5 seconds.

Two things follow. A queue is a shock absorber for bursts, not a fix for sustained overload. And the queue size should be chosen from how much delay and memory you can tolerate, not set as large as possible. A very large queue only converts overload into long delays and eventually an out-of-memory crash.

4. Deep Dive 1: A Bounded Worker Pool

You will often see a two-level pattern in tutorials: a dispatcher, plus a chan chan Job that workers register their private channels into. It is unnecessary in Go. Goroutines blocked on receiving from a single channel are queued by the runtime, and each job goes to exactly one waiting worker. A single shared channel is simpler and easier to shut down correctly:

Code example
func (p *WorkerPool) Start(workers int) {
	for i := 0; i < workers; i++ {
		p.wg.Add(1)
		go func() {
			defer p.wg.Done()
			for job := range p.jobs { // exits when the channel is closed and drained
				p.deliver(job)
			}
		}()
	}
}

The worker count is the concurrency limit toward the downstream. Whatever the downstream does, you never have more than workers deliveries in flight.

5. Deep Dive 2: Non-Blocking Enqueue and Backpressure

When arrivals outpace delivery, the handler must never block waiting for space. A select with a default case makes the enqueue non-blocking:

Code example
func (p *WorkerPool) Enqueue(job Job) bool {
	select {
	case p.jobs <- job:
		return true
	default:
		return false // queue full: shed load
	}
}

When Enqueue returns false, the handler answers 503 Service Unavailable (or 429 Too Many Requests) with a Retry-After header. Providers are not obliged to honor Retry-After, but providers that retry will use their own backoff schedule and try again. Choose your overflow policy deliberately:

  • Shed load. Return 503 and rely on sender retries. This works for Stripe, Shopify and Slack, but for GitHub you need your own redelivery process.
  • Spill to a durable queue. Write overflow to Redis Streams, RabbitMQ or Kafka instead of rejecting.
  • Drop selectively. For low-value events (heartbeats, noisy notifications), dropping may be acceptable. Make that an explicit per-event-type decision.

6. Deep Dive 3: Connection Pooling in Go’s HTTP Client

Go’s http.DefaultTransport is reasonable for general use, but it is tuned for modest concurrency: MaxIdleConns is 100, IdleConnTimeout is 90 seconds, and DefaultMaxIdleConnsPerHost is only 2. A webhook proxy that sends many concurrent requests to a single downstream host will keep at most 2 idle connections for it. Every other connection is closed after use and re-dialed, with a new TCP handshake and, for HTTPS, a new TLS handshake. Under load, that churn also produces many sockets in TIME_WAIT.

Two clarifications on common advice:

  • Creating a new http.Client is not itself the problem. A client with a nil Transport shares DefaultTransport. The problem is creating a new http.Transport per request, which defeats connection reuse entirely. Create one transport and share it.
  • To reuse a connection, the response body must be read to EOF and closed. Skipping the drain makes Go close the connection instead of returning it to the pool.

A transport tuned for a proxy delivering to one primary downstream:

Code example
transport := &http.Transport{
	Proxy: http.ProxyFromEnvironment,
	DialContext: (&net.Dialer{
		Timeout:   5 * time.Second,
		KeepAlive: 30 * time.Second,
	}).DialContext,
	ForceAttemptHTTP2:     true,
	MaxConnsPerHost:       100, // hard cap on connections to a single downstream
	MaxIdleConns:          100,
	MaxIdleConnsPerHost:   100, // the default is 2
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   5 * time.Second,
	ResponseHeaderTimeout: 8 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

Why these values matter for slow consumers:

  • MaxConnsPerHost and MaxIdleConnsPerHost should track the worker count. There is no point keeping 500 idle connections to a host when only 100 workers can use them.
  • Timeouts bound how long a worker can be held hostage. Without Client.Timeout and ResponseHeaderTimeout, a single hung downstream can occupy every worker indefinitely.
  • ForceAttemptHTTP2 is needed here. Go disables automatic HTTP/2 on a custom transport that sets DialContext or a TLS config unless you ask for it. With HTTP/2, requests are multiplexed over few connections, so the per-host idle settings matter less for HTTPS targets.

7. A Complete, Runnable Example

The program below combines everything: size-limited reads, HMAC verification, a non-blocking bounded queue, a worker pool with bounded retries and jitter, a tuned transport, and a graceful shutdown that drains the queue. It requires Go 1.22 or newer (method-based ServeMux patterns, math/rand/v2).

Code example
package main

import (
	"bytes"
	"context"
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"log"
	"math/rand/v2"
	"net"
	"net/http"
	"os"
	"os/signal"
	"strings"
	"sync"
	"syscall"
	"time"
)

const (
	maxWorkers   = 100
	queueSize    = 5000
	maxBodyBytes = 1 << 20 // 1 MiB
	maxAttempts  = 4
)

// Job is one accepted webhook waiting for delivery.
type Job struct {
	EventID string
	Body    []byte // the original bytes, forwarded untouched
}

// WorkerPool delivers jobs with a fixed number of goroutines.
type WorkerPool struct {
	ctx    context.Context // cancelled to abort in-flight work on a hard shutdown
	jobs   chan Job
	wg     sync.WaitGroup
	client *http.Client
	target string
}

func NewWorkerPool(ctx context.Context, client *http.Client, target string, queueSize int) *WorkerPool {
	return &WorkerPool{
		ctx:    ctx,
		jobs:   make(chan Job, queueSize),
		client: client,
		target: target,
	}
}

func (p *WorkerPool) Start(workers int) {
	for i := 0; i < workers; i++ {
		p.wg.Add(1)
		go func() {
			defer p.wg.Done()
			for job := range p.jobs { // exits when the channel is closed and drained
				p.deliver(job)
			}
		}()
	}
}

// Enqueue never blocks: it returns false when the queue is full.
func (p *WorkerPool) Enqueue(job Job) bool {
	select {
	case p.jobs <- job:
		return true
	default:
		return false
	}
}

// Drain stops accepting work and waits for queued jobs to finish or ctx to expire.
// Call it only after the HTTP server has stopped, so nothing can send on a closed channel.
func (p *WorkerPool) Drain(ctx context.Context) error {
	close(p.jobs)
	done := make(chan struct{})
	go func() { p.wg.Wait(); close(done) }()
	select {
	case <-done:
		return nil
	case <-ctx.Done():
		return ctx.Err()
	}
}

func (p *WorkerPool) deliver(job Job) {
	backoff := 500 * time.Millisecond
	for attempt := 1; attempt <= maxAttempts; attempt++ {
		if p.ctx.Err() != nil {
			log.Printf("event %s abandoned: shutting down", job.EventID)
			return
		}
		retry, err := p.post(job)
		if err == nil {
			return
		}
		if !retry || attempt == maxAttempts {
			// In production: write to a dead-letter queue instead of only logging.
			log.Printf("event %s failed after %d attempt(s): %v", job.EventID, attempt, err)
			return
		}
		sleep := backoff + rand.N(backoff/2) // jitter avoids synchronized retries
		select {
		case <-time.After(sleep):
		case <-p.ctx.Done():
			return
		}
		backoff *= 2
	}
}

func (p *WorkerPool) post(job Job) (retry bool, err error) {
	req, err := http.NewRequestWithContext(p.ctx, http.MethodPost, p.target, bytes.NewReader(job.Body))
	if err != nil {
		return false, err
	}
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("X-Event-ID", job.EventID)

	resp, err := p.client.Do(req)
	if err != nil {
		return true, err // timeouts and connection errors are retryable
	}
	defer resp.Body.Close()
	// Read the body to EOF (bounded) so the connection can be reused.
	_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10))

	switch {
	case resp.StatusCode >= 200 && resp.StatusCode < 300:
		return false, nil
	case resp.StatusCode == http.StatusTooManyRequests,
		resp.StatusCode == http.StatusRequestTimeout,
		resp.StatusCode >= 500:
		return true, fmt.Errorf("downstream returned %d", resp.StatusCode)
	default:
		return false, fmt.Errorf("downstream rejected event with %d", resp.StatusCode)
	}
}

func validSignature(secret, body []byte, header string) bool {
	sig, ok := strings.CutPrefix(header, "sha256=")
	if !ok {
		return false
	}
	got, err := hex.DecodeString(sig)
	if err != nil {
		return false
	}
	mac := hmac.New(sha256.New, secret)
	mac.Write(body)
	return hmac.Equal(got, mac.Sum(nil)) // constant-time comparison
}

func webhookHandler(pool *WorkerPool, secret []byte) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
		body, err := io.ReadAll(r.Body)
		if err != nil {
			var tooBig *http.MaxBytesError
			if errors.As(err, &tooBig) {
				http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
				return
			}
			http.Error(w, "bad request", http.StatusBadRequest)
			return
		}

		// Authenticate before doing anything else with the payload.
		if !validSignature(secret, body, r.Header.Get("X-Hub-Signature-256")) {
			http.Error(w, "invalid signature", http.StatusUnauthorized)
			return
		}

		var evt struct {
			EventID string `json:"event_id"`
		}
		if err := json.Unmarshal(body, &evt); err != nil || evt.EventID == "" {
			http.Error(w, "invalid payload", http.StatusBadRequest)
			return
		}

		if !pool.Enqueue(Job{EventID: evt.EventID, Body: body}) {
			w.Header().Set("Retry-After", "5")
			http.Error(w, "overloaded, retry later", http.StatusServiceUnavailable)
			return
		}

		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(http.StatusAccepted)
		_, _ = w.Write([]byte(`{"status":"accepted"}`))
	}
}

func main() {
	secret := os.Getenv("WEBHOOK_SECRET")
	target := os.Getenv("DOWNSTREAM_URL")
	if secret == "" || target == "" {
		log.Fatal("WEBHOOK_SECRET and DOWNSTREAM_URL must be set")
	}

	transport := &http.Transport{
		Proxy: http.ProxyFromEnvironment,
		DialContext: (&net.Dialer{
			Timeout:   5 * time.Second,
			KeepAlive: 30 * time.Second,
		}).DialContext,
		ForceAttemptHTTP2:     true,
		MaxConnsPerHost:       maxWorkers, // hard cap on connections to one downstream
		MaxIdleConns:          maxWorkers,
		MaxIdleConnsPerHost:   maxWorkers, // the default is only 2
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   5 * time.Second,
		ResponseHeaderTimeout: 8 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
	}
	client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

	workCtx, cancelWork := context.WithCancel(context.Background())
	defer cancelWork()
	pool := NewWorkerPool(workCtx, client, target, queueSize)
	pool.Start(maxWorkers)

	mux := http.NewServeMux()
	mux.HandleFunc("POST /webhook", webhookHandler(pool, []byte(secret)))
	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:       10 * time.Second,
		WriteTimeout:      10 * time.Second,
		IdleTimeout:       60 * time.Second,
	}

	sigCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	serverErr := make(chan error, 1)
	go func() { serverErr <- srv.ListenAndServe() }()
	log.Println("webhook ingress proxy listening on :8080")

	select {
	case err := <-serverErr:
		log.Fatalf("HTTP server error: %v", err)
	case <-sigCtx.Done():
		log.Println("shutdown signal received")
	}
	stop()

	shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	// 1. Stop accepting requests and wait for in-flight handlers to return.
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("HTTP shutdown: %v", err)
	}
	// 2. Let workers finish what is already queued, up to the deadline.
	if err := pool.Drain(shutdownCtx); err != nil {
		log.Printf("drain timed out with %d event(s) still queued", len(pool.jobs))
	}
	cancelWork() // abort anything still in flight
	log.Println("stopped")
}

Design notes on the code

  • http.MaxBytesReader instead of io.LimitReader. LimitReader silently truncates at the limit, so an oversized payload turns into confusing invalid JSON. MaxBytesReader returns a typed error, so the handler can answer 413.
  • Signature verification uses the raw bytes. Re-serializing JSON before checking can change the bytes and break the signature. The check uses hmac.Equal for constant-time comparison, and runs before JSON parsing and before the queue, so unauthenticated traffic never consumes queue capacity. The proxy also forwards the original bytes untouched.
  • Shutdown ordering matters. After Shutdown is called, ListenAndServe returns ErrServerClosed immediately, and the Go documentation warns that the program must not exit before Shutdown itself returns. A common bug is to call Shutdown in a goroutine and let main return as soon as ListenAndServe does. Here main waits for Shutdown, then closes the queue (safe, because no handler can still be enqueueing), then waits for workers.
  • Server timeouts are set. ReadHeaderTimeout, ReadTimeout, WriteTimeout and IdleTimeout protect the ingress side from slow senders, the mirror image of the slow-consumer problem.
  • Method routing. Since Go 1.22, ServeMux accepts patterns like "POST /webhook" and automatically replies 405 Method Not Allowed with an Allow header for other methods.

Try it against a slow consumer

Start any stub downstream that sleeps before replying, point DOWNSTREAM_URL at it, and send a signed request:

Code example
BODY='{"event_id":"evt_1","data":{}}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" | sed 's/^.* //')
curl -i -X POST localhost:8080/webhook \
  -H "X-Hub-Signature-256: sha256=$SIG" \
  -d "$BODY"

You should see 202 Accepted within milliseconds regardless of how slow the stub is. A request with a bad signature returns 401, and a GET returns 405. To see load shedding, make the stub sleep for several seconds and fire more than maxWorkers + queueSize events quickly. The surplus gets 503 with Retry-After, while the process’s goroutine count stays flat.

8. Observability: Know When You Are Absorbing Pain

Building the proxy is half the job. These signals tell you whether a slow consumer is hurting you:

  1. Ingestion rate and response codes. Track 202, 401, 413 and 503 separately. A rising 503 rate means the queue is overflowing.
  2. Queue depth (len(jobs)) and oldest-event age. Depth near capacity means your workers cannot keep up. Age is often more useful than depth because it is the delay your users actually experience.
  3. In-flight deliveries against the worker count. If every worker is permanently busy, you are at capacity.
  4. Downstream latency and error rate (p95/p99 of outgoing calls), plus retry counts and dead-lettered events.
  5. Goroutine count via runtime.NumGoroutine(). In this design it should be roughly constant: workers plus per-connection server goroutines. A steady climb indicates a leak. Go 1.27 (released August 2026) also makes the goroutineleak profile in runtime/pprof generally available, after it was an experiment in Go 1.26.

9. Idempotency: Assume Duplicates

Retries mean duplicates, so your downstream must tolerate seeing an event more than once. Stripe’s documentation says events can be delivered more than once and are not guaranteed to arrive in order. Slack documents retry headers (X-Slack-Retry-Num, X-Slack-Retry-Reason). GitHub reuses the same X-GitHub-Delivery GUID when you request a redelivery.

Practical rules:

  • Deduplicate on the provider’s event ID, with a TTL longer than the provider’s retry window.
  • Authenticate first, then deduplicate. GitHub’s delivery header is not itself a signed event ID, so don’t trust identifiers until the signature has been verified.
  • Don’t assume ordering. Design handlers so that a late or repeated event cannot overwrite newer state.
  • The proxy above forwards an X-Event-ID header so the downstream can implement idempotency cheaply.

10. Security Notes

  • Use the right header. GitHub’s SHA-256 signature is in X-Hub-Signature-256. Don’t rely on the older SHA-1 X-Hub-Signature.
  • Reject stale requests where the provider supports it. Some providers include a timestamp in the signed material, which lets you reject replays. Check your provider’s documentation.
  • Consider IP allow-lists as an extra layer. GitHub publishes its delivery IP ranges through its /meta endpoint and notes that they change occasionally, so refresh the list periodically. Treat it as defense in depth, not a replacement for signatures.
  • Terminate TLS properly and require HTTPS. Both GitHub and Stripe expect HTTPS endpoints.

11. Advanced Enhancements

Durability. An in-memory queue loses everything on a crash or restart, and a 202 only promises that the event reached memory. For high-stakes events such as payments, write to a durable broker (Redis Streams, RabbitMQ, Kafka) before acknowledging, and let the workers consume from there. A common compromise is the in-memory queue as a fast path with a durable spillover when it fills.

Per-destination isolation. One shared pool means one slow destination can occupy every worker and starve healthy ones (head-of-line blocking). If you fan out to several downstreams, give each its own queue and concurrency limit.

Circuit breaking. When a downstream is failing, retrying only adds load. The sony/gobreaker library (use the generic v2 module) wraps calls in a breaker that fails fast once failures cross a threshold:

Code example
import "github.com/sony/gobreaker/v2"

cb := gobreaker.NewCircuitBreaker[int](gobreaker.Settings{
	Name:    "downstream",
	Timeout: 30 * time.Second, // how long the breaker stays open before probing
	ReadyToTrip: func(c gobreaker.Counts) bool {
		return c.Requests >= 20 &&
			float64(c.TotalFailures)/float64(c.Requests) >= 0.5
	},
})

status, err := cb.Execute(func() (int, error) {
	// perform the POST and return the status code
})
if errors.Is(err, gobreaker.ErrOpenState) {
	// don't burn a retry attempt; requeue with a delay or pause consumption
}

Decide what workers do while the breaker is open. Failing fast without pausing consumption simply moves events from the queue into your retry or dead-letter path faster.

Rate limiting. Cap per-sender or per-tenant ingress and per-destination egress so one noisy source cannot consume the shared queue.

12. Conclusion

A webhook proxy is resilient when every resource is bounded and every limit has an explicit failure behavior:

  • Bounded body size protects memory at the edge.
  • A fixed worker pool caps concurrency toward the downstream, regardless of how slow it gets.
  • A bounded, non-blocking queue absorbs bursts and turns sustained overload into a fast, explicit 503.
  • A shared, tuned http.Transport with timeouts, connection caps and body draining prevents connection churn and hung workers.
  • Ordered graceful shutdown drains accepted work instead of dropping it.
  • Idempotent consumers and durable queues cover duplicates and crashes.

None of this makes failures impossible. It makes them bounded, visible and recoverable, which is what you want when someone else’s system is slow and your ingress tier still has to stay up.

Sources and Further Reading

Building a Resilient Webhook Ingress Proxy in Go | InstaWebhook