Webhook Reliability in SaaS: Guaranteed Delivery, Retry Strategies, and Dead-Letter Queues

Webhooks are the backbone of real-time integration in modern SaaS. Payment confirmations, CI/CD pipeline events, CRM record updates, shipping status changes — nearly every meaningful cross-system interaction travels as an HTTP POST to a URL your customer registered. Done well, webhooks feel instant and reliable. Done poorly, they silently drop events, deliver duplicates, or fail in ways that are nearly impossible for your customer to debug after the fact.

The gap between the two is not complicated engineering. It’s a small set of deliberate patterns applied consistently. This guide covers the ones that matter most: delivery guarantees, exponential retry logic, dead-letter queues, idempotent consumers, and signature verification. If you’re building or scaling a SaaS product, getting these right is the difference between an integration your customers trust and one they route around.

Why Webhook Delivery Is Harder Than It Looks

A webhook is deceptively simple: receive an event, POST it to a URL, move on. The failure modes are where the complexity lives.

The receiving endpoint can be unavailable. Your customer’s server might be deploying, rebooting, or hit by a traffic spike. If you treat a 5xx response as permanent failure and discard the event, you’ve lost data your customer needed.

The receiving endpoint can be slow. A response that takes 30 seconds is not a success — it’s a hung connection that ties up your delivery worker and times out before the acknowledgement arrives.

Events can be delivered more than once. Network retries, load-balancer timeouts, and idempotency bugs on the sender side mean a customer should expect duplicate deliveries and design for them.

Events can arrive out of order. If you’re retrying an event from three minutes ago alongside live events, the consumer may process a “subscription cancelled” event before the “subscription created” event that preceded it.

None of these are edge cases. In a platform with real integration volume, all of them happen regularly. Your webhook system needs to handle each one explicitly.

Pattern 1: At-Least-Once Delivery with Idempotency Keys

The most practical delivery guarantee for most SaaS products is at-least-once: you commit to delivering every event at least once, and consumers are responsible for handling duplicates safely. This is more achievable than exactly-once semantics without requiring distributed transaction infrastructure, and it’s what Stripe, GitHub, and Twilio all implement.

To make at-least-once work, every event payload must carry a stable idempotency key — typically a UUID generated when the event is first created, stored alongside the event, and included in every delivery attempt.

{
  "id": "evt_01HX8YJKQ2VN3BFKM7TZRG4P6",
  "type": "invoice.payment_succeeded",
  "created_at": "2026-09-11T09:00:00Z",
  "data": { ... }
}

The id field is the idempotency key. The receiving consumer stores it (in a Redis set, a database unique index, or an in-memory bloom filter) and silently discards any delivery carrying an id it has already processed. This makes the consumer idempotent — safe to call any number of times with the same event.

Idempotency keys should be generated at event creation, not at delivery time. If you generate a new ID on each retry, your consumer has no way to detect the duplicate.

Pattern 2: Exponential Backoff with Jitter

When a delivery attempt fails — connection refused, 5xx response, timeout — the correct response is to schedule a retry, not to give up and not to retry immediately. Retrying immediately against an endpoint that just returned a 503 will fail again. Retrying with a fixed interval floods a recovering server with requests at the worst possible moment.

Exponential backoff increases the wait between retries geometrically: 30 seconds, then 2 minutes, then 8 minutes, then 30 minutes, then 2 hours. The exact schedule varies, but the shape is consistent: early retries catch transient errors quickly; later retries give real outages time to resolve.

Jitter adds a small random offset (±10–20%) to each wait interval. Without it, all events that failed during a brief outage retry at exactly the same moment — turning a recovered service into a thundering herd.

A practical retry schedule for most SaaS products:

Attempt	Delay
1	30s (±5s)
2	2m (±20s)
3	10m (±1m)
4	1h (±6m)
5	6h (±30m)
After attempt 5	Move to dead-letter queue

The maximum attempt count and schedule should be configurable per webhook endpoint, since a payment event and a “user logged in” event have very different business criticality.

Pattern 3: Dead-Letter Queues for Undeliverable Events

After your retry schedule is exhausted, the event moves to a dead-letter queue (DLQ): a durable store of events that could not be delivered despite repeated attempts.

The DLQ serves two purposes. First, it gives you operational visibility — an alert on DLQ depth is an early signal that a customer’s endpoint is broken, their server is down, or your delivery logic has a bug. Second, it preserves the event for manual or automated redelivery once the underlying issue is resolved.

Your webhook dashboard (the one your customers see) should surface DLQ events per endpoint and allow both bulk redelivery and manual dismissal. Customers need to know what they missed, and they need the ability to replay it once their receiver is back up.

Events in the DLQ should be retained for at least 72 hours — enough time for a customer to notice the failure, investigate, and request replay. For high-value event types (payment events, compliance-relevant state changes), consider a longer retention window.

Pattern 4: Delivery Ordering and Sequence Numbers

At-least-once delivery doesn’t guarantee order. If event B was created after event A but A’s fifth retry fires after B’s first delivery, your consumer sees B before A.

For most event types, out-of-order delivery is acceptable — the consumer can derive the correct state from each individual event without needing the sequence. For others (subscription lifecycle events, order state machines), it matters.

The pragmatic solution for order-sensitive events is to include a sequence number or a created_at timestamp in the payload and let the consumer use it for conflict resolution. If a “subscription cancelled” event arrives with a created_at earlier than the “subscription upgraded” event already in the consumer’s database, the consumer can safely discard it.

For consumers that need strict ordering within a single customer or resource, a per-resource event queue — where events for the same customer ID are processed sequentially — provides ordering guarantees without blocking unrelated events.

Pattern 5: Signature Verification

A webhook is an authenticated HTTP request from your platform to your customer’s infrastructure. But any server on the internet can POST to your customer’s endpoint — including a malicious one. Without signature verification, your customer’s webhook handler is an unauthenticated API endpoint.

The standard approach is HMAC-SHA256 signing. When a customer registers an endpoint, generate a signing secret unique to that endpoint. Before every delivery, compute:

signature = HMAC-SHA256(signing_secret, raw_request_body)

Include the signature in the request as a header (conventionally X-Webhook-Signature or Svix-Signature). The consumer recomputes the signature from the raw request body and the signing secret they received at registration, and compares it to the header value using a constant-time comparison to prevent timing attacks.

Two caveats that catch teams repeatedly:

• Sign the raw bytes of the request body before any JSON parsing. Parsing and re-serializing can change key ordering, whitespace, or number formatting, producing a different byte sequence.
• Include a timestamp in the signed payload and validate that it’s within a tolerance window (e.g., ±5 minutes). This prevents replay attacks where a valid signature is captured and reused minutes later.

Pattern 6: Timeouts and Async Acknowledgement

Your delivery worker should enforce a hard timeout on the HTTP connection — 30 seconds is a common ceiling. If the receiving server hasn’t responded in that window, treat it as a failure and schedule a retry.

Consumers should be designed to acknowledge quickly (respond HTTP 200 as soon as the event is received) and process asynchronously (enqueue the event to an internal queue and return immediately). A webhook handler that does database writes, sends emails, or calls third-party APIs inline before returning risks exceeding your timeout and triggering spurious retries.

Document this expectation for your customers. “Your handler should respond within 10 seconds. For longer processing, acknowledge immediately and handle asynchronously” is explicit, actionable, and reduces the support load on your team.

Operational Visibility: What to Expose in Your Dashboard

A reliable webhook system needs corresponding operational tooling:

• Per-endpoint delivery log: every attempt, with timestamp, HTTP status, response body snippet, and latency.
• Retry queue depth: how many events are scheduled for redelivery, grouped by attempt number.
• DLQ size and oldest event age: the two numbers that tell you whether a customer’s endpoint is critically broken.
• Success rate over time: a rolling 24-hour chart per endpoint that surfaces degradation before it becomes an incident.
• Manual redelivery: the ability for customers (and your support team) to replay any individual event or bulk-replay a time range.

Stripe’s webhook dashboard is the reference implementation here. The depth of logging and the ability to replay arbitrary events have saved countless customer integrations. If you’re building a SaaS platform with integration-heavy customers, this level of tooling is not optional.

Choosing Your Delivery Infrastructure

Most teams reach for one of three approaches:

1. Build on a managed queue (AWS SQS, Google Pub/Sub, Redis Streams). Your own application layer handles the endpoint registry, retry scheduling, and DLQ logic while delegating durable queuing to the managed service.

2. Use a dedicated webhook delivery service (Svix, Hookdeck, Convoy). These handle delivery, retries, DLQ, the customer dashboard, signature verification, and more out of the box. Worth evaluating for any team that doesn’t want to own the infrastructure.

3. Build it entirely in-house on a job queue (Sidekiq, BullMQ, Temporal). This gives you maximum control and no external dependency, at the cost of implementation time.

The right answer depends on your scale, your team’s capacity, and how differentiated your webhook infrastructure needs to be. For early-stage SaaS, a managed queue plus a thin in-house delivery layer is often the fastest path to production-grade reliability.

---

Building robust webhook delivery is one of those foundational investments that pays forward every time you add a new integration. If you’re designing the integration layer for a new SaaS product or hardening the one you already have, Nevrio’s engineering team can help you get the architecture right from the start.

Start a project and let’s build integrations your customers can depend on.