Your inbound webhooks are failing quietly. Not all providers retry reliably, some demand a 3‑second acknowledgment, and others change policies without notice. A missed event today becomes a reconciliation mess tomorrow: unpaid invoices, orphaned GitHub checks, or a customer whose support ticket never linked to their payment. You won’t notice until a CFO or COO does.
The uncomfortable truth about third‑party webhooks
Most SaaS teams treat webhooks as reliable notifications. They’re not. They’re best‑effort messages delivered over an unreliable network into your often‑cold backend. A few realities you should internalize:
- Retry behavior varies wildly. Stripe will retry delivery for days; Slack expects a 3‑second 2xx and may quickly resend if you don’t; some dev flows (like tunnels for local GitHub testing) won’t retry at all. Policies change, and you won’t be notified.
- Delivery is at‑least‑once, not exactly‑once. Duplicates and out‑of‑order arrivals are normal. If your handlers aren’t idempotent, you’re building a time bomb.
- Middleboxes can break signatures. CDNs and proxies that re‑encode or chunk the body will invalidate HMAC calculations. If your signature code runs on parsed JSON, you’re already wrong; it must run on the exact raw body bytes.
- Cold starts and synchronous work get you dropped. Slack gives you ~3 seconds. GitHub and Stripe expect a fast 2xx. If you do anything non‑trivial before acknowledging, you’re creating your own losses.
One recent developer write‑up highlighted a common pain: in local development, GitHub won’t retry when a tunnel or laptop is down. That’s a dev symptom of a prod‑level disease: if you haven’t made delivery durable on your side, you’re betting your business on someone else’s retry policy.
The durable webhook front door
You don’t need Kafka, three new teams, and a six‑month program. You need a small, explicit front door whose only job is to capture, acknowledge, verify, and queue events safely. Then you can process them asynchronously with proper SLOs.
Design constraints you must meet
- Fast 2xx: Respond within 50–200 ms in normal conditions. Slack’s 3‑second limit is the harshest; aim to spend your entire budget on a durable write and nothing else.
- At‑least‑once safe: Every component after the ACK must be idempotent. Expect duplicates, missing order, and partial deliveries.
- Byte‑exact signature verification: Compute HMAC on the raw request body, not a parsed object. If you deploy via a CDN or gateway, ensure it preserves the raw stream.
- Provider chaos isolation: Treat each provider’s quirks as a policy, not a surprise. Centralize them in one place.
The blueprint
- Dedicated ingress service (thin, boring, fast): Terminate TLS, read the raw body stream, and write it—headers and bytes—into an append‑only store. Don’t parse JSON here. Don’t call your app. Don’t touch business logic. Your only job is to persist and ACK.
- Durable raw event store: Two practical patterns work well:
- Postgres table partitioned by day with columns: provider, path, method, headers (JSONB), raw_body (bytea), received_at (timestamptz), remote_addr, signature_status, checksum, delivery_id (if present). Index on (provider, delivery_id) and received_at.
- S3 + Postgres index: Store raw bodies in S3 (gzipped), and a small Postgres index row with metadata and S3 key. Cheaper for large bodies, trivial to retain for 90+ days.
- Immediate 2xx after durable write: Once the raw event is fsynced (or S3 PUT returns 200 and the index row commits), return 200 or 202. That’s the contract.
- Signature verification worker: A small worker pulls new rows and computes provider‑specific verification using the exact raw bytes. Mark events as verified or rejected. Rejected events never enter the business queue, but you still keep them for forensic and rate‑limit analytics.
- Dedup and enqueue: Derive a dedup key. Prefer the provider’s event ID. If it’s missing, hash the tuple of (provider, path, canonicalized headers subset, raw body bytes). Use a 30–90 day window. Push a small, normalized envelope onto your processing queue (Kafka/Redpanda topic, SQS, or a Postgres outbox table replicated to workers).
- Asynchronous handlers (idempotent, retrying): Business logic subscribes to the queue. Every handler must be idempotent and side‑effect safe. No handler assumes ordering. Handlers emit their own checkpoints so you can replay safely.
- Replay console: Build a minimal UI to search raw events, see verification state, and push selected events back onto the queue. You will use this weekly.
- Backfill and reconciliation jobs: For providers with list APIs, schedule periodic scans to detect drifts. Example: nightly pull of Stripe events by created timestamp and compare with your dedup index; fetch GitHub delivery logs for Apps and cross‑check unknown IDs; for Slack, compare channel history with internal actions for critical workflows.
Numbers that keep you honest
- Latency budget: Durable write in Postgres with synchronous_commit = on and a local NVMe can complete in 3–12 ms at P50 and under 50 ms at P95 under moderate load. S3 PUT typically returns in 30–120 ms from the same region. Combined, you should stay well under Slack’s 3‑second cap with massive headroom.
- Storage cost: 1 million events/day with a 1.2 KB median body is roughly 36 GB/month of raw payload. S3 standard is single‑digit dollars for that footprint; even twice that with metadata is noise compared to the risk of losing events.
- Retry envelopes: Stripe retries for up to ~3 days with exponential backoff; Slack’s behavior focuses on rapid redelivery in seconds and expects quick ACKs; other providers offer best‑effort redelivery and manual replays. Your design cannot depend on their policies.
Security you can prove
Webhook security is not an IP allowlist glued to a WAF. Do the basics well and you’ll avoid 90% of incidents:
- Raw‑body HMAC verification per provider with clock‑skew‑tolerant timestamp checks. Reject if timestamps are too old or too far in the future.
- Secret rotation at least quarterly. Keep KMS‑encrypted copies of current and previous secrets to handle rotation windows.
- Strict content limits: Cap request size by provider. Pre‑verify presence of signature headers before you read large bodies. If signatures are missing, drop early with 400 to avoid wasting cycles.
- CDN/gateway configuration: If you must front webhooks with a gateway, use a pass‑through route. Disable transformations, compression, and any middleware that could mutate payload bytes. Preserve the request stream as‑is.
- Least privilege for processing: The ingress service writes to raw storage and a verification queue. It cannot touch business systems. Verification workers cannot mutate business state. Side‑effecting handlers run in tightly scoped roles.
- Never log secrets: Do not log raw payloads in app logs. Keep payloads in the raw store only, with encryption at rest and structured, audited access.
Multi‑region without tears
If you operate in two regions, you can make the front door active‑active without inventing a consensus protocol.
- Global load balancer routes to the closest region. Expect duplicates due to transient re‑routes and provider retries; your dedup key makes that safe.
- Region‑local durability: Each region writes to its own raw store. Asynchronous replication (e.g., S3 cross‑region replication or logical replication for the Postgres index) provides a unified view for your replay console.
- Consistent secrets: Keep provider secrets in a replicated vault. Automate rotation in both regions.
- Blast radius: If a region melts, the other continues to accept and persist. You may process from a single region until recovery; nothing is lost.
Operational SLOs and the only dashboards that matter
Webhook reliability dies in the gaps between teams. Publish SLOs and wire them to alerts your on‑call actually respects:
- Ingress ACK SLO: P99 time to 2xx under 250 ms per provider.
- Verification lag: P95 under 60 seconds from receipt to verified state.
- Processing lag: P95 under 5 minutes from verified to handler checkpoint committed (tighter for user‑visible actions like CI checks or Slack commands).
- Replay success: 99% of selected raw events replay within 2 minutes.
- Drift rate: Less than 0.01% mismatches between provider list APIs and your dedup index in nightly reconciliation.
A single page should show: incoming volume by provider, ACK latency, verification failure rates (with reasons: bad signature, stale timestamp, oversized), queue depth, handler lag, and a red/yellow/green drift metric from reconciliation jobs.
Provider‑specific footguns to design around
- Slack: 3‑second ACK window. Never call your app before responding. Slack may send quick redeliveries with headers indicating retry count. Build for that.
- Stripe: Exposes strong event IDs; use them as your dedup key. Always verify via raw‑body HMAC with their signing secret and timestamp tolerance.
- GitHub: Policies evolve. GitHub supports manual redelivery and best‑effort retries, but you should not depend on them. For GitHub Apps, store the X‑GitHub‑Delivery ID for dedup and replay.
- CDN and server frameworks: Node/Express body parsers, some API gateways, and HTTP frameworks that auto‑parse JSON will break HMAC verification if you’re not careful. Capture the raw stream before any parser touches it.
Cost and complexity: what you can ship in 4–6 weeks
Here’s a pragmatic delivery plan we’ve executed with lean teams:
- Week 1–2: Stand up ingress service, raw store, and immediate 2xx path. Wire basic dashboards for ACK latency and volume. Put a single provider (often Stripe or Slack) behind the new front door.
- Week 3: Add verification workers with raw‑body HMAC. Implement dedup table/index and normalized envelopes. Start feeding a queue (SQS or Kafka) and port one handler to asynchronous processing.
- Week 4: Build the minimal replay console. Add alerting for verification failures and lag SLOs. Backfill job for the first provider.
- Week 5–6: Migrate other providers. Add reconciliation jobs where list APIs exist. Review CDN/gateway configs for pass‑through routes. Rotate the first batch of webhook secrets to validate the path.
After that, multi‑region is a configuration exercise, not a redesign. You already have dedup, replay, and backfill. The rest is plumbing.
Why this matters more in the age of agents
As more features become event‑driven—CI automations, finance reconciliations, AI agents responding to external changes—your system’s “truth” depends on a clean, durable intake of external signals. If you can’t prove delivery, you can’t trust automation. Durable webhooks are the minimum viable substrate for reliable agents and deterministic workflows.
Trade‑offs and anti‑patterns
- ACK after processing: Simple but fatal at scale. Under cold starts or spikes, you’ll miss events. Don’t do it.
- One queue per provider vs. a shared topic: One per provider keeps blast radius small and debugging clear. A shared topic can work if you enforce strict schemas and routing keys.
- Kafka everywhere vs. S3+Postgres first: If you already run Kafka, great. If not, S3 + a Postgres index plus SQS is enough. Don’t let infra religion delay durability.
- Parsing at ingress: It’s tempting to validate JSON immediately. Resist. Persist raw bytes first so you can re‑verify signatures forever, even if libraries or encodings change.
- Relying on provider dashboards: They’re helpful, but they’re not your system of record. Your raw store is.
What good looks like
Six months from now, you should be able to answer, with evidence, the questions executives and auditors actually ask:
- Can you show that every Stripe event received in the last 90 days was persisted, verified, and either processed or rejected with a reason?
- When Slack had a regional blip last Tuesday, how many events were delayed, and how fast did they catch up?
- When you rotated GitHub webhook secrets last quarter, did any signatures fail due to drift or gateway mutation?
- Can your on‑call replay a specific customer’s missed event in under two minutes without SSH or ad‑hoc scripts?
If you can’t demonstrate those today, you’re flying blind. The durable front door is not a nice‑to‑have; it’s a control surface for your business.
Key Takeaways
- Webhooks are best‑effort messages. Treat them as at‑least‑once and design for duplicates, disorder, and provider quirks.
- Build a thin ingress that persists raw bytes and ACKs fast. Verify signatures and dedup asynchronously before business logic runs.
- Store every event in a durable, queryable raw store for 30–90 days. It’s cheap insurance and your replay source of truth.
- Publish SLOs for ACK latency, verification lag, processing lag, and reconciliation drift. Alert on them.
- Do not depend on provider retries or dashboards. Own replay and backfill with your console and jobs.