Designing reliable webhooks with retries and idempotency

A webhook tells your application that something happened elsewhere. The hard part starts when the network fails: did your application miss the event, save it, or already change the customer's account? Retrying is necessary in some of those cases and dangerous in others.

This guide is for developers building integrations into an existing application. You will learn where to acknowledge a request, what to store, and which operations must be safe to repeat. The reference design uses one application database and a background worker. You do not need a message broker or a particular cloud provider.

Our running example is a hypothetical export service. A paid invoice grants 100 credits to a workspace. The rule is one initial grant per eligible invoice, even if several notifications or repair jobs refer to it. A confirmation email is an optional second effect.

Start with the system boundary

Figure 1. The inbox and business records share one database. A worker reads pending inbox rows directly; there is no separate queue handoff. The dashed path is needed only for an external effect.

An inbox is a durable list of received events. The worker can recover unfinished rows after a restart. An outbox is a durable list of outgoing actions; add it only when a committed business change must trigger an external action. They sit on opposite sides of processing.

Acknowledge the work you can recover

The endpoint first verifies authenticity using the sender's documented signature scheme, then validates the event type and required identifiers. Preserve the raw body when the scheme requires it. Map the verified source account to your tenant; a workspace ID supplied in the payload is not, by itself, authorization. See the official Stripe signature guide for one concrete contract.

For an accepted event, insert an inbox row and commit it before returning a successful HTTP response. Keep the event key, normalized input or a securely retained payload, processing status, attempt count, next attempt time, and last error. Store only what you need, restrict access, and avoid copying payloads into general logs.

That response means 'I have recoverable work.' It does not mean 'the credits have been granted.' If the commit fails or its outcome is uncertain, do not claim successful receipt. A later delivery can safely resolve the uncertainty by checking the durable key. Invalid signatures are rejected; authenticated event types you intentionally ignore can receive a success response under the sender's contract.

On a duplicate, confirm that the existing row is durable and belongs to the same scoped event, then acknowledge it. Leave a pending or failed row recoverable rather than marking it done. If the same ID arrives with inconsistent content, quarantine and investigate instead of silently overwriting it.

Use two identities for two different questions

Event key: (source, source account, event ID). This answers 'Have I accepted this event before?' Choose the sender's stable event or redelivery identity, not an ID regenerated for every HTTP attempt. Make all key fields non-null and enforce a database uniqueness constraint.

Business key: (source, source account, invoice ID, initial-credit-grant). This answers 'Has this invoice already funded its initial grant?' Two different event IDs, a manual repair and a webhook can all refer to this same operation. A subscription ID would be too broad because later invoices should grant new credits.

The database, not a preceding 'does it exist?' query, must arbitrate concurrent attempts. PostgreSQL's unique constraints provide this mechanism. Retain the business record for as long as the operation must remain non-repeatable; a short-lived deduplication cache cannot enforce that rule.

The example assumes a trustworthy mapping from the invoice to its workspace and credit entitlement. Validate eligibility and amount before applying the grant. Refunds and corrections are separate, identified operations, not another execution of the initial grant.

Commit the local effect and its outcome together

For this small design, each worker opens a short transaction, locks one pending inbox row whose next-attempt time is due, and keeps that lock until it commits. Other workers skip locked rows. A crashed connection releases its lock and rolls back uncommitted changes, so another worker can pick up the row. A lease-based claim can also work, but needs expiry, ownership checks and protection against a stale worker. PostgreSQL documents this queue-consumer use of SKIP LOCKED.

The following is transaction-level pseudocode for the credit example. Grant insertion returns whether this transaction created the row. The balance increment is atomic, and every participating table uses the same database transaction.

BEGIN
  event = lock_one_due_inbox_row(skip_locked = true)
  if no event: COMMIT and stop

  validate invoice, tenant and fixed credit entitlement
  created = insert_grant_if_absent(business_key, 100)
  if created:
    increment_workspace_balance_atomically(100)
    insert_outbox_if_needed(business_key + ':confirmation')
  else:
    verify_existing_grant_matches_expected_operation()

  mark_inbox_done(event.key)
COMMIT

A rollback undoes the new grant, the balance change, any outgoing intent, and the completion mark. A commit preserves them together. If the worker loses the commit response, it rereads durable state instead of blindly applying credits again. A separate event for the same invoice may reach the worker later, but the grant's unique business key prevents another increment.

Do not make remote API calls while holding this transaction open. If eligibility requires a remote lookup, obtain the information first and revalidate the relevant local state when committing. Ordering-sensitive lookups need the additional coordination described below.

For transient failures, roll back and persist a retry schedule in a separate, short transaction for rows still pending. Use capped exponential backoff with jitter and a finite attempt or age budget. Record validation failures and exhausted retries for operator review. This bookkeeping must never overwrite a completion committed by another worker.

When the simpler version is enough

If the entire effect is a fast local database change and fits comfortably within the sender's deadline, the endpoint can perform one transaction containing deduplication, the business update and completion, then acknowledge. No background worker or outbox is needed. Use the inbox-and-worker design when processing is slow, bursty, or needs recovery independent of the sender. Add a broker for demonstrated scaling or isolation needs, not just because this is a webhook.

Follow a lost response through the system

Figure 2. The response is lost after receipt commits. Redelivery reuses the inbox row. Processing later commits one grant. A different event ID for the same invoice is stopped by the business key, not the event key.

There are two retry loops here. Sender redelivery retries the HTTP exchange. Your worker retries unfinished processing after receipt has already succeeded. Returning a 2xx ends neither your responsibility for the row nor an operator's need to resolve a permanent failure.

An external effect needs another agreement

Email cannot join your database transaction. Sending before commit risks sending for a rolled-back grant; sending afterward without recording an intent risks losing the email in a crash. If a confirmation must follow the grant, insert its outbox row in the same transaction as the grant. A dispatcher later reads pending outbox rows and calls the email service.

The dispatcher can crash after the email service accepts a request but before the outbox is marked sent. It may send again. Use a stable operation key if that service supports idempotency, and check its scope, expiry and replay rules. If it does not, choose an explicit policy for ambiguous outcomes: reconcile against its delivery records where possible, accept a duplicate for low-risk messages, or require review before retrying high-impact actions.

The transactional outbox pattern closes the gap between committing local state and recording outgoing work. It does not provide end-to-end exactly-once delivery. An outbox dispatcher and its destination still need their own failure handling.

Apply the same design to feedback alerts

Consider a SaaS team that wants new Sinalio feedback to appear in its internal operations panel, with an optional notification to a team channel. This uses the same boundaries, but its business operation is 'create the initial alert for this feedback' rather than 'grant credits for this invoice'.

Sinalio's webhook settings provide a receiver configuration for signed new-feedback notifications. On 4 October 2026, the public site still described webhooks as in Alpha validation. The flow below proposes how a customer application could consume those notifications. It does not specify official event names, payload fields, signature verification steps or retry rules, which must be confirmed before implementation.

Figure 3. Proposed Sinalio-to-customer flow. The receiving endpoint, database, worker and optional outbound integration belong to the SaaS customer. A local alert needs no external dispatch.

Using the verified integration contract, the customer's endpoint authenticates the webhook and saves a receipt in its inbox. Its worker stores or links the feedback and creates one alert record. For a single initial alert, a proposed business key is (Sinalio project, stable feedback ID, initial-alert). These are conceptual identities, not official payload field names. If alerts vary by destination, include the destination in each outgoing operation's identity.

A repeated delivery should find the same receipt. Different events concerning the same feedback should not create another initial alert. A genuine later alert, such as a separately defined escalation, needs its own operation key. The local feedback reference, alert record, inbox outcome and any outgoing intent should commit together.

If the alert lives only in the customer's dashboard, that local record can be the completed effect. If it must also reach an external team destination, the customer's own integration can dispatch an outbox entry. An ambiguous send still needs downstream idempotency or the duplicate-versus-omission policy described above.

Before implementing, confirm the event catalogue and schema, stable event and feedback identities, source-project scoping, authenticity verification, timeout and retry behavior, replay options, and plan eligibility. Public marketing copy is not enough to infer any of these contracts.

Retries do not establish event order

For our credit ledger, invoice A and invoice B are separate legitimate grants. Processing B first must not suppress A. A current-state projection is different: an old 'active' notification must not overwrite a newer cancellation. Deduplicating both events does not decide which state is current.

If the provider exposes an authoritative monotonic version, condition updates on that version. Otherwise, reconcile current state from the provider and coordinate the entire fetch-and-apply cycle per resource so an earlier fetch cannot overwrite a later result. Respect the API's consistency guarantees and run reconciliation to repair drift. Do not use arrival order or an ordinary timestamp as a universal sequence number.

Check the sender's contract before choosing recovery

Design processing to tolerate repeated attempts without assuming every webhook provider guarantees delivery. At-least-once is a transport guarantee of one or more deliveries under its stated conditions; it does not mean a unique business effect. Some retry automatically; others need your recovery job or an operator. No local architecture can process an event it never receives or retrieves.

For example, Stripe documents duplicate events, no ordering guarantee, and automatic retries for up to three days in live mode. GitHub does not automatically redeliver failed webhooks. These are transport policies to configure around; the inbox, business key and transaction boundary stay application concerns.

When adding a broker, avoid 'save inbox, then publish' as two uncoordinated writes. A crash between them strands accepted work. Keep database polling, use a transactional outbox or change-data capture for the handoff, or deliberately make a durable queue your source of pending work. Verify the chosen queue's acknowledgement, retention and redelivery semantics.

Prove recovery with failure tests

• Send the same event concurrently: one inbox row and one grant.
• Send two event IDs for the same invoice: two receipt records, one grant and one initial confirmation intent.
• Crash after receipt commits: the pending row is still discoverable.
• Crash before and after the business commit: rollback or one committed grant, never a second increment.
• Lose an external service response: exercise the chosen ambiguous-send policy.
• Deliver state updates out of order and overlap replay with normal processing: no stale overwrite or duplicate grant.

Alert on the oldest pending row, exhausted retries, processing failures and reconciliation gaps, not just HTTP success rates. Keep event and business keys in structured logs so an operator can trace an invoice from receipt to outcome. Replays should pass through the same idempotent operation instead of bypassing it.

Start with the smallest design that preserves the business rule. Add each boundary deliberately, and make unfinished work visible. The useful result is a system where a repeated delivery has a known outcome and a failed attempt has a clear path to recovery.