How to Handle Payment Gateway Webhooks Reliably: Retries, Failures, and Duplicate Events

A customer completes an online payment, but the order remains pending. The payment gateway confirms the transaction, yet your application never updates its records.
This often happens because of delayed, failed, or incorrectly processed payment gateway webhooks.
Webhooks allow payment gateways to send transaction updates directly to merchant servers. However, network failures, duplicate notifications, and processing errors can create inconsistencies between payment records and actual transactions.
This guide explains how to handle payment webhooks reliably using retries, idempotency, secure validation, and Node.js.

What Are Payment Gateway Webhooks?

A payment gateway webhook is an HTTP notification sent by a payment provider to a merchant's backend when a supported transaction event occurs.
For example, after a successful payment, the gateway sends a notification containing transaction details and payment status.
A typical workflow looks like this:
Customer initiates payment
↓
Payment gateway processes transaction
↓
Gateway generates payment event
↓
Webhook sent to merchant backend
↓
Merchant verifies and stores event
↓
Application updates payment status
An illustrative webhook payload:
{
"eventId": "evt_10293",
"eventType": "payment.success",
"orderId": "ORD12345",
"transactionId": "TXN78901",
"amount": 2500,
"currency": "INR"
}
Note: This is a generic example. Actual payloads differ by payment gateway.
Unlike browser redirects, webhooks communicate directly with the backend, allowing applications to receive updates even when customers close their browsers.

Why Do Payment Gateway Webhooks Fail?

Webhook failures generally occur for three reasons.

1. Network and Delivery Failures

The payment gateway may be unable to reach the merchant's endpoint because of server downtime, network interruptions, DNS issues, or request timeouts.
The payment itself may succeed even when its notification fails.

2. Backend Processing Errors

A webhook can reach the application but fail during processing because of database errors, unavailable internal services, or application exceptions.

3. Duplicate or Out-of-Order Events

Payment gateways may deliver the same event multiple times when acknowledgements are lost.
Events can also arrive out of order, potentially causing an older pending status to overwrite a confirmed successful transaction.
These problems require careful processing rather than assuming every notification arrives exactly once.

How to Handle Webhook Retries

Webhook retries occur when a gateway attempts to deliver an event again after an unsuccessful delivery.
Retry schedules and acknowledgement requirements vary across providers.

• A reliable webhook receiver should:
• Authenticate the incoming request.
• Validate the event structure.
• Store the event in durable storage.
• Return the acknowledgement required by the gateway.
• Process the event asynchronously.
  Avoid performing expensive operations such as invoice generation or email delivery before acknowledging the webhook.
  For internal processing failures, implement retries with exponential backoff.
  For example:
  

These are illustrative application-level delays, not a payment provider's delivery schedule.
After repeated failures, move the event to a dead-letter queue for investigation.

Preventing Duplicate Webhook Events With Idempotency

Suppose a gateway successfully delivers a notification, but the merchant's acknowledgement is lost.
The gateway retries the webhook, and the backend receives the same event again.
Without duplicate protection, this could trigger repeated fulfilment or financial operations.
Idempotency ensures that processing the same event repeatedly does not create additional unintended effects.
One approach is to store a unique event identifier in PostgreSQL:

Insert new notifications using:

The primary key prevents duplicate insertion.
However, some gateways do not provide globally unique event IDs. In those cases, use the provider's documented deduplication fields.
Also make downstream operations idempotent. Preventing duplicate webhook records alone does not guarantee that payment credits, refunds, or fulfilment actions execute only once.

Implementing a Secure Webhook Receiver in Node.js

Let's create a basic webhook endpoint using Express.
Install the required package:
npm install express
The following example demonstrates signature verification using a hypothetical gateway that signs the raw request body with HMAC-SHA256.

const secret = process.env.WEBHOOK_SECRET;

if (!secret || !Buffer.isBuffer(req.body)) {
  return res.sendStatus(400);
}

if (
  typeof signature !== "string" ||
  !/^[a-f0-9]{64}$/i.test(signature)
) {
  return res.sendStatus(401);
}

const expected = crypto
  .createHmac("sha256", secret)
  .update(req.body)
  .digest();

const received = Buffer.from(signature, "hex");

if (!crypto.timingSafeEqual(expected, received)) {
  return res.sendStatus(401);
}

let event;

try {
  event = JSON.parse(req.body.toString("utf8"));
} catch {
  return res.sendStatus(400);
}

try {
  await persistEventIdempotently(event);

  return res.sendStatus(200);
} catch (error) {
  console.error("Webhook persistence failed");
  return res.sendStatus(503);
}

}
);

app.listen(3000);

Important: persistEventIdempotently() represents an application-defined database operation and must be implemented before running this example. The code is illustrative, not a complete production integration.
Real gateways may use different signature formats, checksum algorithms, or authentication mechanisms. Always follow the provider's official documentation.
A production implementation should additionally validate event schemas, protect against replay attacks, enforce request-size limits, and process stored events through a recoverable background worker.

Handling Out-of-Order Payment Events

Webhook notifications may not arrive in the same order as transaction events.
For example:
10:00:00 → Payment pending
10:00:02 → Payment successful
10:00:03 → Success webhook received
10:00:05 → Pending webhook received
If the application blindly processes the last notification received, it may incorrectly change a successful payment back to pending.
Use a payment-state machine that permits only valid transitions.

This is a simplified model. Production systems must account for payment attempts, refunds, reversals, and provider-specific states.
When notifications conflict, verify the authoritative transaction status using the payment gateway's status API.

Payment Webhook Security and Monitoring Best Practices

For reliable production operations:

• Verify authenticity: Validate signatures or the authentication mechanism required by the gateway.
• Use HTTPS: Protect webhook traffic in transit.
• Validate payloads: Confirm transaction references, amounts, currencies, and supported event types.
• Prevent duplicate effects: Apply idempotency to database updates and downstream operations.
• Monitor failures: Track processing latency, duplicate events, unsuccessful deliveries, and queue backlogs.
• Implement reconciliation: Compare internal payment records with gateway transaction data to identify missing updates.
  These measures improve both security and operational reliability.

Payment Gateway Webhook Integration With EnKash

Developers integrating the EnKash Payment Gateway can use its payment APIs and server-to-server notifications to manage transaction updates.
EnKash provides payment-status APIs that developers can use to verify transaction outcomes when notifications are delayed or require further validation.
The EnKash payment gateway webhook documentation provides integration-specific information about notification payloads and related requirements.
When implementing the patterns discussed above, developers should follow the gateway's documented verification, event-handling, and response requirements rather than assuming that all payment providers use identical webhook formats.
Frequently Asked Questions

Can payment gateways send duplicate webhooks?

Yes. Duplicate notifications may occur because of delivery retries or lost acknowledgements. Use event deduplication and idempotent business operations to prevent duplicate processing.

What happens when a payment webhook fails?

Depending on the provider, the gateway may retry delivery. Merchants should also use payment-status APIs and reconciliation to recover from missing notifications.

What is the difference between payment retries and webhook retries?
A payment retry attempts to complete a payment operation again. A webhook retry repeats the delivery of a transaction notification. They are separate processes.

Should webhook processing happen asynchronously?

Generally, yes. Authenticate and durably store the notification before acknowledging it, then process business operations asynchronously to reduce timeouts and improve recoverability.

Conclusion

Reliable payment gateway webhook integration requires more than receiving an HTTP notification.
Developers must account for network failures, duplicate deliveries, delayed events, and unexpected processing errors.
By implementing secure verification, idempotency, asynchronous processing, controlled state transitions, and transaction reconciliation, businesses can maintain accurate payment records even when notifications arrive late or repeatedly.
The goal is not to guarantee that every webhook arrives exactly once, but to ensure repeated or interrupted delivery never results in incorrect financial operations.