How to Avoid Double Payments in Distributed Systems?
How distributed systems make sure that even when you press the payment button twice the money is deducted only once.
You’re planning a trip and find the perfect room on Airbnb. You click “Book” and proceed to pay $10 for the reservation. However, after a few seconds, the app notifies you that the payment failed due to a technical issue.
You try again, and this time the payment succeeds. But when you check your bank account, you notice you’ve been charged twice for the same reservation. You contact Airbnb support. They apologize, explaining that while the first payment had actually gone through, their system failed to confirm it in time, causing the app to prompt you to retry. They refund the duplicate charge, but the experience leaves you frustrated.
Now, what went wrong? For the engineering team, the issue often boils down to one concept: idempotency.
In this blog, we’ll dive into how idempotency ensures payments are processed exactly once, even in the face of retries, and how distributed systems can be designed to avoid double payments. Let’s explore the flow of an idempotent Payment Service, which acts as a centralized gateway for processing payments via external providers like banks, Stripe, or Paypal
What is Idempotency, and Why is it Important?
In distributed systems, idempotency ensures that executing the same operation multiple times produces the same result as executing it once. For a payment system, this means:
If a user retries a payment due to a timeout or error, the system must not process the payment again.
Instead, it should recognize the retry and return the original response — whether success, failure, or pending.
Flow
Let us try to understand the flow of our system:
Phase 1: Handling the Initial Request
The Client (e.g., Order Service or Refund Service) sends a payment request with a unique
idempotency keyto the Payment Service.
This key uniquely identifies the payment operation (e.g.,
order123-payment456).
2. The Payment Service checks its database for the idempotency key:
If the key exists:
It fetches the cached response (e.g., “Payment Success”) from the database and returns it to the client, preventing duplicate processing.
If the key doesn’t exist:
It creates a new record in the database to indicate the request is being processed.
3. The service acquires a row-level lock on the key to ensure no concurrent processes can modify or access it until the request is complete.
Phase 2: Communicating with the Payment Processor
The Payment Service constructs a request object, including the
idempotency key, and sends it to the external Payment Processor (e.g., Stripe, PayPal, or a bank).
Many payment processors such as stripe and paypal support idempotency by requiring the
idempotency keyin the request headers or payload.
2. The processor returns a response (e.g., “Payment Successful” or “Card Declined”).
Phase 3: Finalizing the Payment
The Payment Service stores the processor’s response in the database:
For successful payments: The response is saved, marking the operation complete.
For retryable failures (e.g., temporary network errors): The client may retry later.
For non-retryable failures (e.g., “Insufficient Funds”): The response ensures the client doesn’t retry.
2. The row lock is released.
3. The service sends the response (success or failure) back to the Client.
Why is the Database Needed in This Flow?
The database ensures idempotency by storing the idempotency key to prevent duplicate processing. It tracks the state of requests (e.g., "in progress" or "completed"), provides concurrency control through row-level locks, and ensures reliability by persisting request data for safe recovery during failures. Without it, consistent and error-free payment processing would not be possible.
Let’s explore three edge cases and how our system handles them.
Failure in Phase 2: Timeout After Payment Processor Success
The payment processor successfully charges the user but fails to return the success response to the Payment Service due to a network timeout.
Solution:
When the client retries, the Payment Service checks the database for the idempotency key and identifies the request as a retry.
It resends the request to the payment processor, which confirms the payment was already processed.
The service updates the database with the success response, releases the lock, and returns the success response to the client, ensuring no duplicate charges.
Failure in Phase 3: Failure While Saving to the Database
The payment processor confirms the payment, but the Payment Service fails to update the database (e.g., due to a database outage).
Upon retry, the service queries the processor using the idempotency key to confirm the payment status.
If the processor indicates the payment succeeded, the service updates the database with the response and skips reprocessing. This ensures no duplicate charges despite the initial database failure.
Retryable Errors During Processing
The payment request fails with a retryable error (e.g., a temporary network glitch).
The client retries the request after the lock expires.
The Payment Service checks the database and confirms the idempotency key.
Before re-sending the request, the service verifies with the processor that no payment was processed.
If no payment was made, the service retries the payment request, updates the database with the response, and returns the success response to the client.
Conclusion
Movement of money and ensuring that payment systems are robust and idempotent are one of the major challenges in distributed systems. Ensuring customer trust during payments is one of the top most priorities of products. This simple process has lead to a lot of issues in the past one such case being UberEats offering free orders in India due to Payment Processor issues.
References
https://docs.stripe.com/api/idempotent_requests
https://developer.paypal.com/api/rest/reference/idempotency/
Schedule a mock System Design Interview with me : https://www.meetapro.com/provider/listing/160769
Linkedin: https://www.linkedin.com/in/mayank-sharma-2002bb10b/
My website : imayanks.com