Pointer Walk

Idempotency and Safe Retries

The reply was lost, so the client sends the payment again. An idempotency key is what stops the customer being charged twice.

Distributed SystemsIntermediate8 min read

Did it work?

Your app sends a request to charge a customer's card and waits for the reply. No reply comes. The request has timed out.

What happened? There are two possibilities, and from where your app sits they look identical.

  1. The request never arrived. The card was not charged.
  2. The request arrived, the card was charged, and the reply got lost on the way back.

In the first case you must try again. In the second case you must not. You cannot tell which case you are in.

A retry with no protectionInteractive
Shop app
Payment service
Customer's card
charged: 0

A customer pays 50 for an order. Watch the third message.

This is not a rare corner case. On any network, replies get lost. And every well-built client retries on a timeout. See Retries, Backoff and Jitter. Put those two facts together and duplicate requests are guaranteed to happen eventually.

Make doing it twice harmless

Since the sender cannot know, the fix has to be on the receiving side: build the operation so that doing it twice has the same effect as doing it once.

An operation with that property is called idempotent.

Example

Lift buttons

Press the button for the 5th floor. Press it nine more times. You still go to the 5th floor, once. The button is idempotent.

A vending machine's coin slot is not. Ten coins is not the same as one.

Some operations are idempotent by nature.

OperationRun twiceIdempotent?
Set the address to "12 Oak St"Address is "12 Oak St"Yes
Delete order 81Order 81 is goneYes
Read the balanceSame answer, nothing changesYes
Add 50 to the balanceBalance is up by 100No
Create a new orderTwo ordersNo
Send an emailTwo emailsNo

"Set to a value" is safe. "Change by an amount" and "create something new" are not. The dangerous ones tend to be the ones that matter most.

The idempotency key

For operations that are not naturally idempotent, there is a standard technique.

  1. The client makes up a unique id for the operation

    A random UUID, generated once, when the user presses Pay.

  2. It sends that id with the request

    Usually in a header called Idempotency-Key.

  3. The server stores the id with the result

    In the same transaction as the change itself.

  4. On any retry, the client sends the same id

    Same purchase, same id. A different purchase gets a different id.

  5. If the server has seen the id, it does not repeat the work

    It returns the answer it saved the first time.

With an idempotency keyInteractive
Shop app
Payment service
Customer's card
charged: 0

A customer pays 50 for an order. Watch the third message.

python
def charge(key, amount):
    with transaction():
        saved = db.find_result(key)
        if saved is not None:
            return saved                      # a retry: do nothing new

        result = bank.charge(amount)
        db.save_result(key, result)           # remembered with the charge
        return result
A retry with no protectionInteractive
Request
Shop app
Payment service
Customer's card
charged: 0

A customer pays 50 for an order. Watch the third message.

Details that matter

  • The client creates the key, before the first attempt. If a new key were made for each retry, every retry would look like a new request and nothing would be gained.
  • Save the key and do the work together. If the work is done and the server crashes before recording the key, the retry repeats the work. They must succeed or fail as one.
  • Two copies may arrive at the same moment. A unique constraint on the key makes the database reject the second one.
  • Keys can expire. Retries happen within minutes or hours, so keys are usually kept for a day or so.

"Exactly once" is built from two halves

People ask for a message to be delivered exactly once. Across a network that cannot be guaranteed directly, because of the lost-reply problem at the top of this page. A sender has only two choices.

GuaranteeHowRisk
At most onceSend it and never retryMessages are lost
At least onceRetry until acknowledgedMessages are duplicated

The practical answer is to combine at-least-once delivery with an idempotent receiver. The message may arrive three times. It takes effect once. That is what "exactly once" means in real systems.

Where you need it

  • Payments and orders. The textbook case.
  • Message queues. Nearly all of them deliver at least once, so every consumer must cope with repeats.
  • Webhooks. The sender retries if your server is slow to answer.
  • Any button a user can press twice.

Key takeaways

  • When a reply is lost, the sender cannot tell whether the request was carried out.
  • Retries are necessary, so duplicate requests are certain to happen.
  • An idempotent operation has the same effect whether it runs once or many times.
  • An idempotency key lets the server recognise a repeat and return the saved result.
  • Exactly-once behaviour is at-least-once delivery plus an idempotent receiver.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress