Did it work?
Your app sends a request to charge a customer's card and waits for the reply. No reply comes. The request has timed out.
What happened? There are two possibilities, and from where your app sits they look identical.
- The request never arrived. The card was not charged.
- The request arrived, the card was charged, and the reply got lost on the way back.
In the first case you must try again. In the second case you must not. You cannot tell which case you are in.
A customer pays 50 for an order. Watch the third message.
This is not a rare corner case. On any network, replies get lost. And every well-built client retries on a timeout. See Retries, Backoff and Jitter. Put those two facts together and duplicate requests are guaranteed to happen eventually.
Make doing it twice harmless
Since the sender cannot know, the fix has to be on the receiving side: build the operation so that doing it twice has the same effect as doing it once.
An operation with that property is called idempotent.
Example
Lift buttons
Press the button for the 5th floor. Press it nine more times. You still go to the 5th floor, once. The button is idempotent.
A vending machine's coin slot is not. Ten coins is not the same as one.
Some operations are idempotent by nature.
| Operation | Run twice | Idempotent? |
|---|---|---|
| Set the address to "12 Oak St" | Address is "12 Oak St" | Yes |
| Delete order 81 | Order 81 is gone | Yes |
| Read the balance | Same answer, nothing changes | Yes |
| Add 50 to the balance | Balance is up by 100 | No |
| Create a new order | Two orders | No |
| Send an email | Two emails | No |
"Set to a value" is safe. "Change by an amount" and "create something new" are not. The dangerous ones tend to be the ones that matter most.
The idempotency key
For operations that are not naturally idempotent, there is a standard technique.
The client makes up a unique id for the operation
A random UUID, generated once, when the user presses Pay.
It sends that id with the request
Usually in a header called
Idempotency-Key.The server stores the id with the result
In the same transaction as the change itself.
On any retry, the client sends the same id
Same purchase, same id. A different purchase gets a different id.
If the server has seen the id, it does not repeat the work
It returns the answer it saved the first time.
A customer pays 50 for an order. Watch the third message.
def charge(key, amount):
with transaction():
saved = db.find_result(key)
if saved is not None:
return saved # a retry: do nothing new
result = bank.charge(amount)
db.save_result(key, result) # remembered with the charge
return resultA customer pays 50 for an order. Watch the third message.
Details that matter
- The client creates the key, before the first attempt. If a new key were made for each retry, every retry would look like a new request and nothing would be gained.
- Save the key and do the work together. If the work is done and the server crashes before recording the key, the retry repeats the work. They must succeed or fail as one.
- Two copies may arrive at the same moment. A unique constraint on the key makes the database reject the second one.
- Keys can expire. Retries happen within minutes or hours, so keys are usually kept for a day or so.
"Exactly once" is built from two halves
People ask for a message to be delivered exactly once. Across a network that cannot be guaranteed directly, because of the lost-reply problem at the top of this page. A sender has only two choices.
| Guarantee | How | Risk |
|---|---|---|
| At most once | Send it and never retry | Messages are lost |
| At least once | Retry until acknowledged | Messages are duplicated |
The practical answer is to combine at-least-once delivery with an idempotent receiver. The message may arrive three times. It takes effect once. That is what "exactly once" means in real systems.
Where you need it
- Payments and orders. The textbook case.
- Message queues. Nearly all of them deliver at least once, so every consumer must cope with repeats.
- Webhooks. The sender retries if your server is slow to answer.
- Any button a user can press twice.
Key takeaways
- When a reply is lost, the sender cannot tell whether the request was carried out.
- Retries are necessary, so duplicate requests are certain to happen.
- An idempotent operation has the same effect whether it runs once or many times.
- An idempotency key lets the server recognise a repeat and return the saved result.
- Exactly-once behaviour is at-least-once delivery plus an idempotent receiver.