Idempotency Keys Done Right

Part 1 ended with a payment that committed while its response got lost. The client retried, and the only thing standing between the customer and a second charge was the server recognizing the retry.

The standard answer is an idempotency key: the client sends a unique value with the request, and the server uses it to spot repeats. Most implementations stop there. Store the key, check it on the next request, done.

The key is the easy part. The hard parts are what the key actually identifies, what you store beside it, what happens when two copies arrive at once, and what you do when the process crashes halfway through. This article walks through each one.

A key identifies an intent, not a request

An idempotency key doesn't name an HTTP request. It names something the user meant to do once: pay this invoice, place this order, release this escrow. Every attempt to carry out that intent, the first one and every retry, must send the same key.

That sounds obvious, but it settles three design questions.

The client generates it, before the first attempt. The server can't generate the key, because the server is exactly the party the client can't hear from when things go wrong. A random UUID v4 is the usual choice. Stripe's documentation also warns against building keys from sensitive data like email addresses.

It is generated once per intent, not once per attempt. A very common bug is a retry helper or HTTP interceptor that creates a fresh key on every call. Each retry then looks like a brand new request, and the server has nothing to match. The key has to be created where the intent is created, such as when the checkout form is submitted, and passed down to whatever does the retrying.

It is scoped. A key should be unique per account or tenant, not globally. Store it alongside the caller's identity and look it up by both. Otherwise one customer's key can collide with another's, and in the worst case a replay hands one customer's response to someone else.

The key travels in a header. Stripe uses Idempotency-Key, and the IETF HTTP APIs working group has a draft that standardizes the same field name. It is still a draft, but most APIs that support this already follow it.

A key alone is not enough

Consider a client that sends this:

Idempotency-Key: abc123    amount: 100

Idempotency-Key: abc123    amount: 100

Then, because of a bug or a reused variable, this:

Idempotency-Key: abc123    amount: 1000

Idempotency-Key: abc123    amount: 1000

If the server checks only the key, it replays the response for the 100 charge. The client believes it charged 1000. Nobody finds out until reconciliation.

The fix is to store a fingerprint of the original request beside the key, typically a hash of the method, the path, and the body. On a repeat, compare fingerprints. If they differ, reject the request instead of replaying. The IETF draft recommends HTTP 422 for this case, and Stripe's API also errors when parameters don't match the original.

Two details matter when building the fingerprint:

• Canonicalize before hashing. The same JSON with keys in a different order or different whitespace should produce the same hash. Hash a normalized form, not raw bytes off the wire.
• Hash only what defines the intent. Leave out headers like timestamps, trace IDs, and auth tokens that legitimately change between retries. Include them and every retry looks like a mismatch.

What an idempotency record holds

A production idempotency record carries more than the key. Each field answers a question the server will ask on a retry:

• Scope and key (unique together): Have I seen this intent from this caller before?
• Request fingerprint: Is this really the same request, or a reused key?
• Status (in progress, completed): Is the first attempt finished, or still running?
• Response status code and body: What do I send back without redoing the work?
• Resource ID: Which payment, order, or object did the first attempt create?
• Locked-at timestamp: If it's still in progress, how long has it been stuck?
• Created-at and expires-at: When can this record be safely deleted?

The composite of scope and key carries a unique constraint. That constraint is not a nice-to-have. As the next sections show, it is the mechanism that makes concurrent requests safe.

The lifecycle of a key

When a request arrives with a key, the server's first move is not to look the key up. It is to claim it: insert a new record in the in-progress state, in a single atomic operation. Everything else follows from whether that claim succeeds.

If the claim succeeds, this is the first attempt. The server runs the operation, stores the response on the record, and marks it completed.

If the claim fails, a record already exists, and the server reads it to decide which outcome applies:

• Fingerprint differs: the key is being reused for a different request. Reject it with 422.
• Completed: this is a retry of finished work. Replay the stored status code and body, without touching the business logic.
• Still in progress: another attempt is running right now. Return 409 Conflict and let the client retry later. The IETF draft recommends 409 for exactly this case.

Notice the order: claim first, then decide. That ordering is what the next section is about.

Two identical requests at the same millisecond

The intuitive implementation looks like this:

1. Look up the key
2. If not found, process the request
3. Save the key with the response

1. Look up the key
2. If not found, process the request
3. Save the key with the response

Now send two copies of the same request at once, which happens easily when a gateway retries while the original is still running. Both look up the key. Both find nothing. Both process the payment. The customer is charged twice, and your idempotency layer watched it happen.

This is the check-then-act race, and no amount of fast lookups fixes it. The check and the claim have to be one atomic step, so that exactly one request can win. Two common ways to get that:

• A unique constraint on scope and key, with an insert that fails or does nothing on conflict. The database guarantees only one insert succeeds. The loser reads the existing record and follows the lifecycle above.
• Redis SET with the NX option (set only if absent), which returns success to exactly one caller.

Either way, only the winner proceeds. Every other copy gets a 409 or, once the winner finishes, the replayed response.

The record that never finishes

Atomic claims create a new problem. Suppose the winner claims the key, then the process crashes before it marks the record completed. Every retry now sees "in progress" and gets a 409, forever.

This is what the locked-at timestamp is for. Treat the in-progress state as a lease, not a permanent lock. If a record has been in progress longer than any request could reasonably take, a retry may take over the claim. What it does next depends on how far the crashed attempt got, and that is the hardest question in this whole design. We'll get to it shortly.

What should be replayed

Once a request finishes, its response is stored. But not every response deserves to be replayed forever.

• Successes: always replay. This is the whole point.
• Client errors like a declined card or invalid input: usually replay. The outcome is final, and retrying the same request won't change it.
• Server errors and timeouts: this is a real design choice.

Stripe takes the strict position. Its documentation says it saves the status code and body of the first request for a key whether it succeeds or fails, including 500 errors. The client gets a consistent answer, and if it wants to try again, it sends a new key. The exception is requests that never started executing: if parameters fail validation or the request conflicts with a concurrent one, Stripe saves nothing and the client can retry.

The alternative is to release the claim on a transient failure, so the same key can be retried. That is friendlier to clients, but only safe if you know the failed attempt produced no side effects. If the 500 came after money moved, releasing the key reopens the double charge you built all this to prevent.

The rule that holds either way: never replay a result you aren't sure about, and never release a key you aren't sure about. When in doubt, the record should say so.

The gap between the record and the side effect

Everything so far assumes the idempotency record and the work it protects succeed or fail together. Sometimes they can.

When the work is a local database write, put both in one transaction. Insert the order, update the idempotency record to completed with the response, commit. Either both happen or neither does. A crash before the commit leaves no trace, and a retry simply starts over. This is the easy case, and it is why the record usually belongs in the same database as the business data.

When the work leaves your database, no transaction can span it. Charging a card through a provider, submitting a blockchain transaction, sending an email: these can't be rolled back by your commit. A crash after the external call but before the commit leaves the worst possible state. The side effect happened, and your record has no idea.

There is no single trick that closes this gap, but a few techniques narrow it:

• Pass a key downstream. If the provider supports idempotency keys, send one derived from yours. A retry of your request then becomes a retry of theirs, and they deduplicate it. This is the most effective fix when it's available.
• Record intent before acting, and check before acting again. Save what you're about to do and any external reference you get back. When a retry takes over a stalled record, query the external system for that reference before doing anything. This is the pattern from Part 1's escrow example: check the stored transaction hash instead of submitting again.
• Split the work into recovery points. Break a long operation into phases, each committed locally with the record noting the last finished phase. A retry resumes from there instead of the beginning. Brandur Leach's article on implementing Stripe-like idempotency keys in Postgres is the best public walkthrough of this approach.
• Move secondary side effects out of the request. Emails, notifications, and events don't need to happen inside the request. Write them to a table in the same transaction and deliver them afterward. That's the Transactional Outbox pattern, which gets its own part later in this series.

Database or Redis?

Redis is the tempting choice. It's fast, atomic claims are one command, and expiry is built in. But the previous section changes the question. It isn't "where is the fastest place to store keys?" It's "can the record commit together with the work?"

Use the main database when the protected work writes to that database, which is most of the time. The record and the business write share one transaction, and the easy case from the previous section stays easy. You also inherit the database's durability, backups, and replication without extra thought.

Redis fits when the work doesn't touch your primary database, or as a fast first gate in front of a database-backed record that remains the source of truth. Used alone, it brings two risks. Its writes can't join your database transaction, so the record and the work can disagree after a crash. And depending on how persistence and failover are configured, a key written just before a crash or failover can disappear, which turns the next retry into a fresh request.

A reasonable default: start with the database. Add Redis when measurements say you need it, not before.

How long to keep keys

Keys can't live forever, and they can't expire too early. The right retention comes from one comparison: the record must outlive the longest window in which a client might still retry.

If a mobile app queues failed payments and retries them when the phone is back online the next morning, a one-hour retention means that retry arrives after the key is gone. The server treats it as new work. You've reintroduced the double charge, just with a delay.

Stripe's documentation describes keys becoming eligible for removal once they're at least 24 hours old, and says a key reused after pruning is treated as a new request. That's a sensible baseline for synchronous APIs. Webhooks and queued work often need longer, because their retry schedules can stretch over days.

The contract runs both ways. The server documents how long keys are kept, and clients must not retry an intent past that window with the same key. If they need to try again after that, it's a new attempt, and it deserves a deliberate decision rather than a silent replay.

Common mistakes

If you're reviewing an idempotency implementation, these are the failures worth checking for:

• A retry helper generates a new key on every attempt.
• Keys are unique globally instead of per account or tenant.
• Only the key is checked, never the request fingerprint.
• The fingerprint includes fields that change between retries, so every retry is rejected.
• The lookup and the claim are separate steps, leaving a race between them.
• In-progress records have no timeout, so one crash blocks a key permanently.
• The record lives in a different store than the business data, with no plan for when they disagree.
• External calls don't pass a key downstream, even when the provider supports it.
• Keys expire before the client's retry window ends.
• The replayed response comes from a fresh read of the resource instead of the stored response, so it changes as the resource changes.

What comes next

An idempotency key is a small header with a large contract behind it. The client promises one key per intent. The server promises to claim it atomically, check that the request matches, replay what it already did, and keep the record long enough to matter.

Most of the difficulty isn't in the key. It's in the moments between steps: two requests racing for the same claim, a crash between the work and the record, a side effect your database can't roll back.

Everything here assumed a client waiting on an HTTP response. Part 3 turns to the other half of most systems: message consumers. Brokers like Kafka, RabbitMQ, and SQS redeliver by design, nobody is waiting on a response, and the Inbox pattern takes the place of the idempotency key.

How does your team handle keys that get stuck in progress? I'd like to hear what's worked for you.