Idempotency
An idempotent operation produces the same result whether executed once or many times. In distributed systems, idempotency enables safe retries without side effects.
Idempotency = an elevator button. Pressing it 10 times doesn't call 10 elevators. The first press registers; the rest are no-ops. Your API should work the same way.
Idempotency is essential when using at-least-once delivery: the same message may be delivered multiple times and must be processed safely. Implementation: (1) Natural idempotency — some operations are inherently idempotent (set a value, put a record by primary key). (2) Idempotency keys — client generates a unique key per request; server stores processed keys and returns cached response on duplicate. (3) Conditional writes — use a version number or ETag; the write only succeeds if the current version matches the expected version.
Idempotency key storage: the server must persist idempotency keys durably and check them atomically with the operation. A race condition exists if two identical requests arrive simultaneously — use a DB unique constraint on the idempotency key and handle the uniqueness violation as a duplicate. The idempotency key must be scoped appropriately: per-user + per-operation to prevent one user's key from blocking another's. TTL for idempotency keys: keys should expire after a window long enough to cover all reasonable retry windows (24 hours for async operations, 30 days for financial transactions). Stripe's idempotency implementation stores the full request and response: on duplicate, returns the stored response without re-executing. This handles cases where the original request succeeded but the response was lost.
For any write operation that crosses a network boundary, I design for idempotency. For API endpoints, I require clients to send an idempotency key (UUID) in a header. The server stores (idempotency_key, user_id, response) in a DB table with a unique constraint. On duplicate, return the stored response. For consumer-side idempotency, I store processed message IDs in a Redis set with TTL matching the message retention period. Before processing, I check for the message ID — skip if already processed, else process and add to the set atomically using a Redis transaction.
Implementing idempotency checks outside of a transaction with the actual operation. If the check and write are not atomic (e.g., check in Redis, write in PostgreSQL), a crash between the two leaves the system in a state where the operation completed but the idempotency key was not recorded — allowing reprocessing.