A user taps “Pay” twice. The mobile app retries on timeout. A worker crashes after writing to the database but before sending the acknowledgment. Three different scenarios, one problem: the same action happens multiple times.

Without idempotency, that payment gets processed twice. The booking gets created twice. The notification gets sent twice.

Idempotency fixes this. An idempotent API means calling it multiple times with the same input produces the same result as calling it once. It’s safe to retry.

This isn’t theoretical. Payment APIs require it. Order systems require it. Mobile apps with flaky networks require it. Idempotency is a core requirement for any write operation in distributed systems.

Let’s build idempotent APIs that actually work.

Why Idempotent APIs Matter

Here’s what happens without idempotency.

A user creates an order. The request hits your API gateway. The gateway forwards it to your order service. The service writes to the database. Then the network times out before the response reaches the gateway.

The client never got a response, so it retries. The gateway sees a new request. The service processes it again. Now you have two orders for the same cart.

Or worse: the payment service charges the card twice. The booking system creates two reservations. The notification service sends duplicate emails.

These aren’t edge cases. They happen constantly in production:

• Mobile apps retry automatically on network errors
• Load balancers retry failed requests
• Message brokers deliver messages at least once
• Backend services restart and reprocess queues

You can’t prevent retries. You can only make your APIs safe to retry.

What Idempotency Actually Means

Idempotency means: same call, same effect, safe to retry.

If you call an idempotent API with the same input multiple times, the result is identical to calling it once. The first call creates the resource. Subsequent calls return the same result without creating duplicates.

This is different from idempotency in math. In math, f(f(x)) = f(x). In APIs, it means “same request, same response, no side effects on retry.”

Scope: Write Operations

We’re focusing on write operations: POST, PUT, PATCH. These change state. GET requests are already idempotent (they don’t change state). DELETE can be idempotent (deleting something twice is the same as deleting it once).

The hard part is POST requests that create resources. Creating an order twice is bad. Creating a payment twice is worse.

Failure Modes You Need to Design For

Here are the scenarios that break non-idempotent APIs.

Client Retries

Clients retry for many reasons:

• Network timeouts (client never got a response)
• 5xx errors (server error, maybe it’ll work next time)
• User impatience (tapped the button twice)
• Automatic retry logic (HTTP client libraries retry by default)

When the client retries, it sends the same request again. Without idempotency, your service processes it as a new request.

Network Timeouts and Unknown Outcomes

The request reaches your service. Your service writes to the database. Then the network connection drops before the response is sent.

From the client’s perspective, the request failed. It has no idea whether the operation succeeded or not. It retries.

From your service’s perspective, the operation succeeded. The database has the record. But the client doesn’t know that.

This is the “unknown outcome” problem. The client can’t tell if the operation happened or not.

At-Least-Once Delivery

Message brokers (Kafka, RabbitMQ, SQS) guarantee at-least-once delivery. They might deliver the same message twice. They might deliver it out of order.

If your consumer isn’t idempotent, it processes the same message multiple times.

Backend Restarts and Partial Failures

Your service processes a request. It writes to the database. Then it crashes before sending the response.

The service restarts. The request is still in the queue (or the client retries). The service processes it again. But the database already has the record from the first attempt.

Or worse: the service writes to the database, then fails to send a notification. On retry, it writes to the database again (duplicate) and sends the notification.

Timeline Example

Here’s what happens when the same “create order” request hits your backend three times:

Time 0ms: Client sends request with idempotency key abc123 Time 50ms: Request reaches API gateway Time 100ms: Gateway forwards to order service Time 200ms: Order service writes order to database (order_id: 42) Time 250ms: Network timeout - response never reaches gateway Time 300ms: Client retries with same idempotency key abc123 Time 350ms: Gateway checks idempotency store, finds abc123 already processed Time 360ms: Gateway returns cached response (order_id: 42)

Without idempotency, the second request would create order_id: 43. With idempotency, it returns order_id: 42.

Idempotency Keys: Core Design Pattern

The idempotency key is a unique identifier for a logical action. The client generates it and sends it with the request. The server uses it to detect duplicates.

Who Generates the Key?

The client generates the key. The server validates it and uses it to track requests.

Why client-generated? Because the client knows what action it’s trying to perform. The server doesn’t know if two requests represent the same user action or different actions.

Example: A user wants to pay $100. The client generates key pay-2025-12-10-user123-100. If the request fails and retries, the client sends the same key. The server recognizes it as the same payment attempt.

Key Requirements

The key must be:

• Unique per action: Different actions get different keys
• Stable across retries: Same action, same key
• Scoped appropriately: Per endpoint or per semantic action

Good keys:

• create-order-{user_id}-{cart_id}-{timestamp}
• charge-payment-{user_id}-{amount}-{timestamp}
• book-reservation-{user_id}-{hotel_id}-{checkin_date}

Bad keys:

• Random UUIDs (different on each retry)
• Timestamps only (collisions possible)
• User ID only (same user, different actions)

Storage Strategy

Store idempotency keys in a table or cache:

CREATE TABLE idempotency_keys (
  idempotency_key VARCHAR(255) PRIMARY KEY,
  status VARCHAR(50) NOT NULL, -- 'processing', 'completed', 'failed'
  response_hash VARCHAR(64), -- Hash of response body
  response_body TEXT, -- Full response (or reference to it)
  user_id VARCHAR(255), -- Scope to user
  endpoint VARCHAR(255), -- Which endpoint
  created_at TIMESTAMP NOT NULL,
  expires_at TIMESTAMP NOT NULL,
  INDEX idx_user_endpoint (user_id, endpoint),
  INDEX idx_expires_at (expires_at)
);

The table stores:

• idempotency_key: The key itself (primary key)
• status: Whether the request is processing, completed, or failed
• response_hash: Hash of the response (to detect if request changed)
• response_body: The actual response to return on retry
• user_id: Scope keys to users (optional but recommended)
• endpoint: Which endpoint this key is for
• created_at/expires_at: TTL for cleanup

Common Mistakes

Mistake 1: Key per request instead of per logical action

If the client generates a new key for each HTTP request, retries get new keys. Idempotency breaks.

Fix: Generate the key based on the logical action, not the HTTP request.

Mistake 2: No TTL and table grows forever

Idempotency keys accumulate. After a year, you have millions of keys. Queries slow down. Storage costs increase.

Fix: Set TTL (e.g., 24 hours). Delete expired keys periodically.

Mistake 3: Not including user/account scope

If two users generate the same key (unlikely but possible), they collide. User A’s retry returns User B’s response.

Fix: Include user_id in the key or scope the lookup to user_id.

End-to-End Flow: From Gateway to DB

Here’s how idempotency works end-to-end.

Step 1: API Gateway Receives Request

The client sends a request with Idempotency-Key header:

POST /api/orders
Idempotency-Key: create-order-user123-cart456-20251210
Content-Type: application/json

{
  "cart_id": "cart456",
  "items": [...]
}

The gateway extracts the key and checks the idempotency store.

Step 2: Check if Key Exists

The gateway queries the idempotency table:

SELECT status, response_body, response_hash
FROM idempotency_keys
WHERE idempotency_key = 'create-order-user123-cart456-20251210'
  AND expires_at > NOW();

If the key exists and status is completed, return the cached response immediately. No need to hit the backend.

If the key exists and status is processing, return 409 Conflict or wait (depending on your design).

If the key doesn’t exist, proceed to step 3.

Step 3: Create Processing Record

Before forwarding to the backend, create a “processing” record:

INSERT INTO idempotency_keys (
  idempotency_key,
  status,
  user_id,
  endpoint,
  created_at,
  expires_at
) VALUES (
  'create-order-user123-cart456-20251210',
  'processing',
  'user123',
  '/api/orders',
  NOW(),
  NOW() + INTERVAL 24 HOUR
);

This uses a unique constraint on idempotency_key. If two requests arrive simultaneously, one will fail the insert. That request knows another is processing and can wait or return 409.

Step 4: Forward to Backend

Forward the request to the order service. The service processes it normally, unaware of idempotency (or aware, depending on your architecture).

Step 5: Write to Database in Transaction

The order service writes to the database in a transaction:

async function createOrder(orderData: OrderData, idempotencyKey: string) {
  return await db.transaction(async (tx) => {
    // Check if order already exists (defense in depth)
    const existing = await tx.orders.findOne({
      where: { idempotency_key: idempotencyKey }
    });
    
    if (existing) {
      return existing; // Already created, return it
    }
    
    // Create new order
    const order = await tx.orders.create({
      ...orderData,
      idempotency_key: idempotencyKey,
      status: 'pending'
    });
    
    return order;
  });
}

The transaction ensures atomicity. Either the order is created or it isn’t. No partial state.

Step 6: Update Idempotency Record

On success, update the idempotency record with the response:

UPDATE idempotency_keys
SET status = 'completed',
    response_body = '{"order_id": 42, "status": "pending"}',
    response_hash = SHA256('{"order_id": 42, "status": "pending"}')
WHERE idempotency_key = 'create-order-user123-cart456-20251210';

On failure, mark it as failed:

UPDATE idempotency_keys
SET status = 'failed',
    response_body = '{"error": "Insufficient inventory"}'
WHERE idempotency_key = 'create-order-user123-cart456-20251210';

Step 7: Return Response

Return the response to the client. Subsequent retries with the same key will return the cached response from step 2.

Exactly-Once vs At-Least-Once Semantics

Idempotency gives you at-least-once semantics with exactly-once effects.

The request might be processed multiple times (at-least-once delivery). But the effect happens exactly once (exactly-once semantics).

This is different from true exactly-once processing, which is harder and often unnecessary. For most use cases, at-least-once delivery with exactly-once effects is enough.

Data Model and Storage Options

You have several options for storing idempotency keys.

Relational Database (PostgreSQL, MySQL)

Pros:

• ACID guarantees (atomic inserts, transactions)
• Unique constraints prevent race conditions
• Durable (survives restarts)
• Easy to query and debug

Cons:

• Higher latency than cache
• More expensive at scale
• Requires connection pooling

When to use: When you need strong consistency and durability. Good for payment systems, order systems.

Schema:

CREATE TABLE idempotency_keys (
  idempotency_key VARCHAR(255) PRIMARY KEY,
  status VARCHAR(50) NOT NULL,
  response_hash VARCHAR(64),
  response_body JSONB, -- or TEXT
  user_id VARCHAR(255),
  endpoint VARCHAR(255),
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  expires_at TIMESTAMP NOT NULL,
  INDEX idx_user_endpoint (user_id, endpoint),
  INDEX idx_expires_at (expires_at)
);

-- Cleanup expired keys (run periodically)
DELETE FROM idempotency_keys WHERE expires_at < NOW();

Redis (In-Memory Cache)

Pros:

• Low latency (sub-millisecond)
• Built-in TTL (automatic expiration)
• High throughput
• Cost-effective for high volume

Cons:

• Not durable by default (can lose data on restart)
• No unique constraints (need application-level locking)
• Memory limits

When to use: When you need low latency and can tolerate some data loss. Good for high-volume APIs, short TTL keys.

Implementation:

async function checkIdempotency(key: string): Promise<CachedResponse | null> {
  const cached = await redis.get(`idempotency:${key}`);
  if (cached) {
    return JSON.parse(cached);
  }
  return null;
}

async function storeIdempotency(
  key: string,
  response: any,
  ttlSeconds: number = 86400
): Promise<void> {
  await redis.setex(
    `idempotency:${key}`,
    ttlSeconds,
    JSON.stringify(response)
  );
}

Hybrid Approach

Use Redis for hot keys (recent requests), PostgreSQL for cold keys (older requests, audit trail).

Or use Redis for short TTL (1 hour), PostgreSQL for longer TTL (24 hours).

Multi-Region Considerations

If your API is multi-region, you need to decide:

• Local cache only: Each region has its own idempotency store. Keys might collide across regions (unlikely but possible).
• Global store: Single idempotency store (Redis Cluster, DynamoDB Global Tables). Higher latency but true global idempotency.
• Regional with replication: Each region has a store, but they replicate. More complex but balances latency and consistency.

For most APIs, regional stores are fine. The chance of the same user retrying from different regions simultaneously is low.

Idempotency with Queues and Event-Driven Flows

Idempotency isn’t just for HTTP APIs. It’s also needed in queue consumers and event-driven systems.

Producer Side

When producing messages, include the idempotency key as a message attribute or header:

async function publishOrderEvent(order: Order) {
  await kafka.producer.send({
    topic: 'orders',
    messages: [{
      key: order.idempotency_key, // Use idempotency key as message key
      headers: {
        'idempotency-key': order.idempotency_key,
        'event-type': 'order.created'
      },
      value: JSON.stringify(order)
    }]
  });
}

Using the idempotency key as the Kafka message key ensures messages with the same key go to the same partition. This helps with ordering and deduplication.

Consumer Side

The consumer checks if it has already processed this message:

async function processOrderEvent(message: KafkaMessage) {
  const idempotencyKey = message.headers['idempotency-key'];
  
  // Check if already processed
  const processed = await db.processedMessages.findOne({
    where: { idempotency_key: idempotencyKey }
  });
  
  if (processed) {
    console.log(`Already processed: ${idempotencyKey}`);
    return; // Skip duplicate
  }
  
  // Process the message
  await processOrder(JSON.parse(message.value));
  
  // Mark as processed
  await db.processedMessages.create({
    idempotency_key: idempotencyKey,
    processed_at: new Date()
  });
}

Outbox Pattern with Idempotent Consumer

The outbox pattern ensures exactly-once delivery from your database to the message broker:

1. Write to database (including outbox table) in a transaction
2. Poll outbox table for new messages
3. Publish to message broker
4. Mark outbox message as published

The consumer still needs idempotency because the message might be delivered twice:

async function consumeFromOutbox() {
  const messages = await db.outbox.findUnpublished();
  
  for (const message of messages) {
    try {
      // Publish to Kafka
      await kafka.producer.send({
        topic: message.topic,
        messages: [{
          key: message.idempotency_key,
          value: message.payload
        }]
      });
      
      // Mark as published
      await db.outbox.markAsPublished(message.id);
    } catch (error) {
      // Retry later
    }
  }
}

Handling Reordered Events

If events arrive out of order, idempotency still works. Each event has its own idempotency key. Processing event B before event A is fine if they’re independent.

If events are dependent (event B requires event A to complete first), you need ordering guarantees (single partition, sequence numbers) in addition to idempotency.

Security and Abuse Considerations

Idempotency keys can be abused if not designed carefully.

Don’t Expose Internal IDs

Don’t use internal database IDs as idempotency keys. If a user guesses another user’s key, they might see that user’s response.

Example of bad key: order-{order_id} where order_id is a sequential integer. User A can try order-42, order-43 to see other users’ orders.

Fix: Include user context in the key or scope lookups to the authenticated user.

Limit TTL

Set reasonable TTLs. 24 hours is common. Longer TTLs increase storage costs and security risk (keys valid longer).

Shorter TTLs (1 hour) work for most APIs. Users rarely retry after an hour.

Enforce Key Reuse Policies

Prevent users from reusing keys for different requests. If a user sends the same key with different request bodies, reject it.

async function validateIdempotencyKey(
  key: string,
  requestHash: string
): Promise<boolean> {
  const existing = await db.idempotencyKeys.findOne({
    where: { idempotency_key: key }
  });
  
  if (!existing) {
    return true; // New key, valid
  }
  
  if (existing.request_hash !== requestHash) {
    throw new Error('Idempotency key reused with different request');
  }
  
  return true; // Same request, valid
}

Rate Limiting

Rate limit idempotency endpoints to prevent abuse. But don’t count idempotent retries against rate limits (they’re not new requests).

Testing Idempotent APIs

Testing idempotency requires simulating retries and failures.

Unit Tests

Call the same handler multiple times with the same key:

describe('Idempotent Order Creation', () => {
  it('should return same order on retry', async () => {
    const key = 'test-key-123';
    const orderData = { cart_id: 'cart1', items: [...] };
    
    // First call
    const response1 = await createOrder(orderData, key);
    expect(response1.order_id).toBeDefined();
    
    // Retry with same key
    const response2 = await createOrder(orderData, key);
    expect(response2.order_id).toBe(response1.order_id);
    
    // Retry again
    const response3 = await createOrder(orderData, key);
    expect(response3.order_id).toBe(response1.order_id);
  });
});

Integration Tests

Introduce artificial timeouts and retries:

it('should handle timeout and retry', async () => {
  const key = 'test-key-456';
  
  // Simulate timeout after DB write
  const createOrderWithTimeout = async () => {
    const order = await db.orders.create({ ... });
    throw new Error('Network timeout'); // Simulate timeout
  };
  
  // First attempt (times out)
  await expect(createOrderWithTimeout()).rejects.toThrow();
  
  // Retry (should return existing order)
  const retryResponse = await createOrder(orderData, key);
  expect(retryResponse.order_id).toBeDefined();
});

Chaos Tests

Kill the worker after database write but before response:

1. Start order service
2. Send request
3. Kill service after DB write (before response)
4. Restart service
5. Retry request
6. Verify no duplicate order created

This tests the “partial failure” scenario.

Checklist: Making a Non-Idempotent API Idempotent

Here’s a step-by-step migration plan.

Step 1: Add Idempotency Key to API Contract

Update your API to accept Idempotency-Key header:

interface CreateOrderRequest {
  cart_id: string;
  items: OrderItem[];
  // Client must send Idempotency-Key header
}

Document it in your API docs. Tell clients they must send unique keys per logical action.

Step 2: Create Idempotency Storage

Choose your storage (PostgreSQL, Redis, or both). Create the table/schema.

Step 3: Add Gateway Middleware

Add middleware that:

1. Extracts Idempotency-Key header
2. Checks idempotency store
3. Returns cached response if found
4. Creates processing record if not found
5. Forwards to backend

Step 4: Update Backend Services

Update services to:

1. Accept idempotency key (from gateway or directly)
2. Check database for existing record with same key
3. Return existing record if found
4. Create new record with key if not found

Step 5: Add Response Caching

Store successful responses in idempotency store. Return them on retry.

Step 6: Add Cleanup Job

Create a job that deletes expired idempotency keys:

async function cleanupExpiredKeys() {
  await db.idempotencyKeys.deleteMany({
    where: {
      expires_at: {
        lt: new Date()
      }
    }
  });
}

// Run daily
setInterval(cleanupExpiredKeys, 24 * 60 * 60 * 1000);

Step 7: Roll Out Gradually

Don’t break existing clients. Make idempotency key optional initially:

const idempotencyKey = req.headers['idempotency-key'];

if (idempotencyKey) {
  // Use idempotency
  return handleWithIdempotency(req, idempotencyKey);
} else {
  // Legacy path (no idempotency)
  return handleLegacy(req);
}

Once all clients are updated, make the key required.

Step 8: Monitor and Alert

Monitor:

• Idempotency key usage (how many requests use keys)
• Cache hit rate (how many retries are served from cache)
• Duplicate detection (how many duplicates were prevented)

Alert on:

• High cache miss rate (clients generating new keys on retry)
• Storage growth (keys not expiring)
• Errors in idempotency logic

Summary and Practical Guidelines

Here are the rules to follow:

Always design writes to be safely retried. Assume every request will be retried. Design for it from the start.

Treat idempotency keys as first-class citizens. They’re not optional metadata. They’re core to API reliability.

Test with chaos, not just happy paths. Simulate timeouts, crashes, network failures. Verify idempotency holds.

Use the right storage for your needs. PostgreSQL for strong consistency. Redis for low latency. Hybrid for both.

Scope keys appropriately. Include user context. Include action context. Make them unique but stable.

Set reasonable TTLs. 24 hours is usually enough. Shorter for high-volume APIs. Longer for critical operations.

Monitor idempotency usage. Track cache hit rates. Track duplicate prevention. Alert on anomalies.

Document it clearly. Tell clients how to generate keys. Show examples. Make it easy to get right.

Idempotency isn’t optional. Users expect reliable APIs. Networks are flaky. Systems fail. The only way to build reliable systems is to assume everything will be retried and design for it.

Start with one endpoint. Add idempotency. Test it. Then roll it out to more endpoints. Once you have the pattern, it becomes routine.

Your users will thank you. Your support team will thank you. Your accounting team will definitely thank you.