# Incident postmortem: checkout latency in eu-west (INC-2417)

| Field    | Detail                                                |
| -------- | ----------------------------------------------------- |
| Severity | SEV-2 — degraded customer-facing checkout             |
| Duration | 94 minutes (09:47–11:21 UTC), 14 September 2026       |
| Impact   | 3.4% of checkout attempts failed; p95 240 ms → 11.2 s |
| Status   | Resolved — 4 of 6 remediation items complete          |

On 14 September 2026, checkout requests served from the `eu-west` region slowed from a 240 ms p95 to 11.2 s p95 for 94 minutes, and 3.4% of attempts failed outright. Every failed authorisation was retried automatically or surfaced to the customer with a clear error, so no payment was lost and no card data was exposed.

## How it unfolded

```mermaid
sequenceDiagram
    autonumber
    participant Edge as Edge / CDN
    participant Checkout as Checkout API
    participant Cache as Session cache
    participant Payments as Payments service

    Edge->>Checkout: POST /checkout (eu-west)
    Checkout->>Cache: GET session:{id}
    Cache--xCheckout: pool exhausted, no reply in 250 ms
    Checkout->>Cache: immediate retry, no backoff
    Note over Checkout,Cache: retries saturate the pool
    Checkout->>Payments: authorise (queued 8-14 s)
    Checkout-->>Edge: 504 for 3.4% of requests
```

> [!WARNING]
> The retry helper had `backoff: "none"` because a flat retry loop passed review as "simpler than the jittered one". Review guidance for shared connection pools now requires jittered backoff with a retry budget.

## Timeline (UTC)

```mermaid
timeline
    title INC-2417 key events
    09:41 : Pool config pushed to eu-west
    09:47 : p95 latency alert fires (> 2 s)
    09:52 : Incident declared, SEV-2
    10:03 : Retry storm identified in cache metrics
    10:19 : Config rolled back to previous pool size
    10:34 : Error rate back below 0.5%
    11:21 : Incident closed after 45 minutes stable
```

## Root cause

The session cache pool was reduced from 128 to 32 connections to free capacity for a batch job. At peak checkout volume the smaller pool drained in under a second, and the retry helper re-issued every failed `GET` immediately and without jitter, so retries consumed the pool faster than it could recover.

```ts
// config/session-cache.ts — what shipped
connectionPool: {
  max: 32,                    // reduced from 128 for the batch job
  acquireTimeoutMs: 250,
  retry: { attempts: 3, backoff: "none" },
},

// config/session-cache.ts — fixed
connectionPool: {
  max: 128,
  acquireTimeoutMs: 250,
  retry: { attempts: 3, backoff: "jittered", budgetPerMinute: 500 },
},
```

No dashboard tracked pool wait time, so the failure mode was invisible until latency alerts fired six minutes later.

## Action items

| #   | Action                                                          | Owner          | Due    |
| --- | --------------------------------------------------------------- | -------------- | ------ |
| 1   | Restore pool size and add jittered retry budget                 | Payments Plat. | Done   |
| 2   | Dashboard pool wait time, queue depth and retry rate            | Observability  | Done   |
| 3   | Alert on pool saturation at 80% for 2 minutes                   | Observability  | Done   |
| 4   | Load-test config changes to shared pools at peak volume         | Payments Plat. | 30 Sep |
| 5   | Enforce jittered backoff for all pooled clients in review guide | Engineering    | 10 Oct |
| 6   | Add config-freeze check to the maintenance-window runbook       | SRE            | 17 Oct |