Post-incident review · Checkout · Northwind Ltd

Incident 2026-09-12 — checkout timeouts

SEV-2 · resolved Window 14:02 – 15:37 HKT Duration 95 min Owner Payments team Written 2026-09-13

For 95 minutes on Friday afternoon, roughly one in three checkouts on the web store timed out at the payment step. A routine deploy of the payments service shipped with a connection pool one quarter of its previous size; under Friday traffic the pool filled within minutes and every request behind it waited for a connection that never came. Rolling the deploy back restored the service. No payment was charged twice, but 412 orders were abandoned before completion.

Impact

1,214checkouts timed out at the payment step

412orders abandoned, not retried within 24 h

38 sp99 checkout latency at the peak (normal: 1.4 s)

0duplicate charges; the gateway idempotency keys held

Timeline

Time (HKT)	What happened
13:51	payments-service v2.41.0 deployed to production. The release notes mention the config refactor; nothing flags the pool size.
14:02	Checkout p99 latency alert fires (threshold 8 s). On-call acknowledges within 3 minutes.
14:09	Gateway dashboard shows healthy; the first hypothesis is a downstream slowdown at the card processor. Support ticket opened with the processor.
14:31	Second on-call joins. Pool metrics show `pool.active = 20` flat and `pool.waiting` climbing since 13:55.
14:48	Config diff against v2.40.3 finds `PAYMENTS_POOL_MAX` read from the new config key, which defaults to 20 when unset. The old key held 80.
15:12	Rollback to v2.40.3 started. Slower than usual because the rollback runbook still pointed at the old cluster name.
15:37	Latency back under 2 s. Alert clears. Incident closed at 16:05 after 30 minutes of clean metrics.

Where the request stalled

01BrowserPOST /checkout/confirm

02Checkout APIValidates the cart, calls payments

03Payments pool20 connections, all busy; requests queue here

04Card processorHealthy the whole time, 300 ms

05Order writtenOnly after the payment returns

Fig. 1 — The checkout path. The wait was entirely inside step 3; the processor we suspected first was never slow.

Root cause

v2.41.0 moved service settings from environment variables to a typed config file. The pool size was renamed from PAYMENTS_POOL_MAX to payments.pool.max, and the new loader falls back to its own default of 20 when the key is absent. Production still set the old variable, so the new build silently ran with a quarter of the pool it had on Thursday. Staging never showed it: staging traffic never needs more than a dozen connections.

- PAYMENTS_POOL_MAX=80            # env, read by v2.40.x
+ payments:
+   pool:
+     max: 80                    # config.yaml, read by v2.41.x

Two things made a 20-minute problem into a 95-minute one. The first hypothesis pointed outward at the card processor, and the pool metrics that would have settled it in a minute were on a dashboard nobody had bookmarked. Then the rollback runbook was stale, which cost another 25 minutes.

What changed since

The config loader now refuses to start when a required key is missing instead of applying a default. Pool size is required.
Pool saturation is its own alert: pool.waiting > 0 for 60 seconds pages the on-call, with the pool dashboard linked from the alert.
Checkout API applies a 5-second budget to the payments call and returns a retryable error instead of holding the browser for 38 seconds.
The rollback runbook was corrected and is now exercised on every release by the release checklist.

Action items

Action	Owner	Due	State
Required-key validation in the config loader	M. Leung	2026-09-15	done
Pool saturation alert with dashboard link	R. Osei	2026-09-16	done
Timeout budget and retryable error on the payments call	M. Leung	2026-09-19	done
Load test at 3× Friday peak before every payments release	R. Osei	2026-09-30	open
Reach out to the 412 customers whose orders were abandoned	Support	2026-09-17	done