August 21, 20262 min

Checking Idempotency Against the Wrong Clock

A duplicate-transfer guard was redesigned to key off immutable inputs, evaluated before any resolution of current state — because checking a retry against current balance and eligibility could wrongly reject a transfer that had already succeeded.

System Design · Concurrency · Fintech

TL;DR

  • A transfer system needed to decide whether a retried request was "the same" as one already processed.
  • The instinct — re-check the request against current recipient and balance state — is wrong, because a retry of an already-successful transfer can look ineligible against state its own first attempt changed.
  • The fix keyed idempotency off the request's immutable inputs as of submission time, and separately widened the exclusivity lock to span the whole resolve-and-settle window rather than just the dedup check.

The obvious way to check "have we already done this?" is to re-run the eligibility logic against current state and see whether the answer is still yes:

kotlin
// Tempting, and wrong for a retry.
val recipient = resolveRecipient(req.recipientId)      // current state
if (!recipient.isEligible()) reject(INELIGIBLE)
if (balanceOf(req.sender) < req.amount) reject(INSUFFICIENT)

For a transfer, that's backwards in one specific case: the first successful attempt changes the balance, and possibly the recipient's eligibility for further transfers, as a direct consequence of succeeding.

The failure this avoids

The retry is judged against state the first attempt already changed, so a successful transfer reports as ineligible.

Rejecting the retry as ineligible is worse than either accepting it or correctly deduplicating it: it tells the caller something failed when in fact their money already moved.

Retry evaluated against…Duplicate correctly detected?Risk
Current balance / eligibilityno — state already movedreports failure for a completed transfer
Submission-time immutable inputsyesnone for this case

Why the fix keys off submission-time inputs

The transfer's immutable inputs — sender, recipient, amount, client reference — describe what was requested, independent of how the world has changed since:

kotlin
1// Idempotency decided before any current-state resolution.
2val fingerprint = Fingerprint(req.senderId, req.recipientId, req.amount, req.clientRef)
3
4existingByFingerprint(fingerprint)?.let { prior ->
5    return prior.result          // replay the original outcome, don't re-judge
6}
7
8// Only now does current state matter — for doing the work, not for dedup.
9val recipient = resolveRecipient(req.recipientId)
10

Current state is still used for the actual transfer logic. It's just no longer part of the duplicate-detection decision.

The lock has to cover the whole window

A related fix: the exclusivity lock preventing two concurrent attempts must be held from before the transactional boundary through the entire resolve-and-settle sequence — not released after the dedup check and re-acquired later.

A lock covering only the dedup check leaves a window where two retries both pass before either commits.

The transferable part

Idempotency logic that reads current state to answer "have I seen this before?" is vulnerable whenever the operation being deduplicated is the thing that changes the state it's checked against. That's most of them, in a payments system.

The safer design evaluates duplicate-ness against the request's own fixed inputs and replays the recorded outcome, treating current state as relevant only to performing the operation — never to deciding whether it already happened.

Share this note

Comments

responses

0/2000

Loading comments…