August 21, 20262 min

The Flag Meant 'Done', But 'Done' Hadn't Happened Yet

A reward system marked an entity as 'rewarded' before attempting the reward. When the attempt legitimately declined, the flag stuck — permanently locking out every future legitimate attempt, with no error anywhere.

System Design · Correctness · Fintech

TL;DR

  • A referral-style reward system wrote a "reward granted" flag, then attempted the grant. That ordering assumed the grant would succeed.
  • It didn't always succeed — a legitimate case existed where there was no eligible tier to grant against — and when it declined, the flag was already permanently set.
  • The entity was now marked "rewarded forever" with no reward ever paid, and no future eligible attempt could retry, because the flag itself was the only thing gating retries.

The bug wasn't a race condition and it wasn't a crash. It was an ordering assumption baked into two lines:

ts
await markRewarded(userId);        // record the outcome
await grantReferralReward(userId); // …then produce it

Read quickly, those two lines look like they belong in either order. They don't. The first line is a claim about something the second line hasn't done yet.

The case that broke it

The grant could legitimately decline — not from an error, but from a real business rule: no eligible offer tier existed for that entity at that moment. A decline is a valid outcome, not a bug. The bug was entirely in what happened around the decline.

The flag is written before the outcome is known, so a legitimate decline is recorded as a completed reward.

Nothing downstream could tell the difference between "rewarded and paid" and "marked rewarded, paid nothing." Both look identical in the data.

Genuinely rewardedDeclined, then flagged
rewarded columntruetrue
Payout recordexistsnone
Error loggednone
Future attemptscorrectly skippedwrongly skipped, forever

Because the flag was also the retry gate, a single declined attempt closed the door on every future attempt — including ones that would have been eligible.

Why nobody noticed immediately

This isn't a failure mode that throws. The write succeeded, the decline was a normal return value, and the system moved on. The only visible symptom was a report, much later, that eligible users weren't receiving rewards they should have qualified for — a business observation, not an engineering alert.

The fix

Two changes, not one.

Reorder, so the flag records an outcome rather than predicting one:

ts
1const result = await grantReferralReward(userId);
2if (result.granted) {
3  await markRewarded(userId);   // only now is it true
4}
5// declined → nothing written → still retryable
6

Generalise the trigger, so the flag fires off whichever of the two required facts lands second, rather than being written speculatively ahead of either:

The completion flag is derived from both facts being present, not written ahead of them.

The two facts could arrive in either sequence depending on the caller, so keying off "whichever lands second" closes the same class of bug for the other ordering too.

The transferable part

A completion flag is a promise about something that already happened. The moment it gets written before the thing it describes, it stops being a record and becomes a guess — and a guess that also gates retries turns one declined attempt into a permanent lockout.

If a flag both records success and blocks future attempts, write it only after the outcome it claims to describe is actually known. And be suspicious of any decline path that returns normally: those are the ones that quietly walk past everything you built to catch failures.

Share this note

Comments

responses

0/2000

Loading comments…