TOCTOU fix in Bankara (pg advisory locks)
The problem
Bankara is a P2P digital wallet I built with a C++ backend (Drogon framework) and PostgreSQL. The core feature is simple: users can send money to each other. But I kept running into a bug during stress testing where an account would end up with a negative balance, which should never happen.
Here's what was going on. Say an account has $100. Two transfer requests come in at almost the
exact same time, both trying to send $100. Both requests read the balance, both see $100, both
pass the balance >= amount check. Both go through. Now the account is at -$100.
That's a classic TOCTOU — Time-of-Check to Time-of-Use — race condition. The balance was valid
when I checked it, but by the time I actually used it, someone else already spent it.
What I tried first
The obvious fix is locking. PostgreSQL gives you a few options here, and I went through most of them before landing on something I was happy with.
My first attempt was SELECT ... FOR UPDATE, which grabs a row-level lock on the
account row. It works — it serializes access to that specific row so only one transaction can
touch it at a time. But under load, it caused problems. When a lot of requests hit the same
popular account, they'd all queue up waiting for that row lock. The connection pool started
filling up with threads just sitting there waiting, and eventually I'd get timeouts.
Table-level locks (LOCK TABLE) were even worse. That basically makes every single
transfer sequential, even for completely unrelated accounts. Not an option.
The fix: advisory locks
I ended up using PostgreSQL advisory locks, specifically pg_advisory_xact_lock.
These are application-level locks that live in shared memory — they don't actually lock any rows
or tables. You just pick a number, and PostgreSQL gives you a lock on that number.
The way I set it up: before doing anything with an account, I hash the account ID into a 64-bit
integer using hashtext(account_id) and call pg_advisory_xact_lock on
that hash. This means only one transaction can operate on a given account at a time, but
transactions touching different accounts don't block each other at all.
The nice part is that these locks are tied to the transaction. When the transaction commits or rolls back, the lock is automatically released. No manual cleanup, no risk of forgetting to unlock something.
Did it work?
I wrote a stress test that fires 1,000+ concurrent transfer requests at the same account. Before the fix, the account would reliably go negative. After, every single invalid transfer gets rejected correctly. Different accounts still process in parallel without blocking each other, so throughput stays fine for the normal case.
Advisory locks turned out to be a good middle ground — lighter than row locks under contention, way more granular than table locks, and the automatic cleanup on transaction end means I don't have to worry about leaked locks.