← Back to Home

TOCTOU fix in Bankara (pg advisory locks)

The problem

Bankara is a P2P digital wallet I built with a C++ backend (Drogon framework) and PostgreSQL. The core feature is simple: users can send money to each other. But I kept running into a bug during stress testing where an account would end up with a negative balance, which should never happen.

Here's what was going on. Say an account has $100. Two transfer requests come in at almost the exact same time, both trying to send $100. Both requests read the balance, both see $100, both pass the balance >= amount check. Both go through. Now the account is at -$100. That's a classic TOCTOU — Time-of-Check to Time-of-Use — race condition. The balance was valid when I checked it, but by the time I actually used it, someone else already spent it.

What I tried first

The obvious fix is locking. PostgreSQL gives you a few options here, and I went through most of them before landing on something I was happy with.

My first attempt was SELECT ... FOR UPDATE, which grabs a row-level lock on the account row. It works — it serializes access to that specific row so only one transaction can touch it at a time. But under load, it caused problems. When a lot of requests hit the same popular account, they'd all queue up waiting for that row lock. The connection pool started filling up with threads just sitting there waiting, and eventually I'd get timeouts.

Table-level locks (LOCK TABLE) were even worse. That basically makes every single transfer sequential, even for completely unrelated accounts. Not an option.

The fix: advisory locks

I ended up using PostgreSQL advisory locks, specifically pg_advisory_xact_lock. These are application-level locks that live in shared memory — they don't actually lock any rows or tables. You just pick a number, and PostgreSQL gives you a lock on that number.

The way I set it up: before doing anything with an account, I hash the account ID into a 64-bit integer using hashtext(account_id) and call pg_advisory_xact_lock on that hash. This means only one transaction can operate on a given account at a time, but transactions touching different accounts don't block each other at all.

The nice part is that these locks are tied to the transaction. When the transaction commits or rolls back, the lock is automatically released. No manual cleanup, no risk of forgetting to unlock something.

Did it work?

I wrote a stress test that fires 1,000+ concurrent transfer requests at the same account. Before the fix, the account would reliably go negative. After, every single invalid transfer gets rejected correctly. Different accounts still process in parallel without blocking each other, so throughput stays fine for the normal case.

Advisory locks turned out to be a good middle ground — lighter than row locks under contention, way more granular than table locks, and the automatic cleanup on transaction end means I don't have to worry about leaked locks.