TL;DR
- A production payment system failed intermittently at peak hours. The instinct — "connection pool is too small" — was checked against JMX and turned out to be wrong twice: the pool wasn't full, and even a bigger pool wouldn't have fixed it.
- The actual defect was connections leaving the pool in a bad state: never returned, or returned still inside an open transaction from a request that threw before reaching its
finally. - The fix was leak detection plus explicit cleanup at every exit path, not a larger
maximumPoolSize— a bigger pool of the same leaking connections just delays the same failure by a wider margin.
The symptom was specific: transaction failures clustered at peak hours, and cleared up on their own once traffic dropped. That shape usually means resource exhaustion, and the first hypothesis was the obvious one — the connection pool is undersized for peak concurrency, requests queue for a connection, some of them time out.
What JMX actually showed
HikariCP exposes its pool state through JMX MBeans — active connections, idle connections, threads waiting, total created. Watching those numbers live during a peak window ruled out the first hypothesis fast: the pool wasn't maxed out. Active connections sat well under maximumPoolSize, and there were idle connections sitting in the pool the whole time failures were happening.
That's the tell that the problem isn't count, it's state. A pool with idle capacity that still can't serve a request is handing out connections that are unusable the moment a caller gets one — already inside a transaction from whoever held it last, or already invalid.
Where the leak actually was
The legacy request path acquired a connection at the top of a servlet method and released it at the bottom, by hand — no try-with-resources, no framework-managed transaction boundary. That pattern is correct exactly as long as every exit path reaches the release call. It wasn't: a validation exception thrown partway through the method skipped straight to the servlet's error handler, and the connection it was holding never got returned to the pool. Under low traffic that's a slow leak nobody notices for a long time. Under peak traffic, with more requests hitting that validation path per minute, the pool's usable fraction shrinks fast enough to produce clustered failures inside a single peak window and then quietly recover once traffic — and the leak rate — drops.
The second-order problem was worse: on some paths, a connection that leaked out of a request came back to the pool later still inside an uncommitted transaction from a different request's try block that had also thrown early. The next caller to acquire that connection inherited a live transaction it never opened.
The fix, in order
- HikariCP's leak detection threshold (
leakDetectionThreshold) turned on first, in a lower environment — it logs a stack trace for any connection held longer than the threshold, which is what actually located the specificcatchblocks that returned early without releasing. try/finallyaround every acquisition, not just the happy path, so the release call runs regardless of which exit the method takes.- Explicit rollback on the finally path before returning the connection, so a connection that leaves a failed request never re-enters the pool mid-transaction.
Sizing maximumPoolSize up was deliberately not the fix, even though it would have made peak-hour failures less frequent — it dilutes the leak rate without closing the leak, and it costs more DB-side resources to buy the same eventual failure at a higher concurrency ceiling.
The transferable part
A resource pool failing under load reads as "not enough resources" by default. JMX (or the equivalent instrumentation for whatever pool you're running) turns that from a guess into a measurement — and the measurement that mattered here wasn't a number going to zero, it was idle capacity coexisting with failures, which is the signature of a state bug wearing a capacity bug's symptoms.