A technique note, not a post-mortem: there is no specific incident behind it, and no numbers from one.
TL;DR
- "Connection errors at peak hours" has at least three causes — the pool is genuinely too small, connections are leaking, or the pool is handing out connections the database already closed — and they all look the same in an error log.
- HikariCP's JMX metrics tell them apart in minutes: saturation and leaks both pin
ActiveConnectionsat the ceiling, but only a leak keeps it there after traffic drops; dead connections fail whileIdleConnectionsis still above zero. - Raising
maximumPoolSizefixes exactly one of the three, and it hides the other two by moving the failure to a higher traffic level.
You see SQLTransientConnectionException: Connection is not available, request timed out clustered around the busiest hour of the day, and it clears up on its own afterwards. The obvious reading is that peak concurrency outgrew the pool. Sometimes it did. Before you change the pool size, spend ten minutes finding out which failure you actually have.
What to watch
Set registerMbeans=true and HikariCP publishes a HikariPoolMXBean per pool. Four values matter:
| Metric | Meaning |
|---|---|
ActiveConnections | Connections currently borrowed by application code |
IdleConnections | Connections in the pool, ready to hand out |
ThreadsAwaitingConnection | Callers blocked waiting for one |
TotalConnections | Active + idle |
Watch them through a peak and the following quiet period. The three failures leave different shapes.
Shape 1: saturation
ActiveConnections sits at maximumPoolSize, ThreadsAwaitingConnection climbs, and both fall back when traffic falls. This is the only shape where a bigger pool is a real fix — and even then check the database side first, because HikariCP's own sizing guidance is that a pool much larger than the database's core count usually makes throughput worse, not better. Slow queries holding connections longer look identical here; the query log decides which one you have.
Shape 2: a leak
ActiveConnections also climbs to the ceiling — a leaked connection is still borrowed, so it counts as active — but it does not come back down when traffic drops. Every request that took a connection and never closed it has removed one from the pool permanently.
The usual cause is manual acquire-and-release code where an exception skips the release:
Connection conn = dataSource.getConnection();
validate(request); // throws → conn is never closed
doWork(conn);
conn.close();
leakDetectionThreshold makes HikariCP log the stack trace of where the connection was borrowed for any connection held longer than the threshold — which points straight at the method that forgot to close it. The fix is try-with-resources on every acquisition, so close() runs on every exit path:
try (Connection conn = dataSource.getConnection()) {
validate(request);
doWork(conn);
}
A bigger pool only buys a leak more time before it fails again.
Shape 3: dead connections
Requests fail while IdleConnections is above zero. The pool has capacity; the connections in it are no longer valid, because something between you and the database — the database's own idle timeout, a firewall, a load balancer — closed them while they sat idle. This is the shape the title refers to.
It rarely shows up as the clean timeout message. HikariCP tests a connection before handing it out if it has been idle longer than about half a second, and evicts it if the test fails — so the symptoms are slower borrows while the pool replaces what it evicted, plus link-failure or broken-pipe errors when a connection dies between that test and the query.
HikariCP's documentation is direct about the fix: set maxLifetime several seconds shorter than any connection time limit imposed by the database or the network, so the pool retires connections before anything else kills them. Pool size is irrelevant here; a larger pool is just a larger supply of dead connections.
The transferable part
Read the pool before you resize it. The error message is the same for all three failures, and the metrics are not. Two numbers — whether ActiveConnections recovers after peak, and whether failures happen while idle connections exist — tell you which problem you have.
A capacity fix on a state bug delays the failure, it doesn't remove it. Leaks and dead connections both look like "not enough connections" from the outside. Adding connections changes the traffic level at which they show up, which makes the next investigation harder, not easier.