All writing

4 min

The Pool Wasn't Out of Connections, It Was Out of the Right Ones

Peak-hour database errors read as an undersized connection pool by default. HikariCP's JMX metrics separate three different failures — real saturation, a leak, and dead connections — and only one of them is fixed by a bigger pool.

Contents

A technique note, not a post-mortem: there is no specific incident behind it, and no numbers from one.

TL;DR

  • "Connection errors at peak hours" has at least three causes — the pool is genuinely too small, connections are leaking, or the pool is handing out connections the database already closed — and they all look the same in an error log.
  • HikariCP's JMX metrics tell them apart in minutes: saturation and leaks both pin ActiveConnections at the ceiling, but only a leak keeps it there after traffic drops; dead connections fail while IdleConnections is still above zero.
  • Raising maximumPoolSize fixes exactly one of the three, and it hides the other two by moving the failure to a higher traffic level.

You see SQLTransientConnectionException: Connection is not available, request timed out clustered around the busiest hour of the day, and it clears up on its own afterwards. The obvious reading is that peak concurrency outgrew the pool. Sometimes it did. Before you change the pool size, spend ten minutes finding out which failure you actually have.

What to watch

Set registerMbeans=true and HikariCP publishes a HikariPoolMXBean per pool. Four values matter:

MetricMeaning
ActiveConnectionsConnections currently borrowed by application code
IdleConnectionsConnections in the pool, ready to hand out
ThreadsAwaitingConnectionCallers blocked waiting for one
TotalConnectionsActive + idle

Watch them through a peak and the following quiet period. The three failures leave different shapes.

Shape 1: saturation

ActiveConnections sits at maximumPoolSize, ThreadsAwaitingConnection climbs, and both fall back when traffic falls. This is the only shape where a bigger pool is a real fix — and even then check the database side first, because HikariCP's own sizing guidance is that a pool much larger than the database's core count usually makes throughput worse, not better. Slow queries holding connections longer look identical here; the query log decides which one you have.

Shape 2: a leak

ActiveConnections also climbs to the ceiling — a leaked connection is still borrowed, so it counts as active — but it does not come back down when traffic drops. Every request that took a connection and never closed it has removed one from the pool permanently.

The usual cause is manual acquire-and-release code where an exception skips the release:

java
Connection conn = dataSource.getConnection();
validate(request);          // throws → conn is never closed
doWork(conn);
conn.close();

leakDetectionThreshold makes HikariCP log the stack trace of where the connection was borrowed for any connection held longer than the threshold — which points straight at the method that forgot to close it. The fix is try-with-resources on every acquisition, so close() runs on every exit path:

java
try (Connection conn = dataSource.getConnection()) {
    validate(request);
    doWork(conn);
}

A bigger pool only buys a leak more time before it fails again.

Shape 3: dead connections

Requests fail while IdleConnections is above zero. The pool has capacity; the connections in it are no longer valid, because something between you and the database — the database's own idle timeout, a firewall, a load balancer — closed them while they sat idle. This is the shape the title refers to.

It rarely shows up as the clean timeout message. HikariCP tests a connection before handing it out if it has been idle longer than about half a second, and evicts it if the test fails — so the symptoms are slower borrows while the pool replaces what it evicted, plus link-failure or broken-pipe errors when a connection dies between that test and the query.

HikariCP's documentation is direct about the fix: set maxLifetime several seconds shorter than any connection time limit imposed by the database or the network, so the pool retires connections before anything else kills them. Pool size is irrelevant here; a larger pool is just a larger supply of dead connections.

The transferable part

Read the pool before you resize it. The error message is the same for all three failures, and the metrics are not. Two numbers — whether ActiveConnections recovers after peak, and whether failures happen while idle connections exist — tell you which problem you have.

A capacity fix on a state bug delays the failure, it doesn't remove it. Leaks and dead connections both look like "not enough connections" from the outside. Adding connections changes the traffic level at which they show up, which makes the next investigation harder, not easier.

Share this note

Comments

— responses

0/2000

Loading comments…