서버가 한 시간 넘게 죽어 있었는데 아무도 몰랐다 — 커넥션 풀 고갈과 이벤트 루프

That day

Every endpoint was timing out. Fly's health check was critical, and logins failed. It stayed that way for over an hour, until I noticed and restarted the machine by hand. It came back immediately.

That it recovered on restart was the important clue. This was not a code bug. It was state that had wedged, and the process could not unwedge itself.

What stopped the server

HikariCP had left a trace:

leaked connection PgConnection@... on thread eventLoopGroupProxy-4-1
was returned to the pool (unleaked)   path=/health

eventLoopGroupProxy is a Netty event-loop thread. A JDBC connection had been opened on an event-loop thread.

Every database access in this app goes through newSuspendedTransaction(Dispatchers.IO), off the event loop. Every one except /health. The health check is a tiny query ("can the pool hand me a connection, does Postgres answer SELECT 1"), so it called blocking JDBC straight from the handler.

Normally harmless. When the pool runs dry, it is not.

1. The connection pool runs dry. I still do not know exactly what drained it that day. My best recollection is that I had pointed another tool at this server and forgotten about it, but Fly logs are gone after 24 hours and metrics collection was cut off at the time, so there is no record to check. A slow query, a leak, a burst: it could have been any of them.
2. /health waits up to connectionTimeout (3s) for a connection and queryTimeout (3s) for the query. For that time, one event-loop thread is pinned.
3. This VM has one shared CPU, so there are only a handful of event-loop threads.
4. Fly's health check arrives every 30 seconds. One by one, the remaining loop threads get pinned too.
5. With every loop thread pinned, the server cannot accept any request, even ones that never touch the database. Total unresponsiveness.
6. It cannot serve the requests that would let the pool recover, so there is no way out.

Why nobody knew

Fly's http_service check does not restart a machine when it fails. It only removes it from the load balancer. On a one-machine service, that means "the whole service quietly disappears."

There were no alerts either. The deploy script had Telegram and Slack hooks, but no tokens were configured, so even deploy notifications were silent. And that script runs on the deploy machine anyway; it could never report a runtime failure.

What I fixed

1. Moved the health check onto the IO dispatcher.

get("/health") {
    try {
        // JDBC blocks. Called directly on the event loop, it pins a loop thread
        // for the whole connection acquire + query. Always push it to IO.
        withContext(Dispatchers.IO) {
            ds.connection.use { conn ->
                conn.createStatement().use { st ->
                    st.queryTimeout = 3
                    val rs = st.executeQuery("SELECT 1")
                    require(rs.next() && rs.getInt(1) == 1)
                }
            }
        }
        DbHealthWatchdog.recordSuccess()
        call.respond(HttpStatusCode.OK, HealthResponse(status = "ok", db = true))
    } catch (e: Exception) {
        DbHealthWatchdog.recordFailure(e)
        call.respond(HttpStatusCode.ServiceUnavailable,
            // The health check is unauthenticated — exception messages carry the
            // JDBC URL and host. Report only the type; details stay in the logs.
            HealthResponse(status = "unhealthy", db = false, error = e::class.simpleName ?: "unknown"))
    }
}

Now when the pool is dry, /health returns 503 after three seconds and the event loop keeps serving everything else.

2. Self-restart when the database cannot be recovered. DbHealthWatchdog counts consecutive health-check failures and, at six (30-second interval, so about three minutes), exits the process. Fly restarts a dead process per its restart policy, and the pool is rebuilt from scratch. I had just confirmed that a manual restart fixes it, so the fix is to let the process do that instead of a human.

The threshold is deliberately conservative. If transient database latency caused restart loops, availability would get worse, not better. A test pins down that sporadic failures never trigger it. Three minutes to detect plus restart and JVM boot puts total downtime at about five minutes. If the database itself dies, recovery is still manual.

3. Found one more bug of the same class. Apple sign-in verification fetches appleid.apple.com/auth/keys over blocking HTTP, and the caller invoked it with no dispatcher. Every Apple login pinned an event-loop thread for one round trip to Apple. A few concurrent logins, or one slow response from Apple, would reproduce the outage.

The function became suspend, with only the JWKS fetch wrapped in withContext(Dispatchers.IO). Making the signature suspend was deliberate: leave it looking non-blocking and the next person calls it from the event loop again.

4. Built runtime alerting. OpsAlert sends to Telegram from inside the app. The watchdog alerts before it exits the process. If a restart silently fixes things, the same failure can repeat forever and nobody learns.

What followed

Five days later I audited the whole codebase. R2 object deletion was running on the event loop through a synchronous S3 client; it went to Dispatchers.IO for the same reason. You cannot verify a blocking fix by looking at it, so the test intercepts the delete call, records the executing thread's name, and checks that all three calls run on DefaultDispatcher-worker. I also noted that JWT validation reads the full user row on every request, holding a connection one or two extra times.

Twelve days later I added ResourceWatchdog. Every five minutes it reads Hikari's timeout counter and pending-thread count, RSS, and database volume usage, and sends a pool exhaustion warning to Telegram. A rising timeout count means users have already received 503s, so this catches the incident early in its course, not ahead of it. It alerts only on the transition from healthy to at-risk, re-alerts every six hours while the condition persists, and sends one recovery message. It creates no new metric series; it only reads existing values. That mattered because two things were fixed on the evening of the outage: the blocking call that froze the server, and 24 minutes later, the metrics pipeline that should have shown it (the Grafana free-tier overrun, unrelated in cause). The series budget had been a sensitive number ever since.

The same day, the pool size went from the default 10 to an explicit 5. The ops doc's formula (3 per core) says 3; with short queries and flycast round-trip waits, 5 sits a little above the formula and well below a default of 10 that was oversized for one vCPU.

What I learned

And on a one-machine service, "removed from the load balancer" means "the service is gone." What the platform's health check does and does not do is something to read before the incident. I read it after.