대시보드가 매달 중순이면 비는 이유 — 포트 스캐너가 메트릭 라벨이 되기까지

Symptom

Past the middle of every month, the Grafana dashboard went blank. The server was fine. Logs kept flowing. Only metrics stopped.

The Grafana Cloud free tier allows 10,000 active metric series. And billing is on the monthly average, not the instantaneous value. One spike and you are cut off for the rest of the month. I did not know either of these things, so for a while I filed it under "flaky."

First cause — histogram buckets (August 14)

The first time we went over, I counted the production /metrics output by hand. The app had just restarted, so only two route combinations existed. Even so:

Micrometer's percentilesHistogram(true) creates about 69 buckets for every route × method × status × address × throwable combination. That is 72 series per combination. 139 combinations fill the quota. On an API with 100 endpoints, that happens fast.

Two changes:

1. percentilesHistogram(false), exporting only app-computed p50, p95 and p99 instead. 72 → 6 series per combination.
2. Drop the address label. Its value was <machine-id>:8080, so every redeploy split the same route into a new series. With one machine, it carried no information.

What we lose is recomputing arbitrary quantiles with histogram_quantile, and heatmaps. The p95 we use to find bottlenecks stays.

Second cause — scanner paths became labels (August 26)

Twelve days after the buckets were gone, we were cut off again. Same month. This time I started from the org-wide usage page. One stack. Logs, traces, profiles all at 0 GB. Metrics at 20K. No other project to blame. It was this app.

Plotting series count over time in Grafana Explore showed a sawtooth. 42 right after a deploy, 2,600 a few days later, back to zero on the next deploy. The app as a whole hovered at 4,000–5,000, and between August 10 and 14 it had spiked to 14–15K.

The server's own /metrics showed the culprit:

ktor_http_server_requests_seconds_count{method="GET",route="/_rNd9xZ7kL3",status="404"} 1
ktor_http_server_requests_seconds_count{method="GET",route="/wp-admin/setup-config.php",status="404"} 1
ktor_http_server_requests_seconds_count{method="GET",route="/.env",status="404"} 1

api.piano.nariinfo.dev is a public API, so vulnerability scanners hammer it with random paths around the clock. Ktor's metrics plugin defaults to distinctNotRegisteredRoutes = true, which puts the literal path of every unmatched request into the route label. Six series per path (count, sum, max, three quantiles). Internet background noise was being turned directly into observability cost.

Because the sawtooth reset on every deploy, the instantaneous value was always under the limit. That is why it hid for so long.

The fix — one line

// ① Registry — the address label is dropped here (Aug 14 fix)
val appMicrometerRegistry: PrometheusMeterRegistry =
    PrometheusMeterRegistry(PrometheusConfig.DEFAULT).apply {
        config().meterFilter(MeterFilter.ignoreTags("address"))
    }
 
// ② Distribution config — no buckets (Aug 14 fix)
val httpTimerDistributionConfig: DistributionStatisticConfig =
    DistributionStatisticConfig.Builder()
        .percentilesHistogram(false)
        .percentiles(0.5, 0.95, 0.99)
        .build()
 
// ③ install — collapse unregistered paths (Aug 26 fix, this one line)
install(MicrometerMetrics) {
    registry = appMicrometerRegistry
    distributionStatisticConfig = httpTimerDistributionConfig
    distinctNotRegisteredRoutes = false
}

Every unmatched path now folds into a single route="n/a". The series count is tied to the API surface (number of endpoints), not to scanner traffic. On paper that brings usage down to 10–15% of the quota.

One line, but I checked the side effects in a table.

Locking it in with tests

A single config flag is easy for the next person (or next month's me) to flip back. So two tests pin it down.

@Test
@DisplayName("Unregistered paths hit by scanners collapse to one label — request count preserved")
fun unmatchedRoutesCollapseToSingleLabel() {
    val junk = listOf(
        "/_rNd9xZ7kL3", "/wp-admin/setup-config.php", "/.env",
        "/phpmyadmin/index.php", "/actuator/health",
    )
    val (before, lines) = requestCountLines { client -> junk.forEach { client.get(it) } }
 
    junk.forEach { path ->
        assertTrue(lines.none { routeOf(it) == path }, "unregistered path leaked into a label: $path")
    }
    // Only the GET series this block created or grew.
    val beforeCounts = byLabels(before)
    val grew = byLabels(lines).filter { (labels, count) ->
        routeOf(labels) == "n/a" && labels.contains("""method="GET"""") &&
            count > (beforeCounts[labels] ?: 0.0)
    }
    assertEquals(1, grew.size, "unregistered paths must merge into one series")
    val (labels, count) = grew.entries.single()
    assertEquals(junk.size.toDouble(), count - (beforeCounts[labels] ?: 0.0),
        "request count must survive — only the label collapses")
}
 
@Test
@DisplayName("A registered route keeps its template even on 404 — the not-found signal survives")
fun registeredRouteKeepsTemplateEvenOn404() { /* /api/v1/songs/{id} → 404 → route label is the template */ }

The first test originally asserted that exactly one route="n/a" series exists, and it broke. The meter registry is JVM-global, so when another test produced a single 404 with a non-GET method, that series leaked in. It now checks the delta produced by this block, not absolute values. The histogram side is guarded by the five cases in MetricsCardinalityTest (no buckets, quantiles kept, address dropped, and the two allow-list cases below). Seven tests in total guard cardinality.

What I deliberately did not do

• A cardinality cap (maximumAllowableTags): with 100 endpoints, a cap of 50 silently drops half the real routes. Worse, which ones survive depends on request arrival order after restart, so it changes on every deploy.
• Scrape interval 30s → 60s: that exactly halves billable series, but the fix above already brought us far enough down that there is no reason to give up resolution. It stays in the back pocket.
• Paying: at a calculated 10–15% of quota, too early.

What I learned

In fact, the same structure had already been blocked once before this. Five days after the August 14 outage, while auditing the whole codebase, I found a place where client-supplied role and difficulty strings went straight into a Prometheus tag with no validation. An authenticated user POSTing random strings could grow series without bound. Anything outside the allow-list (two roles, four difficulties) now folds into other, capping the combinations at 3 × 5, and the guard lives in the metrics function rather than the route, so it stays safe no matter how many callers appear. The test throws 50 kinds of junk at it and checks that the label set is exactly the allow-list plus other.

When I wrote up the scanner cause on August 26, I cited that fix as the precedent. The audit had, for once, gotten ahead of an incident.

Every path where external input touches a label gets collapsed or allow-listed. That is the rule two incidents left behind.