LLM을 이용한 곡 만들기 — 연습곡을 무한 생산하는 AI 작곡 파이프라인

Why — if songs run out, everything collapses

Measure-level diagnosis, spaced repetition, tier leagues — they only matter when there are enough songs to play. With just a few dozen, a carefully built learning loop runs dry in a month. Unlike large learning apps with thousands of songs, a solo pipeline that uploads one hand-made MIDI at a time can't close that gap.

So the goal:

Specify a difficulty, key, and skill focus, and a NoteChart-compatible educational practice song is auto-generated and validated, with the human only there to listen and approve.

The goal is to drop the marginal cost per song from hours of production to tens of seconds of generation + a short review (P1 still feeds IR by hand, so this saving is a target to verify in P2, not a measured number). As a bonus, generated songs need no copyright negotiation, and they're born with accurate hand labels and measure structure — which even helps diagnosis accuracy.

Picking one of five paths

There are many ways to automate composition: a dedicated music model (Music Transformer et al.), pure rules (chord progressions + pattern libraries), audio generation (Suno-style), and LLM symbolic generation. The deciding criterion wasn't "the best song" but educational control — our songs exist not as art but as practice for a specific skill.

Conclusion: start the MVP with LLM symbolic, then evolve to a hybrid where rules guarantee the skeleton and the LLM fills in melody within it. For an MVP, what mattered more than a dedicated model's average musical quality was how reliably you can control the target difficulty and skill focus — and the lowest control cost was LLM symbolic.

The core: don't make the LLM do milliseconds

This is the most important design decision. Ask an LLM to emit a song directly in "milliseconds and MIDI numbers," and arithmetic errors pour out. Computing, for every note, exactly how many ms beat 3 is at 76 BPM or which number E4 is — that's asking the LLM to do the very thing it's worst at.

So there's an intermediate representation (IR). The LLM composes like a human would — in measures and beats, with pitches as note names ("E4", "F#3"). The ms conversion and MIDI mapping are handled deterministically, afterward, by the compiler.

{
  "title": "Summer Field", "key": "C", "tempo_bpm": 76,
  "time_signature": [4, 4], "difficulty": "Normal",
  "measures": [
    { "n": 1, "notes": [
      { "hand": "R", "pitch": "E4", "beat": 1, "len": 2 },
      { "hand": "R", "pitch": "G4", "beat": 3, "len": 1 },
      { "hand": "R", "pitch": "A4", "beat": 4, "len": 1 },
      { "hand": "L", "pitch": "C3", "beat": 1, "len": 4 }
    ] }
  ]
}

Note names have a lower LLM error rate than MIDI numbers and, above all, are human-readable and debuggable. You can spot "the left hand in measure 3 is off" with your eyes. The name → MIDI conversion is done deterministically by the compiler, not the LLM.

/// "C4" → 60, "F#3" → 54, "Bb4" → 70. This function does it, not the LLM.
int pitchToMidi(String pitch) {
  final m = RegExp(r'^([A-Ga-g])([#b]?)(-?\d+)$').firstMatch(pitch.trim());
  if (m == null) throw FormatException('bad pitch: $pitch');
  const base = {'C': 0, 'D': 2, 'E': 4, 'F': 5, 'G': 7, 'A': 9, 'B': 11};
  var semis = base[m.group(1)!.toUpperCase()]!;
  if (m.group(2) == '#') semis += 1;
  if (m.group(2) == 'b') semis -= 1;
  return (int.parse(m.group(3)!) + 1) * 12 + semis; // C4 = 60
}

Don't trust — verify: the validator and the regeneration loop

LLM output is not trusted. Every generated IR must pass the validator in full. It checks three broad things:

• Physics — how many keys one hand can press at once (≤5) and how wide (≤a 9th), the key range (C3–C5), no hand-crossing on Easy.
• Musical fit — the ratio of out-of-key notes, and of sustained/repeated vertical dissonance (minor-2nd / major-7th family).
• Difficulty envelope — whether it exceeds the per-difficulty caps (table below).

The important part is how failures are handled. A validation-failure message isn't there for a human to fix — it's there to be handed straight back to the LLM. It states mechanically "which measure, what, and by how much it's off," and that sentence goes into the next generation prompt to regenerate.

── 04_normal_pop.json ──
  metrics: nps=5.12 chord=41% leap=14 …
  ⚠ musicality: melody is mostly short cells (long notes 8%) — add ≥1 anchor long note (≥1 beat) per phrase
  ✗ 2 validation failures (regeneration feedback):
    - difficulty(Normal) violation: NPS 5.12 > 4.5
    - difficulty(Normal) violation: max leap 14 semitones > 12

Measured by one ruler — the difficulty envelope

Difficulty isn't a feeling; it's a measurement. Metrics like notes per second (NPS), chord ratio, max leap, and out-of-key ratio have per-difficulty caps, and those caps are shared — as the same constants — by the generation prompt's constraints, the validator, and the progression gate. The one who builds and the one who grades use the same ruler.

The Easy row is this constant in code — the generation prompt, the validator, and the progression gate all share this one.

static const easy = DifficultyEnvelope(
  maxNps: 2.0,              // ≤ 2 notes/sec
  maxChordRatio: 0.10,      // ≤ 10% simultaneous
  maxLeapSemis: 7,          // ≤ a perfect fifth
  maxAccidentalRatio: 0.0,  // strictly in-key
  forbidHandCrossing: true, // no hand crossing
);

What the machine can't catch — musicality

Here we meet the pipeline's limit. The validator filters out "physically playable, difficulty-appropriate" songs, but it cannot judge a good melody. When I generated 6 songs in P1 and listened by ear, the one repeated weakness was exactly that.

So it split in two. The validator mechanically blocks structural problems, while musical quality is accumulated as rules distilled from human evaluation. Rules like "it must be singable — at least one anchor long note (≥1 beat) per phrase" and "stepwise motion is the default, leaps are for the peak, once or twice" go straight into the generation prompt. And the machine shadows of those rules (long-note ratio, stepwise ratio) surface as warnings that flag which songs a reviewer must listen to. Every new evaluation grows this ruleset.

The IR made it easy to test

A hidden benefit of this pipeline is testability. The IR is pure data (JSON) and the compile is a pure function, so it can all be pinned down with table tests — no device, no audio. The pipeline is in fact held together by 43 assertions.

• Note name → MIDI — C4 = 60, sharps/flats/low notes, and the key's diatonic set (G major includes F#, not F), fixed as unit tests.
• Deterministic compile — nailing down values like "at 4/4 tempo 70, measure 2 beat 1 = 3429ms," so any drift in the time math fails immediately.
• Validator negative cases — the things that must be rejected are pinned as tests: measure overflow, out-of-keyboard-range (D5 = 74 > 72), Easy hand-crossing, Easy out-of-key notes, the Easy leap envelope (an octave jump), vertical dissonance (a sustained E over an F-major chord = a major 7th), a one-hand span beyond a 9th, and the musicality warning (all-short-cell melodies).

A validator is defined less by "what it passes" than by "what it rejects." Pin that rejection list as tests, and the moment you change one rule, any hole you accidentally opened shows up at once.

Now, and what's next

Everything so far is P1 — the offline pipeline (IR → validate → compile → NoteChart) runs, 6 songs were evaluated by ear, and inside the app they were synthesized through a dev-only preview screen for listening.

• P2 (next) — right now IR files are fed in by hand, so metrics like first-pass rate and average regenerations per song don't exist yet. Attaching a Claude API generation step in front of the CLI automates the generate → validate → (on failure) regenerate loop. The success criteria are clear — measure the first-pass rate of songs approvable without human edits and the average regenerations per song, so a single CLI run yields catalog candidates.
• After that (the differentiator) — tie it to measure-and-hand diagnosis to generate personalized practice songs targeting the measures and skills a student misses most. Because generated songs are born with accurate hand labels and measure structure, their diagnosis accuracy is actually higher than uploaded songs.

Closing

"AI composition" tends to conjure "AI makes art," but the hard part was the opposite — how far to let the LLM's creativity run, and where to fence it in with code. Milliseconds to the code, melody to the LLM. Verification to the machine, musicality to the human ear. Drawing that boundary precisely was the whole of the pipeline. It was a way to solve the biggest risk — running out of songs — with engineering rather than art.