MIDI 피아노를 게임으로 — LevelUp Piano 개발기

Why I built it

Practicing piano alone, you always hit the same wall: "Am I even playing this right?"

Without a teacher in the room, it's hard to catch your own wrong notes or off beats. One approach I weighed was grading sound through the microphone, but a mic struggles with ambient noise and chords, and it can't tell you which note was late and by how much.

A digital piano already knows the answer — through MIDI. Exactly which key, at exactly which millisecond, with exactly how much velocity, all as precise numbers. Use that signal well and you can give note-by-note feedback a microphone never could.

So I set the goals like this:

• Connect a digital piano to an iPad and judge playing note by note, in real time.
• Turn that judgment into score, XP, and progression — so solo practice feels like a game.
• Add a habit loop that brings people back to the keys every day.

What I built

Real-time performance judging

The heart of it is the judgment engine. It aligns the song's note chart (which note should sound, and when) with the user's MIDI input on a shared timeline, and marks each note as perfect / early / late / missed. Notes flow down the screen, and pressing a key gives instant feedback.

Four roles

Very different users live inside one app:

• Independent learner — a general user following a self-paced curriculum without a teacher. Subscription-based.
• Student — practices songs assigned by a teacher, who follows their progress.
• Teacher — uploads MIDI files to create songs and assign them to students.
• Operator — manages the official song catalog everyone shares.

Gamification that keeps you coming back

Practice is boring. So I borrowed the language of games:

• XP and levels: a sense of progress that accrues as you practice
• Streaks + streak freeze: protect your run, but don't let one missed day collapse it
• Daily goals: let users promise themselves what to do today
• Tier leagues and friends: weekly ranking and social motivation
• Missions and the mascot "Maestro": short goals and a character cheering you on

Diagnosis that points to your weak spots

Instead of just "80 points," the app records which measures and which hand you miss most, at the measure and hand level. From that data it suggests targeted re-practice — "just measures 3–7, left hand" — and uses spaced repetition to lock in weak spots over time.

Look around even without a piano

So people without a MIDI keyboard can still experience the app, onboarding lets you play a sample song on an on-screen touch keyboard (no scoring). It lowers the entry barrier of a hardware-dependent app.

Tech stack and architecture

Note charts and backing audio live in R2; game state, progress, and social data live in PostgreSQL. The app is designed iPad-first — the larger screen fits the falling-note gameplay and the keyboard layout much better.

Why this stack. Being iPad-first, the cross-platform UI is built fast in Flutter — but real-time audio synthesis is beyond Flutter, so it bridges to iOS native (AVAudioEngine). The backend carries heavy domain logic for judging, scoring, and social, so it's statically-typed Kotlin/Ktor; large files like note charts and audio go to R2 rather than the DB, while relational game state lives in PostgreSQL.

Key design decisions

MIDI hardware as a premise

The core experience only completes with a physical piano. So a connection wizard explicitly guides device selection (one piano often enumerates as several endpoints), and if there's no keyboard, a touch trial routes around it. Connection and judging themselves are covered in the real-time MIDI judgment engine, down to the ±50/150/300ms windows.

Baking MIDI into audio, on the device

Arbitrary teacher-uploaded MIDI has to become practice backing audio at three tempos (1x / 0.75x / 0.5x). Instead of hosting a synthesizer on the server, I render it on the iPad itself, using iOS AVAudioEngine's offline rendering and a SoundFont. The separate post covers that pipeline down to the two-pass, sub-chunking, and 64-frame trim.

One app, four roles

Each role has a different home screen, permissions, and data access, yet shares a lot — like the practice screen. Students and independent learners run the same gameplay engine while only the song source (assigned songs vs. catalog) differs — a shared core separated from per-role shells. The role-design post goes all the way to the permission matrix.

From Korean to the world

It started Korean-only, but adding internationalization meant handling not just UI strings but content data such as song titles per locale (Korean and English today, then Chinese and Japanese). Those three layers of localization are written up in From Korean to the World.

Generating endless practice songs with AI

The biggest risk — running out of songs to play — is being solved with a pipeline that generates and validates practice songs via an LLM: it composes in measures and beats instead of milliseconds, and feeds validation failures straight back for regeneration. The design is detailed in Making Songs with an LLM.

Closing

LevelUp Piano began from a simple idea — "a precise signal (MIDI) plus a game's habit design" — but became a project where real-time judging, on-device audio rendering, multiple roles, gamification, and localization all interlock. Getting a solo learner to sit down for "one more round today" — that small moment is what it's all for.