Why the GPT approach stumbles over symbolic music: compression only works in the right coordinate system
It seems like symbolic music was made for tokenization: notes are discrete, chords repeat, motifs form recognizable structures. GPT models work great with reusable tokens, so it's tempting to directly transfer the language approach to the musical staff. But practice shows: tokenizing music isn't just about slicing a sequence—it's about choosing a coordinate system in which the model can actually see patterns.
Tokens are not a universal interface
The success of language models rests on a finite vocabulary of tokens that can be reused across different contexts. This interface is so convenient that people have started applying it to other domains—from code to biological sequences. However, when moving beyond language, a problem arises: there's no single criterion for what tokenization should represent or where it should stop. What works well for text may prove useless or even harmful for music.
Research by Yi Wang (arXiv:2608.18025) proposes a general criterion—predictive codelength. This measures how well a model compresses information when predicting the next element. If tokenization helps reduce this length, it's capturing real structure in the data; if not, it merely creates the illusion of order.
Music requires coordinates, not just a vocabulary
Symbolic music is deceptively similar to language: the same notes, the same chords. But behind this discreteness lie continuous parameters of time and tonality. When we slice music into tokens by analogy with words, we implicitly fix a frame of reference. For example, we can encode pitch as an absolute value—but then the model has to figure out the key on its own. We can represent time as an integer counter—but then shifts and tempo become indistinguishable.
The author of the Effectiveness–Losslessness Framework draws two boundaries. The Fact–Token Boundary defines where observed structure should enter the token interface—for instance, through explicit construction of coordinates. The Token–State Boundary indicates where tokenization should stop: the remaining dependencies the model should compute on its own, in its internal state, rather than receiving them ready-made from the tokenizer. If you compress too much, the model loses the ability to use context; if you compress too little, it drowns in irrelevant details.
Experiment: time and tonality decide
In controlled multi-seed experiments on symbolic music, it turned out that explicitly specifying musical time consistently reduces predictive code. That is, instead of hiding timestamps inside tokens, it's better to make them a separate coordinate—this way the model starts predicting both pitch and note duration more accurately. An additional gain comes from canonicalizing tonal context: bringing all phrases to a single tonal center. And factoring pitch—splitting it into scale degree and octave—also helps.
But what doesn't work matters too. Fixed pitch coordinates based on the circle of fifths—beautiful music theory imposed without regard for context—increase predictive code. A pre-fixed pitch relationship prevents the model from adapting to the specific harmony. The tokenizer decided for the model how notes relate to each other, leaving the model with nothing to do.
Compression for its own sake doesn't help
A separate surprise is reversible BPE. It noticeably shortens the carrier—the token sequence itself—but predictive codelength grows on every seed. From a data storage standpoint, it's an excellent compressor; from a training standpoint, it's a poor helper. That's because BPE collapses repeating fragments but loses contextual relationships: the model saves on length but spends effort learning artificial abbreviations.
Conclusion
The GPT approach stumbles over symbolic music not because of a lack of power, but because of the wrong choice of coordinate system. Compression only works when the token interface aligns with the real structure of the data: time should be time, tonality should be tonality, and everything else should be left for the model to work out on its own. The predictive codelength criterion makes it possible to separate useful tokenization from window dressing and to understand where the line between fact and computation should actually be drawn.
