DSUs capture sounds better than tone, and speech AI has to notice
The visual shows how tone can weaken when continuous speech is compressed into discrete tokens.📷 AI-generated / Tech&Space
- ★DSUs are useful speech tokens, but in Mandarin and Yorùbá they preserve lexical tone less reliably
- ★The authors find that SSL latent representations do encode tone, while quantization pushes the units toward phonetic structure
- ★The paper points to tone-aware or prosody-aware representations, including a second K-means step on the residual
The paper arXiv:2604.07467, by Opeyemi Osakuade and Simon King, tests one of speech AI's neatest compromises: discrete speech units, or DSUs. The idea behind DSUs is to turn a continuous audio signal into a sequence of tokens. That is useful because models can then handle speech more like text, which helps tasks such as text-to-speech and multimodal dialogue systems.
The problem is that speech is not only a sequence of sounds. Segmental structure means pieces such as vowels and consonants. Suprasegmental features sit above that: tone, stress, rhythm, duration, and intonation. In tone languages, lexical tone is not decoration. In Mandarin and Yorùbá, the pitch level and contour can carry word meaning. If a model loses it, the output does not merely sound less natural - it can point to the wrong word.
The authors therefore probe what happens when representations from self-supervised learning models are turned into DSUs. SSL means the model learns from large amounts of speech without every detail being manually labeled. Before quantization, the latent representations still carry tone information. Quantization is the step where a smooth, continuous signal is squeezed into discrete buckets. That is where the paper finds the bottleneck: after quantization, DSUs preserve phonetic structure more reliably than lexical tone.
A Mandarin and Yorùbá study shows that SSL representations carry tone, but discrete speech units often preserve it poorly after quantization.
The tone-contour comparison shows why the same tokenization step is not neutral across languages.📷 AI-generated / Tech&Space
The important nuance is that tone does not disappear because the earlier SSL model never saw it. According to the authors, the latent representations themselves do encode tone, but quantization reorganizes them so that phonetic structure wins priority. Put simply: the model has evidence of the melodic contrast somewhere, but the final tokens prefer to separate sounds cleanly rather than preserve the pitch curve. This holds for multiple quantization methods, not only the common K-means approach.
K-means is a clustering method: similar points in a feature space are placed in the same cluster, and the cluster becomes a token. If clustering optimizes for the strongest and most frequent structure in the signal, tone can become a weaker detail, especially when it competes with articulatory information. That is a problem for systems that need to be reliable in languages where prosody carries meaning.
The paper does not stop at critique. The authors call for tone-aware or prosody-aware techniques in speech representation learning. As one possible direction, they point to running K-means once to encode phonetic information, then running it again on the residual representation. The residual is what remains after the first pass has explained part of the signal. If the first pass captures the sounds, the second pass may have a better chance of capturing tone.
For industry, the lesson is dry but important: DSUs are convenient, not complete. If TTS systems, voice assistants, or multimodal agents use DSUs as a foundation, evaluation cannot stop at English or non-tonal test sets. It has to measure whether the system preserves meaning in languages where word melody is not style, but grammar and lexicon. The next gain in speech AI may therefore come not from a larger model, but from a better way to keep tokens from flattening the most informative parts of speech.