Researcher develops ‘SpeechSSM,’ opening up possibilities for a 24-hour AI voice assistant

Recently, spoken language models (SLMs) have been highlighted as next-generation technology that surpasses the limitations of text-based language models by learning human speech without text to understand and generate linguistic and non-linguistic information.

However, existing models show significant limitations in generating long-duration content required for podcasts, audiobooks, and voice assistants.

Ph.D. candidate, Sejin Park, from Professor Yong Man Ro’s research team at the Korea Advanced Institute of Science and Technology’s (KAIST) School of Electrical Engineering, has succeeded in overcoming these limitations by developing “SpeechSSM,” which enables consistent and natural speech generation without time constraints.

The work has been published on the arXiv preprint server and is set to be presented as at ICML (International Conference on Machine Learning) 2025.

A major advantage of SLMs is their ability to directly process speech without intermediate text conversion, leveraging the unique acoustic characteristics of human speakers, allowing for the rapid generation of high-quality speech even in large-scale models.

However, existing models faced difficulties in maintaining semantic and speaker consistency for long-duration speech due to increased “speech token resolution” and memory consumption when capturing very detailed information by breaking down speech into fine fragments.

SpeechSSM employs a “hybrid structure” that alternately places “attention layers” focusing on recent information and “recurrent layers” that remember the overall narrative flow (long-term context). This allows the story to flow smoothly without losing coherence even when generating speech for a long time.

Furthermore, memory usage and computational load do not increase sharply with input length, enabling stable and efficient learning and the generation of long-duration speech.

SpeechSSM effectively processes unbounded speech sequences by dividing speech data into short, fixed units (windows), processing each unit independently, and then combining them to create long speech.

Additionally, in the speech generation phase, it uses a “Non-Autoregressive” audio synthesis model (SoundStorm), which rapidly generates multiple parts at once instead of slowly creating one character or one word at a time, enabling the fast generation of high-quality speech.

While existing models typically evaluated short speech models of about 10 seconds, Se Jin Park created new evaluation tasks for speech generation based on their self-built benchmark dataset, “LibriSpeech-Long,” capable of generating up to 16 minutes of speech.

Compared to PPL (Perplexity), an existing speech model evaluation metric that only indicates grammatical correctness, she proposed new evaluation metrics such as “SC-L (semantic coherence over time)” to assess content coherence over time, and “N-MOS-T (naturalness mean opinion score over time)” to evaluate naturalness over time, enabling more effective and precise evaluation.

Through these new evaluations, it was confirmed that speech generated by the SpeechSSM spoken language model consistently featured specific individuals mentioned in the initial prompt, and new characters and events unfolded naturally and contextually consistently, despite long-duration generation.

This contrasts sharply with existing models, which tended to easily lose their topic and exhibit repetition during long-duration generation.

Sejin Park explained, “Existing spoken language models had limitations in long-duration generation, so our goal was to develop a spoken language model capable of generating long-duration speech for actual human use.”

She added, “This research achievement is expected to greatly contribute to various types of voice content creation and voice AI fields like voice assistants, by maintaining consistent content in long contexts and responding more efficiently and quickly in real time than existing methods.”

This research, with Se Jin Park as the first author, was conducted in collaboration with Google DeepMind.