Stability AI debuts Stable Audio bringing text to audio generation to the masses

What comes after building generative AI technology for image and code generation? For Stability AI, it’s text-to-audio generation.

Stability AI today announced the initial public release of its Stable Audio technology, providing anyone with ability to use simple text prompts to generate short audio clips. Stability AI is best known as the organization behind the Stable Diffusion text-to-image generation AI technology.  

Back in July, Stable Diffusion was updated with its new SDXL base model for improved image composition. The company followed up on that news by expanding its scope beyond image to code, with the launch of StableCode in August.

StableAudio is a new capability, though it is based on many of the same core AI techniques that enable Stable Diffusion to create images. Namely the Stable Audio technology makes use of a diffusion model, albeit trained on audio rather than images, in order to generate new audio clips.

“Stability AI is best known for its work in images, but now we’re launching our first product for music and audio generation, which is called Stable Audio,”Ed Newton-Rex, VP of Audio at Stability AI told VentureBeat. “The concept is really simple, you describe the music or audio that you want to hear in text and our system generates it for you.”

How Stable Audio works to generate new pieces of music, not MIDI files

Newton-Rex is no stranger to the world of computer generated music, having built his own startup called Jukedeck in 2011, which he sold to TikTok in 2019.

The technology behind Stable Audio however does not have its roots in Jukedeck, but rather in Stability AI’s internal research studio for music generation called Harmonai, which was created by Zach Evans.

“It’s a lot of taking the same ideas technologically from the image generation space and applying them to the domain of audio,” Evans told VentureBeat. “Harmonai is the research lab that I started and it is fully part of Stability AI and it is a basically a way to have this generative audio research happening as a community effort in the open.”

The ability to generate base audio tracks with technology is not a new thing. Individuals have been able to use what Evans referred to as ‘symbolic generation’ techniques in the past. He explained that symbolic generation commonly works with MIDI (Musical Instrument Digital Interface) files that can represent something like a drum roll for example. The generative AI power of Stable Audio is something different, enabling users to create new music that goes beyond the repetitive notes that are common with MIDI and symbolic generation.

Stable Audio works directly with raw audio samples for higher quality output. The model was trained on over 800,000 pieces of licensed music from audio library AudioSparks.

“Having that much data, it’s very complete metadata,” Evans said. “That’s one of the really hard things to do when you’re doing these text based models is having audio data that is not only high quality audio, but also has good corresponding metadata.”

Don’t expect to use Stable Audio to make a new Beatles tune

One of the common things that users do with image generation models is to create images in the style of a specific artist. For Stable Audio however, users will not be able to ask the AI model to generate new music, that for example sounds like a classic Beatles tune.

“We haven’t trained on the Beatles,” Newton-Rex said.”With audio sample generation for musicians, that has tended not to be what people want to go for.”

Newton-Rex noted that in his experience, most musicians do not want to start a new audio piece by asking for something in the style of The Beatles or any other specific musical group, rather they want to be more creative.

Learning the right prompts for text to audio generation

As a diffusion model, Evans said that the Stable Audio model has approximately 1.2 billion parameters, which is roughly on par with the original release of Stable Diffusion for image generation.

The text model used for prompts to generate audio was all built and trained by Stability AI. Evans explained that the text model is using a technique known as Contrastive Language Audio Pretraining (CLAP). As part of the Stable Audio launch, Stability AI is also releasing a prompt guide to help users with text prompts that will lead to the types of audio files that users want to generate.

Stable Audio will be available both for free and in a $12/month Pro plan. The free version allows 20 generations per month of up to 20 second tracks, while the Pro version increases this to 500 generations and 90 second tracks

 “We want to give everyone the chance to use this and experiment with it,” said Newton-Rex.