Welcome to our blog! Here, we bring you the latest and greatest in the world of virtual currencies. Whether you're a seasoned pro or just getting started, we've got you covered. Our goal is to offer you with informative and useful content to help you navigate the ever-changing world of virtual currencies. So sit back, grab a cup of coffee, and let's jump into the exciting world of crypto together!
visual music —
Stable Diffusion-powered music generator processes sound in the visual space.
Benj Edwards - Dec 16, 2022 9:50 pm UTC
Enlarge / An AI-generated image of musical notes exploding forth from a computer monitor.
Ars Technica
On Thursday, a pair of tech hobbyists released Riffusion, an AI model that generates music from text prompts by creating a visual representation of sound and converting it to audio for playback. It uses a fine-tuned version of the Stable Diffusion 1.5 image synthesis model, applying visual latent diffusion to sound processing in a novel way.
Created as a hobby project by Seth Forsgren and Hayk Martiros, Riffusion works by generating sonograms, which store audio in a two-dimensional image. In a sonogram, the X-axis represents time (the order in which the frequencies get played, from left to right), and the Y-axis represents the frequency of the sounds. Meanwhile, the color of each pixel in the image represents the amplitude of the sound at that given moment in time.
Since a sonogram is a type of picture, Stable Diffusion can process it. Forsgren and Martiros trained a custom Stable Diffusion model with example sonograms linked to descriptions of the sounds or musical genres they represented. With that knowledge, Riffusion can generate new music on the fly based on text prompts that describe the type of music or sound you want to hear, such as "jazz," "rock," or even typing on a keyboard.
After generating the sonogram image, Riffusion uses Torchaudio to change the sonogram to sound, playing it back as audio.
Enlarge / A sonogram represents time, frequency, and amplitude in a two-dimensional image.
"This is the v1.5 Stable Diffusion model with no modifications, just fine-tuned on images of spectrograms paired with text," write Riffusion's creators on its explanation page. "It can generate infinite variations of a prompt by varying the seed. All the same web UIs and techniques like img2img, inpainting, negative prompts, and interpolation work out of the box."
Visitors to the Riffusion website can experiment with the AI model thanks to an interactive web app that generates interpolated sonograms (smoothly stitched together for uninterrupted playback) in real time while visualizing the spectrogram continuously on the left side of the page.
Enlarge / A screenshot of the Riffusion website, which lets you type in prompts and hear the resulting sonograms.
It can fuse styles, too. For example, typing in "smooth tropical dance jazz" brings in elements of different genres for a novel result, encouraging experimentation by blending styles.
Of course, Riffusion is not the first AI-powered music generator. Earlier this year, Harmonai released Dance Diffusion, an AI-powered generative music model. OpenAI's Jukebox, announced in 2020, also generates new music with a neural network. And websites like Soundraw create music non-stop on the fly.
Compared to those more streamlined AI music efforts, Riffusion feels more like the hobby project it is. The music it generates ranges from interesting to unintelligible, but it remains a notable application of latent diffusion technology that manipulates audio in a visual space.
The Riffusion model checkpoint and code are available on GitHub.
AI DeFi Blog is a top resource for all things related to blockchain-based finance and digital assets. Our team of professionals is dedicated to providing our readers with the newest news, insights, and analysis on the rapidly evolving world of DeFi. At AI DeFi Blog, we are enthusiastic about all things DeFi, from leverage trading to yield farming and beyond. We believe that DeFi has the capacity to transform the way we think about finance and financial systems, and we are excited to be a part of this expanding movement. One of the main features of DeFi is that it is built on blockchain technology, which allows for peer-to-peer transactions that do not require a third party, such as a bank, to facilitate. This means that you can be in control of your own financial transactions and assets, which can be especially appealing to those who are skeptical of traditional financial systems. DeFi also allows for greater accessibility and inclusion, as it enables anyone with an internet connection to participate in financial transactions and activities. This is particularly important in regions where traditional financial systems may be less advanced or inaccessible. In addition to DeFi, we also cover a variety of topics related to cryptocurrency, including the king of crypto, altcoins, mining, and more. We understand that the world of cryptocurrency can be intimidating, especially for those who are new. That's why we strive to provide our readers with clear and concise and user-friendly content that covers the most important aspects of cryptocurrency and DeFi. Whether you're a seasoned pro or just getting started, we've got something for you. Our goal is to provide our readers with the knowledge and tools they need to navigate the exhilarating world of DeFi and cryptocurrency. So join us as we explore the exciting world of DeFi and cryptocurrency together! From margin trading to yield farming and beyond, we've got you covered.