At Solarflare, we love unearthing new and thrilling technological trends. Right now, we’re diving into the captivating world of sound and visual design – a space that not only fulfils us professionally but also serves as a springboard for creative musings.
The Task: More Than Just Notes and Rhythms
Our team isn’t made up of button-pushers and code-writers alone. Many of us moonlight as musical maestros under aliases that shall forever remain in the shadows, however, it did lead us to wonder:
Can AI maintain the excitement and joy of musical exploration while producing sounds worth listening to?
“This experiment has provided valuable insights into the integration of technology and artistry. AI, rather than replacing human creativity, emerged as a muse and a catalyst for new artistic avenues.”
– John Martinelli
Head of Production, Solarflare Studio
Want to explore the possibilities of new technologies? Get in touch to explore ideas.
The Journey
In this intersection of technology and creativity, we embarked on a compelling experiment to understand if AI could be a catalyst for innovation in audio and visual creativity. Can AI ignite new artistic expressions, or does it limit the creative process?
The AI SOUND Workflow
Our journey begins with MusicGen. This powerful tool allowed us to generate a core soundscape, starting from simple generative prompts. We created a swirling, ethereal chillwave soundscape, which became the foundation of our project. Figuring out the right bpm and melody was key, and MusicGen’s suggestions really helped us nail the overall sound we were going for.
Next, we introduced AudioGen into our creative process. The model can generate environmental sounds corresponding to the description with realistic recording conditions and complex scene context. We used this system to craft unique sound elements and FX, like pulsating drums, DJ decks scratching, and other elements to enhance the sound design and complement the overall theme.
As we fine-tuned the bpm and notes, Splice came into play. This AI-assisted creation tool was instrumental in orchestrating multiple layers, adding depth and complexity to our soundscape, and it was incredibly impressive how you could straightforwardly piece together samples that matched bpm and the musical scale.
Balancing AI and Manual Creativity
The AI outputs served as an essential baseline, sparking our creative journey. These systems significantly accelerated our process, providing a solid foundation. However, relying solely on AI was not enough to reach our particular taste. The joy and effectiveness of work stemmed from actively directing and refining the AI output. By orchestrating and manipulating these elements, we created sounds that complimented each other, adding a necessary layer of precision and touch that AI currently lacks.
Audio Reactive Visuals
Following our exploration into AI-driven audio, we then ventured into the world of visual design. Using the latest AI technology and visual development software, our aim was to bring the sounds visually to life.
Stable diffusion, a powerful AI tool for image generation, played a key role in the process. We used it to run specific text-to-image prompts that harmonised with our sounds. These prompts were carefully crafted to produce visuals that resonated with the tones and rhythms of our music. The idea was to create images that weren’t just reactive to the music but also an elegant extension, adding depth and texture to the overall experience.
Integrating Stable Diffusion into TouchDesigner was the next crucial step. TouchDesigner, known for its real-time visual processing capabilities, allowed us to take our static images and turn them into dynamic, audio-reactive textures.
By linking sound and visuals, we sought to create an environment where each enhanced the other. The visuals were designed to reflect the mood and feel of our audio work. For example, the serene yet dynamic nature of clouds and storms became a visual echo of our ambient and breakbeat tracks.
Conclusion: The Future Sounds Good
This experiment provided valuable insights into the integration of technology and artistry. AI, rather than replacing human creativity, emerged as a muse and a catalyst for new artistic avenues.
Each AI tool offered unique contributions, valuable for exploration and development. From generating raw materials to providing emotional landscapes, these technologies significantly expanded our creative possibilities.
Our explorations align with the launch of Youtube x Google DeepMind, a collaboration that promises to revolutionise the music content creation landscape. Additionally, there’s Musicfy, an innovative AI musical tool allowing users to craft music with their voice, generate instrumental tracks, and even compose entire songs in a matter of seconds through text-to-music. This initiative mirrors the Studios’ commitment to harnessing new technologies that enhance professional musicians’ work and positively impact the future of music and arts.
Ultimately, our exploration into sound and visual AI has not only streamlined our workflow but also, opened new pathways for creative expression. It highlights AI’s potential as a collaborative partner in art, where the blend of human creativity and machine intelligence paves the way for groundbreaking advancements in audio and visual design.
Kind Regards
Jay-Z
