YuE

YuE: Open Full-song Music Generation Foundation Model

Introduction

YuE, pronounced as "Yu" (乐), meaning "music" and "happiness" in Chinese, is an innovative open-source foundation model designed for music generation. It provides capabilities for transforming lyrics into complete songs, which encompasses generating both vocal tracks and accompanying musical elements. The model has been developed to work across diverse genres, languages, and vocal techniques, making it a valuable resource for artists and creators in the music industry.

Overview of YuE

YuE is built as a comprehensive tool for generating full-length songs. It offers features similar to proprietary models like Suno.ai but is uniquely open in nature, inviting collaboration and democratization in music generation. Users can experience YuE through a demo, enabling them to explore its vocal performance and creativity.

Updates and Licensing

Recently, the model has transitioned to an Apache 2.0 license, responding to community demand for more open access to AI-generated music tools. This shift aligns with the broader movement towards transparency and collaboration in AI, paralleling advances seen in other AI fields such as image and text generation.

• Encouraged Use: Artists and content creators are urged to integrate outputs from YuE into their work, provided they give appropriate credit to "YuE by HKUST/M-A-P."
• Plagiarism and Responsibility: Creators are responsible for ensuring that their work does not infringe on existing copyrights. While YuE facilitates creativity, users must navigate potential legal issues stemming from generated content.

Capabilities of YuE

The YuE model focuses on automatic music generation, where users can input lyrics, and the model outputs a completed song. The following features are key components of YuE:

• Genre and Language Versatility: The model supports multiple languages, including English, Mandarin, Japanese, and Korean. It caters to various genres, accommodating a broad range of musical styles.
• User-Friendly Environment: Quick start instructions and video tutorials facilitate ease of use, even for those unfamiliar with machine learning. Technical guidance is provided for setting up environments, downloading necessary files, and executing music generation scripts.
• Customization Options: Users can input their own lyrics, define musical prompts, and employ in-context learning (ICL) to refine outputs based on existing music. This allows for creativity while enabling users to tailor results to specific needs.

Technical Details

Hardware Requirements

Generating music with YuE will demand substantial GPU resources. Recommended configurations are:

• For GPUs with 24GB memory or less: Run no more than 2 sessions concurrently.
• For full-length song generation, GPUs with at least 80GB memory are necessary to handle multiple sessions effectively.

Execution Time

The time taken to generate audio varies based on the hardware employed. For instance, generating a 30-second audio segment takes:

• 150 seconds on an H800 GPU
• 360 seconds on an RTX 4090 GPU

Quickstart Guide

A streamlined installation process includes creating a Python environment, installing dependencies, and downloading inference code. Users are prompted to execute specific commands to set up and run music generation scripts that make use of the transformer architecture for AI-driven outputs.

Applications and Use Cases

YuE primarily serves:

• Musicians and Songwriters: Artists looking for inspiration or a complete song structure can utilize YuE to generate ideas or complete tracks.
• Content Creators: Those in the digital content space (e.g., video producers, social media influencers) can use AI-generated music to enhance their projects without the need for licensing traditional music tracks.
• Educational Institutions: Music production classes can leverage YuE for practical applications in AI music generation, allowing students to explore new technologies.

Conclusion

YuE represents a significant advancement in the field of music AI, providing an accessible platform for creators to generate innovative music. Its open-source nature, flexible licensing, and rich capabilities present a unique opportunity for musicians and creators seeking to interact with and utilize AI in their workflows. By combining artistry with advanced technology, YuE enhances the creative process and sets a precedent for future developments in music generation.