The first domestic music SOTA model is here! Optimized specifically for Chinese, free to use, no restrictions on genres

On the first anniversary of the release of the "Tiangong" large model, Kunlun Wanwei announced that the "Tiangong 3.0" base model and the "Tiangong SkyMusic" music model have officially launched public beta .

Since AI has allowed humans to achieve the freedom of music creation, even quarrels have become interesting.

In the past, Aran Komatsuzaki, a well-known AI blogger on the generated. You know, in the past, the war of words between these big guys was mainly to post a post, and then you and I would follow up. This time, Aran Komatsuzaki's approach can be said to have played a new trick. I don't know if it was inspired by "Thank you, thank you, I want to diss you".

Like Aran Komatsuzaki, many people at home and abroad who know music but don’t know music are trying to play AI music creation tools such as Suno, and have generated many very interesting music works. .

However, many netizens have reported that Suno sometimes generates Chinese that is not stable, and there are problems such as Chinese songs having an English feel and uncommon words being sung incorrectly.

#                                                                                                                                                                                      . Video address: https://b23.tv/gVqTUOu

So, is there an AI music generation model specifically optimized for Chinese?

Kunlun Wanwei’s “Tiangong SkyMusic”, which is open to the public for public testing today, is such a model. The Chinese vocals it generates are clear, authentic, and have no abnormal sounds, and there are no "Broadway-style Chinese songs" and other acclimatization problems. Moreover, it is not only optimized for Mandarin, but also takes care of dialects such as Cantonese, Chengdu, and Beijing.

So, how does SkyMusic perform compared to Suno? Horizontal evaluation data shows that SkyMusic is superior in several indicators such as vocal and BGM sound quality, naturalness of vocals, and intelligibility of pronunciation. Its comprehensive performance surpasses Suno V3, becoming China's first music AIGC SOTA The model also made China's self-developed large model technology lead the world in the AIGC field for the first time.

Such excellent performance is naturally inseparable from the powerful base model, that is, the large model "Tiangong 3.0" released and open sourced by Kunlun Wanwei at the same time. This model has 400 billion parameters, surpassing Grok-1 with 314 billion parameters, and is the world's largest open source MoE model.

In many authoritative multi-modal evaluation results such as MMBench, “Tiangong 3.0” surpassed GPT-4V and took the lead in the world.

With the support of this base model, SkyMusic has a deeper understanding of songs. It can control emotional changes through lyrics and implement various singing techniques such as vibrato, opera, chanting, etc., making the generated musical works more emotionally rich and suitable for the situation.

So, how to use this model specifically? What is the technical route? What is the innovation behind "Tiangong 3.0"? Let’s look at them one by one.

The first domestic music SOTA model with unlimited gameplay

In fact, it is very simple to use SkyMusic to generate songs: you only need to enter the song title, lyrics, and select a reference track, and it will generate songs with similar style and vocals.

#If you don’t want to write lyrics yourself, you can also try the “AI writing” function in the lower right corner of the input box. It can be written from the first sentence, and only one sentence is generated at a time. Unsatisfactory sentences can be deleted in time until the entire song is completed.

Of course, you can also try to use "Tiangong 3.0" to write songs. For example, this song "This Site" was written by us using "Tiangong 3.0":

The next step is to select the reference song. This is also the unique feature of SkyMusic, that is, it can generate music based on the sample sound source.

In this step, SkyMusic provides many reference tracks, you can choose from them, or you can choose to upload song files. Here, we have uploaded a song by Luo Tianyi to see how the generation effect is: This ability to generate music based on sample sound sources has greatly enriched the gameplay of SkyMusic. In the user works display area, we can see that "New Man" (the episode of the movie "Zhou Chu Eliminates Three Evils") has five versions including ancient rock and DJ. During the trial process, we also discovered that, in fact, the music generated by Tiangong SkyMusic covers rap, folk, funk, ancient style, electronic and other genres. In the next step, the team also plans to allow users to generate songs based on hummed melodies, which will be of great help to professionals.

Currently, Tiangong SkyMusic is fully open. You can experience it by downloading the "Tiangong" APP. This is the only publicly available large-scale AI music generation model in China, and its emergence fills the gap in domestic AIGC tools in this field.

Although this model is still in its infancy, it has already allowed many people to experience the joy of music creation. Everyone uses it to transform divine comedies, create second hits, rewrite ancient poems to support education... and develop various new directions for music creation.

Self-developed Sora architecture, technical roadmap has been made public

Tiangong SkyMusic is an end-to-end music generation model, so it feels very simple for us to use it. However, the development of the entire model is not that simple.

First of all, from a technical perspective, SkyMusic chose the large-model music audio generation route, which means that it directly learns and generates audio waveforms instead of using symbolic music generation routes (such as MIDI) to generate musical scores. . This approach allows for integrated end-to-end generation of elements such as instruments, vocals, melodies, volumes, notes, and more, providing a more direct, higher-quality music creation experience. However, this direction is also more difficult and requires high computing power and funds, so there are very few people doing it.

Moreover, in this direction, there are even fewer people who are determined to conquer the field of "human voice Song". Most research focuses on the field of BGM without voices, because there is almost no open information or open source in the former. Models are available for reference.

Withstanding these pressures, Kunlun Wanwei has conducted countless research and development experiments, invested a lot of computing power, and constructed a data set containing 20 million songs (the largest music data set in human history), and finally explored An effective and reproducible solution was developed. Moreover, they also made the technical schematic diagram of this solution public.

^{Tiangong SkyMusic technical schematic diagram: Large-scale Transformer is responsible for composing music, learning the contextual dependencies of Music Patches, and completing music controllability sex; the Diffusion Transformer is responsible for singing, and the Music Patches are restored to high-quality audio through LDM. This model architecture works extremely well when processing video, audio, and music.}

As can be seen from the picture, the framework of SkyMusic is a DiT technology path similar to Sora, but the development time was before Sora came out, so it is inevitable to step on many pitfalls.

For the industry, this schematic diagram is very valuable, because there is no large-scale AI music model company on the market that has disclosed its own technical path, including SUNO, and Kunlun Wanwei is the only one.

The powerful base behind it - Tiangong 3.0

The success of SkyMusic is inseparable from the base model behind it - Tiangong 3.0. Its core upgrade is reflected in "independent thinking". This is reflected in the model's new multi-round search and comprehensive tool calling, chart drawing, research mode, enhancement mode and other functions.

Given a data statistics task, it can not only help you collect all the data, but also write your own code and call various functions to draw charts. Each intermediate step has been dismantled in a clear manner, and subsequent execution requires little human intervention. Even details such as "avoiding label overlap" and "centering text display" have been taken into consideration. This is the embodiment of the ability of "independent thinking".

This improvement in "independent thinking" ability is inseparable from the optimization of "Tiangong 3.0" in semantic understanding, logical reasoning and other aspects. Compared with the previous generation "Tiangong 2.0" MoE large model, "Tiangong 3.0" has amazing performance improvements in areas such as model semantic understanding, logical reasoning, versatility, generalization, uncertainty knowledge, and learning capabilities. The model technical knowledge ability has increased by more than 20%, and the mathematics/reasoning/coding/cultural and creative abilities have increased by more than 30%.

Take the search task as an example. In the "Search Enhancement" mode, given a simple search request, "Tiangong 3.0" will not only give a summary answer, but also refine some important information into charts.

In the "Research" mode, it will also provide an "In-depth Research" module to discuss extended issues not mentioned in the search query, giving you a The feeling of reading a paper. Finally, it also organizes this information into a mind map for quick reference.

On the basis of greatly improved semantic understanding and logical reasoning capabilities, "Tiangong 3.0" also focuses on the model's ability to independently plan, call, and combine external tools and information. Received special training. By independently planning and calling and combining external tools and information, it can help you accurately and efficiently complete various complex needs such as industrial research, product reviews, and information analysis.

This ability to think independently is crucial for large artificial intelligence models. Firstly, this ability enables AI to perform autonomous reasoning in the absence of direct instructions, improving its ability to handle complex problems; secondly, independently thinking AI models can design innovative solutions to meet personalized and scenario-based needs. demand; finally, this ability enables AI to continuously optimize its performance through self-learning and adaptation when encountering novel or changing environments. The cumulative effect of these three aspects has greatly promoted the breadth and depth of application of AI technology, making it more intelligent and efficient in a variety of practical applications.

"Tiangong 3.0" includes many large model capabilities such as AI music, AI search, AI writing, and AI painting. Talking about the 400 billion parameter MoE large model "Tiangong 3.0" and the technology behind Tiangong SkyMusic At the time of the relationship, Kunlun Wanwei Chairman and CEO Fang Han explained, "Everyone knows that the AI ​​base large model is the solid foundation of AIGC, especially the text large model. Vincent Picture, Vincent Music and Vincent Video (these AIGC models) are the basis of capabilities They are all large text models. If the text model capability is not strong enough, AIGC capabilities will be greatly limited."

This effect is also reflected in the AI ​​painting and other capabilities of "Tiangong 3.0". "Tiangong 3.0" has added new functions such as image size expansion, image orientation adjustment, mat image generation, mat image evolution, and mat image expansion, and the actual test results are excellent.

"Our 400 billion large model is the base model that provides support for all our C-end products. The better my base model is, the better my base model will be. Music, games, videos and animation products will do better, so we have a very strong motivation to make large base models." Fang Han said.

Realize general artificial intelligence,

Let everyone better shape and express themselves

In our vision about AGI, we I often hear some AI company executives mention that they want to use AI tools to improve the productivity and efficiency of human society. Therefore, they mostly focus on the expansion and enhancement of model intelligence. But in Fang Han’s view, an important issue has been overlooked, that is, how to use AI to help people better understand and express emotions.

In the music section of SkyMusic, we have seen many examples of this: the sorrow of students about to graduate, the sorrow of young people unable to love, and the fatigue of middle-aged people supporting their families are all expressed through singing. This is the real "singing to express one's ambition".

The support for dialects is more like a kind of cultural equality, which is Fang Han A very important point. In the future, they hope to add more languages ​​so that every person speaking a small language can easily create their own cultural content.

"In the field of AIGC, our ambitious goal is to hope that everyone in the world can create content equally. We want to lower the creative threshold for everyone, so that everyone can better shape and express Self." Fang Han said.

Recently, these contents have also been written into Kunlun Wanwei’s latest mission.

In fact, this approach also makes sense commercially. "Once everyone can create music, I believe that any public place, such as every restaurant and bar, can create their own background music to meet their own business needs." Fang Han explained.

With continued optimization and improvement in the future, SkyMusic will gradually evolve into a professional and easy-to-use national music creation platform.

Of course, Kunlun Wanwei’s efforts are not limited to music. Based on “Tiangong 3.0”, they have formed six major AI business matrices. In the future, these matrices will form an AI UGC platform.

#This platform can not only help ordinary people express themselves, but also help creators who want to use AI to create content to complete a fully closed loop of IP creation. This closed loop takes "good stories (IP)" as the core and spans various forms such as text, comics, music, and videos. Consumers' content consumption is also completed on this platform. This is the business logic built by Kunlun Wanwei .

"Our essence is to allow more people to join the team of creators. The premise is that you must be able to tell a good story. If you can create a good IP, then you can create content. Yes. It is foreseen that all content industries will be redone. The number of creators will expand a hundred times, and the content that can be consumed will also expand a hundred times. This is why we propose "All in AGI and AIGC". The logic of strategy." Fang Han said.

What will this era be shaped into? We'll see.

The above is the detailed content of The first domestic music SOTA model is here! Optimized specifically for Chinese, free to use, no restrictions on genres. For more information, please follow other related articles on the PHP Chinese website!