Cloud Hosting

When he died in 1827 aged 56, Ludwig van Beethoven left his 10th symphony unfinished. Only a few handwritten notes briefly detailing his plans for the piece have survived, with most just being incomplete ideas or fragments of themes or melodies.

Now, a multidisciplinary team of computer scientists at Rutgers University-based start-up Playform AI have trained an artificial intelligence to mimic the great composers style and used it to write a complete symphony based on these initial sketches.

We spoke to the lead researcher on the project, Professor Ahmed Elgammal, to find out more.

Beethoven left sketches in different forms, mainly musical sketches, but also some written notes with some ideas in as well. Previously, in 1988 [English musicologist] Barry Cooper used the majority of these sketches, about 250 bars of music, that were meant for a first movement [in his attempt to complete the symphony].

But what was left behind is really very little. So basically, like three bars of music here and four bars of music there and some rough sketches, which sound like basically the starting points of the main themes in the movements that he [Beethoven] wanted to write.

When you look at Beethoven and other classical composers, thats usually the case. I mean, usually they work with a main theme and develop it into a sequence of a couple of minutes and then another theme comes. Thats the traditional way of composing, and thats exactly what the AI needed to learn how Beethoven and other classical composers start with a theme and develop it. Like in the Fifth Symphony da da da dah. And then take that and evolve a whole movement around it.

Read more about artificial intelligence and music:

The way AI generates music in general is very similar to the way your email, for example, tries to predict the next word for you. So, when you write an email, you find it jumps into suggesting what you might want to write next.

Its the same concept, basically the AI has to learn from a lot of musical data. It asks what would be the next note given what you just wrote? And if you can predict the next note, then you can predict the next note and the next note and so on. Thats the main concept.

But what we soon realise is that if you start picking up the suggestions from the phone for next word and start writing just based on the AIs suggestions, it doesnt really hold for a long time. And thats what happens with music. If you just give it a starting point and leave it to predict, yes, it can predict a couple of notes. But then after that, it becomes nonsense more or less, and is no longer faithful to the main theme.

So that was the main challenge. How can we let the AI stick to the main theme and develop it? So this is where the role of the human expert working with the AI comes in. So we had to work with human experts to annotate and label a lot of music for us to tell the AI what the theme was and where the development of the theme was in a lot of pieces of music. So basically, the AI learnt as a student. That made a big difference because then the AI could really keep sticking to the theme.

Also, the AI had to compose the music in a specific musical form. So if you are composing for a scherzo movement or a trio part of the movement or a fugue etc, each of these musical forms have certain specific structure. The AI also had to learn how to write a fugue, how to write a trio, how to write a fugue, and how to write a scherzo.

It was very challenging because Beethoven only wrote nine symphonies. Thats a very small dataset compared to the scale of what the AI needed to do. So, the way we approached this was to first imagine ourselves like a young Beethoven learning about music. What he would have listened to?

So, we trained our first version of the AI as if it was somebody living in the 18th Century listening to baroque music like Bach, as well as Hayden and Mozart. And so that was the first version of the AI, which basically would be the kind of music anyone living in that era would study to compose. And then we took that and trained it specifically on Beethoven on old Beethoven sonatas, concertos, string quartets and the symphonies as well, so not only symphonies.

We first trained the AI to generate the composition as two lines of music, not as a full symphony, which is a typical way of a composer works by just composing first and then orchestrating. So then, we had another AI that would take that composition and learn how to orchestrate it. I believe this is very similar to the way humans learn you cannot really master fourth-level college without going through the first and second and third levels first. Its always incremental.

The symphony was premiered by The Beethoven Orchestra Bonn on 9 October 2021 Deutsche Telekom

The way we harmonise music is very similar to how we use AI to translate languages. Like when you use Google Translate or another AI to translate a sentence from one language to another. These kind of models used in translation learn a lot of background sentences. So, what is the sentence in German? What is the sentence in English? And from that, they try to learn how to translate them.

So basically, imagine you have these models [for harmonisation]. You put the melody in one side and on the other side you put in how Beethoven would harmonise it so the AI learns how to translate a melody line into harmonised music.

The thing about music is that its very structured and follows a lot of rules. But this is very hard for us to capture and write down. You really have to have a PhD in musicology with a speciality in Beethoven to really understand that. But the machine is able to capture that statistically and mathematically in a very implicit way and be able to use that to give us this harmonisation.

You got it right. That decision is just an extension of the harmonisation. We wanted the machine to translate the composition into multi-track instrumentation, which we also did by training the AI based on how Beethoven and other composers would do so.

Their response is really mixed. There are people who loved this very much, and love the idea of having an AI that understands music and can help you finish your composition or have you explore different musical ideas.

But on the other side of the spectrum, there are people who just reject even the concept of being able to complete a Beethoven symphony using AI. They are afraid of AI taking their jobs and think that it has nothing to do with this kind of thing.

Yeah. I have no doubt about that, we did that in visual art a couple of years ago where we developed an almost autonomous AI artist we had look at, lets say, the last 500 years of western art. The task was basically to generate new artworks that didnt follow any existing style.

If the AI generated an impressionist or a Picasso kind of art or a Renaissance-style artwork, it could realise and so it would have to learn how to create something new.

The challenge with this project was actually the constraints the fact that the AI was not generating music by itself but generating music that is based on Beethovens genius and also following the sketches. This makes it even more difficult. The high bar, of course, of expectation was due to the sketches coming from Beethoven. But when it comes to generating music autonomously I think thats an easier task.

Listen to the symphony below:

Read more about music:

Read more from the original source:

How an AI finished Beethoven's last symphony and what that means for the future of music - BBC Focus Magazine