ai.bythebay.io: Feynman Liang, BachBot: Composing Bach Chorales using Deep Learning
Recording: ai.bythebay.io: Feynman Liang, BachBot: Composing Bach Chorales using Deep Learning
[Music] what you just heard was a four-part chorale however did you know that that chorale there was composed entirely firing artificial intelligence so here I have some instructions for those who are practically minded in case you want to follow along you can actually train and build the model that I will be presenting just by following these instructions right here my talk today will be about Bok Bok Bok Bok goal is to generate Bach chorale so for part harmonies in a manner that is indistinguishable from actual Bach to the average human listener and so today we're going to start by covering some music theory front of music theory primer so that we're all on the same page well then talk about how we process musical scores such that they're amenable for processing by recurrent neural networks after that we'll discuss the model that Bach use and introduce recurrent neural networks long short-term memory and how we apply these in order to encode and process musical scores will then demonstrate some results showing that recurrent neural networks seem to attenuate to music theoretic features such as Cadence's chords and four models and finally well release results about the largest of musical Turing test today expanding over 1,700 participants where we found that only 5% of partner only 5% of responses significantly differed from random guessing I want to start off by talking about music theory with everyone and so when I think of music this is what we're thinking of we have on the x-axis here time and on the y-axis we have pitch so a pitch is how high or how low a note is and notes are held for various durations now what this representation here is called is a piano roll and we can interpret it as follows every bar when it appears is when a note is pressed down and it continues it continues until the bar disappears at which the note is released since we're dealing with four part harmonies at any point in time there is a maximum of four possible bars in a vertical slice now this kind of musical representation may seem unfamiliar or kind of strange in fact you're probably used to seeing music that looks like this groups like this this is much more normal this is standard musical notation and some parts I want to points out here are four part chorale so soprano alto tenor bass corresponding to the four different bar lines pitch here is now represented in terms of how high or how low the note is with respect to each of their bar lines duration is represented by the flags on the notes so in this case the quarter notes have no flags and you can see this crazy little note right here which has three bars underneath this is actually a thirty second note one thing that's also important to note here is the presence of four mottos these little hats right here so that little hat there right after the thirty second note it's a Fermata and what this signifies to the performer is I want you to hold and I want you to attenuate and I want you to hold on this note and I want you to pause four models were used by box to signify the end of phrases phrases are what actually give boxed Corral's a sense of conclusion the sense of progression over time so capturing this notion of phrases is critical to generating realistic sounding core outs awesome all the music that we're going to talk about let's move on to the AI so we start with the corpus of all known Bach chorale provided by the music 21 dataset and the first thing we do is we transpose it so we take all of the scores and we transpose them to C major in a minor the reason why we do this is because it's not really the absolute pitch of a score that is unique rather it's the differences in pitch that really give up or that really is define what of what a musical piece is you could sing happy birthday at any tone you'd like it's still happy birthday we also quantize so notice that the 32nd note the note that has three bars under it has now become quantized to a sixteenth note this is done in order to turn this is done in order to turn notes into a sequential into a sequence of frames where by which we can then up which we can then process if we do not quantize then you wouldn't be able to process this sequentially using a discrete-time recurrent neural network here's the question for the reader here's a question for the audience I have four parts the soprano alto tenor and bass each of them in standard MIDI encoding can play one of the 128 different pitches how many possible frames could I could have how many possible frames so simultaneity is with up to four voices can be in kit can be created the answer to this is on the order of 128 there's some there's some adjustments you know the soprano can't play lower than the alto so but on the order of its scale 128 number of notes exponentiated to the number of parts so for high degrees of polyphony so if you have more than four voices which is very common in modern music it actually becomes intractable to represent the entire vocabulary of possible chords so we can't actually use a frame based representation where we model every sequence at every point in time as an individual token inside of our sequence how do we deal with this well this problem also arises in in in text and text classification one way you can classify texts is by individual words but another you could also imagine processing individual characters of words there over time and that's the exact analogy we take rather than representing individual frames as like four parts simultaneously we break it down into individual parts by ordering it soprano then also then tenor then bass followed by a frame delimiter and so it so the promising looks like this we start at the very top note the soprano note and we work our way down until we get to the bottom we then omit this triple bar thing frame delimiter and then we go to the next note and so the encoded sequence looks as follows you have a start token you have the four notes and you have the frame delimiter but there's one thing I have neglected to mention and that's how do we represent duration and so that is actually what you see right there with those boolean is following the note numbers true means that the note is articulated at this point in time whereas false means it's actually a continuation from the previous note in the previous time frame so if you look right here we see that the truth indicate that this note has been played at this point in time it is newly articulated so we do this to every single every single Bach Corral that we have in our data set and we arise at a corpus where the vocabulary size of 108 this is a little surprised this is not too surprising it indicates that not all 128 possible pitches were actually utilized after we had transposed things to c-major however we see that there are about four hundred and twenty three thousand tokens in our in our entire corpus it's a pretty big size and it makes us excited because now we can consider more sophisticated models such as recurrent neural networks and deep recurrent neural networks if you're following along with the doctor demonstration what I've described here is implemented by the following command so transposing quantizing and encoding into the sequence happens in this first left part of this sequence diagram with the command box lot datasets prepared you then receive these encoded sequences encoded sequences with the dot UTF extensions where you call václav datasets the concatenate corpus and that does is it takes all of these course all of these encoded scores puts them into one hdf5 file which is then used for our training data set awesome we have our encoded corpus the next step is to model it to model this we're going to use a model called a recurrent neural network which I'm sure you've heard plenty of today so I'll go kind of quickly through these slides this is a neuron it's the fundamental unit of computation of our current neural network it takes a d-dimensional input Kate and then and then linearly combines them using the weight 2w and then it passes sums them all together to get this activation Z and then the activation Z is passed through a sigmoidal activation function Sigma in order to produce the output Y you can put a bunch of them side by side and you can start stacking them into layers and this form is what's known as a feed-forward Network where you have your input layer which is the tokens and then there's some sequential units of calm composed excessive hidden layers to finally resulting in the output layer a recurrent neural network takes this idea and adds the notion of memory so again we have our input here represented on the bottom left here but we have this new thing called the memory cell and what the memory cell is doing is it's taking the hidden layer activations HT from the previous time so that Z minus 1 is a unit delay and it passes it back in as inputs for the current time so in effect this hidden state is being propagated forwards in time and it acts as a form of memory allowing us direct allowing us to remember where we are in the composition and helping us to generate more realistic notions of phrasing conclusions and conclusions to phrases I've taken that I've taken that memory cell and I just put it in this little shaded cell called H there yeah so I've taken a memory cell and I've compactified it into this H unit right there and if we stack them together so we take the hidden units from one layer and we put it into another hidden unit layer we have the stack recurrent neural network on the left hand side of course that looks a little strange we don't really see z inverses when we see recurrent neural networks instead we're used to looking at this unrolled format on the right hand side so on the right hand side what I've shown is the sequence of tokens on the very bottom that x1 to xn and on top of it I've shown the first hidden layers the second hidden layers and finally the output of this recurrent neural network to train this network we trained it to predict the next possible token inside of our sequence of tokens and this is done using the back propagation algorithm where we back propagate the error signal through gradients up to the up to the network model parameters and we step the model parameters using stochastic gradient descent to optimize the next note prediction of course what I've described is something that's known as a Jordan type recurrent neural network you can have more sophisticated hidden units and so in this case the memory cell is used we actually utilize in baccarat is a long short-term memory this has been talked about in previous presentations it is basically the constant error carousel is the Jordan RNN we've looked at previously but we've added an input gate of forget gate and an output gate represented by I and O if you're following along the command which actually implements this recurrent neural network and trains it from the concatenated corpus is called bak bot train and it utilizes torch RNN under the hood okay after you have the trained model you can then call bak bot sample and that what that will do is that will sample the model which the sample the model in order to generate novel compositions and the sample model is still a sequence of tokens so you can finally call BA kua D code and that will take the sequence of tokens and decode it back into music which you can then listen to so you're following along run this command and you should you should be patient you should be able to produce your own novel compositions awesome so we did exactly what we were described inside of the practical inside of the practical instructions and here are the results we found as expected drop out causes higher loss in training but lower loss and validation this is expected because drop out turns off hidden turns off connections at random essentially forcing the network to learn features which are robust and not code codependent on other features and so we find that the best dropout occurs with 0.3 so a 30% probability of turning off any unit during training and that's represented by the purple curve which has the lowest validation loss but definitely not the lowest training loss the lowest training loss is actually achieved with zero dropout this is a little more interesting here I'm showing the activations of the individual neurons as we promised a frame as we process a score where's my mouse so this is a little hard to see but what I'm showing the top here is this is the input to the network and here are the activations of the network over time the first layer the second layer the third layer the fully connected layer and the outputs of the network now there's some obvious patterns you see these little smears right here those correspond to rest tour notes that are held at the end of a phrase and that little smear seems to seems to show up across all of the different layers but you know other than that when I look at these when I look at these hidden unit activations I couldn't tell you very much about what's going on there's a common criticism of deep learning models that they're black box and not interpretable but surprisingly produce great results and indeed if we look at the very bottom the outputs of this network we have here what kind of looks almost like a piano roll this shouldn't be very surprising the network was trained to predict the next note predictions given the current note and the current hidden state so if the network is doing perfectly you would actually expect to see the input reconstructed but delayed by one time unit in this very bottom layer so somehow this jumble of noise produces pretty good the next note predictions here that's flat I find particularly interesting what I'm showing up top here is an input score represented in piano world notation and here I'm showing some some neurons and their activation values over time now these neurons actually seem to be learning something about music theory for example um I believe this is actually a g7 chord that this neuron is it's very selectively firing on and this neuron here seems to be doing the same there's other neurons there's other neurons being shown there's other neurons shown here which are more fully described in our paper but the conclusion from this slide is that these neurons seem to be adapting to learn music theoretic features without any form of prior supervision we didn't tell the model about what a cadence is what a chord is what a tonic is yet the model has chosen to learn these sorts of features in order to help itself optimize on a composition I'm not at a composition task finally we demonstrate that the model is able to compose realistic sounding music by generating some samples from it so what you heard at the beginning was one sample [Music] you might recognize that song it's obviously not saw but we've still been able to but this demonstrates the flexibility of our model by clamping the soprano notes to be those of twinkle twinkle little star we're actually able to generate a baroque harmonization of what twinkle twinkle little star might have sounded like if Bach was tasked to harmonize it finally I want to share some results about what happens when we can when we when we put up Bach Bach calm which you're all welcome to go to and test yourself on it's a musical Turing test where participants are tasked with discriminating between is this sample real Bach or is this sample generated by Bach pot and so we put the stuffs online received about 1700 actually almost eighteen hundreds as of today responses most of them come from the US and the UK where this research was conducted but Curtis would span all over the world here's a breakdown of participants by their age and their music experience something that's interesting to note is that we actually have a surprisingly large number of advanced participants so advance is defined as someone who is either formally studied music theory and an expert is someone who is designed as a professor or teacher of music theory and we can see that their participant amount is non-trivial this is not something you would expect if you were to run this study on Amazon MTurk here are results these notes this code down here designates the problem type so s means that Bach law was tasked with harmonizing just the soprano part given the other / with the scrander part given the other parts and same with a Phoebe 80 means that the soprano part and the bass part were provided and Bach bought was asked to fill in the intermediate parts the two in between and same with a TV it was just given the soprano part and it was asked to compose a harmonization finally SAT B means block wat was not clamps to do anything and it was asked to tap to generate its own harmonized its own compositions the result that's important that we should all look at is this one right here what this is showing is that on average participants are only able to differentiate box from box bought seven percent better than fifty percent now if I give you a discrimination question with two choices and you randomly guessed you would get 50% and so what this is showing is that buck and václav the gap between the two is only amongst seven percent of question respondents of course here you see the expected trends a few segments further by music experienced with increasing music experience you're able to differentiate box and box lot better novices so casual listeners of music are only able to differentiate Bach from Bach bought three percent better than random guessing and something that I find a little anomalous is this base column right here where the trend that increasing music experience leads to better discrimination does not hold for some reason experts are doing significantly worse than every other participant group one possible explanation for this is recall when we ordered these notes sequentially we had ordered them soprano alto tenor bass and so what that means is by the time bot bot is tasked to produce what the bass note might be it has already seen the soprano alto and tenor note for that harmonic context so it may be easier to generate the bass parts than the other parts so what can we conclude from this um Bach lot is a three-layered deep LS TM and what this shows is a deep lsdm model can compose complete and generate polyphonic music by the way Bach bot has also been integrated into Google magenta a tensorflow project that's focusing on generative art if you look at the polyphonic RNN model in magenta it is the Bach Bach model the model appears to learn music theoretic and not notions without any form of prior knowledge or preemption so we just asked it to optimize an automatic composition task and it shows to use this notion of chords and Cadence's and tonics in order to do well and finally we conducted the largest musical Turing test to date with over 1,700 participants and our results suggest that the average participant can only differentiate Bach from Bach lot 7 percent better than random guessing that's the end of my talk thank you so much for listening [Applause] [Music] thank you very much for this talk by the way my girlfriend studies music and I'm sure she will like it my question is maybe I didn't understand that really how do you synchronize bass - alto and tenor and so on that's because you were saying each of these was or was looked at individually doesn't even matter how how they're composed or do they do they have some kind of relation to each other so we don't explicitly encode any notion of synchronization between the four parts we kind of cross our fingers and hoped that after seeing enough data the LS TM would learn that if the bass note is player if the soprano and also note are playing these things then the tenor and bass notes ought to be playing these things so there's no explicit synchronization mechanism rather you know the hormone the harmony of any frame is really learned from data thank you any more questions I thank you for the presentation and a quick question when you came up with a 57 percent success rate or I don't know how you want to call it did you take into account the fact that some of the people you asked they are they actually know that back compositions so they think that's the reason they can actually tell yes that is a very valid concern we did not account for that that perhaps explains why experts performed much better than novices who are unfamiliar with the corpus we had actually done a follow-up or we actually segmented the responses and their discrimination percentages by question type and when we had a music theorists look at it they found that people were differentiating well-known Bach Corral's better than the more obscure Bach Corral's so that is a valid concern we didn't address it in the aggregate statistics reported any more questions that's a short question I hate to have you walk the mic over I wanted to know you know you obviously built bought bribe do you have plans for building other musical musician BOTS or impersonators I don't personally have plants I know that so magenta has implemented Bach bot as their polyphonic RNN model and they actually have a variety of corpuses that they provide so they have one for like 8-bit game music so if you are interested in training the Bach bot model on any of these polyphonic data sets I'm sure magenta has provided really great frameworks for doing so any other questions we have time probably for one more hi what's the the input to the model do you require it to be already transcripted so it's are you processing MIDI files or is there a step you know to transcribe audio or take regular music notation and turn that into those blocks um so we decoupled the actual transcription of music into the generative modeling of it so we assume as input MIDI music XML pre transcribed music that we can already could like you know run software packages on [Music] you