Machine Learning on Source Code

Scale By The Bay 2018: Francesc Campoy Flores, Machine Learning on Source Code

Recording: Scale By The Bay 2018: Francesc Campoy Flores, Machine Learning on Source Code

you I don't have this on but I guess you can hear me so we go so yeah I'm Franciscan boy and that's not on the slide so no what I point there but today I'm gonna be talking about machine learning on source code there's a little bit of an introduction of why is this even a thing so I'm sure you've all heard this and this phrase software's in the world and I this has been going on for a while but it keeps on accelerating and I didn't realize how much source code there's in the world until I actually started trying to figure out how much source code is there outside right like in open source but also not even open source how much source code do we have in the world and I started by looking like prehistory or at least pre RGB and you look at Adobe Photoshop this was a pretty powerful tool even though there were no colors and this was 128 thousand lines of code and for me a programmer as a programmer this seems like a lot but also as a user you know it's not that much Windows 3.1 in 93 four to five million lines of code and this was pretty serious I used to play Prince of Persia on this so you know that works OpenOffice nine million lines of code chrome 18 million lines of code you will recognize Windows XP I was gonna say Vista Windows XP 45 million lines of code and this starts to be decent amount of code right and this is a fourth car but pick up huge pick up 150 million lines of code running in the car yeah so when you think about 150 million lines of code driving the car that you think you're driving start things start to get scary right and when you think about the fact that basically no matter how good you're writing code there's bugs right say there is one but one critical part per million lines of code you have 115 that in that car right that's scary and if you look at Google 1 billion line of code and if you ask github they will tell you they don't know and actually if you keep on asking more they'll tell your dates somewhere around like four to five billion lines of code but they don't really know because they don't comment which interesting the problem with this is that yeah the code keeps evolving and we we keep on having more and more lines of code which is great but we keep when you need the same tooling right yes we have better editors sure but those editors other than our completion they don't do much and especially editing the code is just one of the pieces right we have great tooling now for once you want to test things continuous integration continuous deployment all of these things have helped a lot the industry but the editing power of things well yeah there's women Emacs MVS code and all of these things but it doesn't really have changed that much and if you think about the way not the way we write by the way we review code it's even worse how do we review code well you read it good luck right there should be better better ways to do this and we believe it can be machine learning with all tools you can do cute stuff with cute dog inside and it will probably survive unless it's maybe Linda but that but the house I'm pretty bad at that but if you try to build something as big as Google you're gonna need better tooling right and that's where we're trying to do is apply machine learning to the previous tooling we had to build better products so now yes I am Franciscan boy and I'm gonna be talking about machine learning on source code I origin create this talk for gopher con and it was a much shorter version and people thought it was machine learning in go it was actually machine learning on go so we're not gonna be talking about how to write machine learning but actually how to use machine learning to write code so I'm VP of Product and developer relations at source which is a tiny startup we are now 35 people and we work on this this what we do machine learning on source code machine learning for large-scale code analysis and I'll show you a little bit what what we do but this is really not a product pitch this is about like what we're trying to do and like the ideas and projects that are wrong I'm Franciscan Twitter and if you want to use hashtags scale SF is apparently the one for the conference and ml encode is the one way years so we're gonna be talking about what machine learning on source code is then we're gonna talk a little bit about the research and where it's going and then we're gonna be talking about the way this actually works for us right like what can you do there's machine learning is super cool there's a lot of really advanced techniques but what does that mean for us and then we're gonna be talking about okay in the future assuming that everything goes well what could we do what will the what will the world look like for developers in 20 years so what is machine learning on source code simple it's machine learning that you apply on source code that's it right machine learning on source code is simply the fact that instead of having the input data being say images or tags or videos or anything like that you get source code that's it if you're doing that that's machine learning on source code I don't know exactly about what it is if you're fancy you might call it a ion source code I don't care we call it MLM codons any ml technique applied to source code so what are the fields that are related to this data mining is one of them why well when you have billions of lines of code that you want to analyze that's a lot of text that's a lot of data right you want to analyze it correctly there's a lot of things that you can learn from data mining text human text and programming languages are different but they're not that different they actually share a lot of aspects right they have grammars they have syntax all of these things coexist although they're not exactly the same I will talk a little bit later on why there's that if there's there's a little bit of a difference on the way we we handle natural language and primary languages but we can learn a lot from an LP natural language processing most of the techniques that they use for natural language processing can be adapted quite successfully to machine learning on source code and then graph based machine learning because source code is graphs source code if you think about go to definition that's a relationship that is over a tree and all of a sudden there you go now you have a graph right so we need to do graph and machine learning on top of those but graphs all the time which is really interesting and actually quite hard I'm VP of Product and developer relations so I don't I don't do those things that the VP of machine learning that does all of these cool stuff but I know enough to know it's really hard so yeah what do we need for this lots of data luckily we have it but we need to fetch it but that date exists there's lots of data really really lots of data you need some fancy ml algorithms and then luck hopefully everything works not always when you do machine learning there's a lot of different aspects that can change a little bit the way a model is gonna it's gonna act so there's a lot of trial and error involved in anything related to machine learning and also machinery on source code so first challenge data retrieval we have billions of lines of code out there how do we fetch them git clone is not the solution so there's a couple datasets that are very useful one is github archive or gh archive I think they cannot technically say github that's why they call gh archive and that contains a lot of metadata about source code on github what we've done is we created a different one which is public archive that does not contain all the metadata but also all of the source code on github for repositories with 50 stars or more that's a lot repositories that's 50 now let me see replaster is 180 mm repositories compared to previous efforts that were like one order of magnitude smaller it's around 3 terabytes of source code and it does not only contain the current source code but actually all of the iterations before you have all of the blobs in git repositories in order to do this there's actually a lot of fancy things that we do one of them is what we call rooted repositories github has an API that allows you to know whether two repositories are a fork of each other that unfortunately is pretty useless because most people will get clone and then get pushed somewhere else and the API will never know about this right so we actually do an analysis on the commits to figure out whether two projects are independent or not and this allows us to minimize a lot like red blobs only ones all these things there's much more than that but it's the least interesting part just fetching the data the second part is analyzing that data and these were things start to get interesting because when you think about source code what is source code like if you're going to analyze this how do you do it and the easiest way is to say source code is bytes that's it you write with bytes and the that is the code right that's it you can fetch this you can fit these to a machine learning model see what happens and it actually works which is kind of crazy but this is like the lowest level that you can say like code is text which is parts cool but also you can say actually that's not high read the code right I read package I mean I do go all the time so I'm sorry for people that do not know go is pretty straightforward do not do not worry too much and there's nothing fancy to do so I rehear package main that is where every those are tokens sure so let's say you can also have tokens so tokens a different level of abstraction and this allows you to allows you to give more information to the machine learning model that you're training you also have abstract syntax trees and as soon as you start adding things on top of an abstract syntax tree what you get is a graph you have many many different kinds of graphs you have control flow graphs you have definition graphs semantic graphs all of these different kinds of graphs you want to learn from those and it's actually really interesting these were most of the researches right the first levels you can do fun things and they're cute but this is where the research actually goes so yeah so we have sequences trees and graphs so it goes from easier to understand and easier to do to actually quite complex so all the things that you need to do when you want to analyze code well first thing is you need to know what language a program is written in that's the first step the second one is you need to be able to parse that language then you need to be able to extract the pieces that you're interested in and then you're also may want to do that over time right so say I want to find all of the function sure when right head a month ago what commit do you want all of those things are also very important and then reference resolution which is basically you're calling the function foo where is this defined what is food right and that could be in the same file could be in a completely different repository you never know we have a bunch of different projects that we implemented for each one of them the last one for reference resolution it's called kithe and we did not created was created at google and instead we hire the person that created the project that was easy it was not easy actually but it worked well so these are a bunch of different things that we've created over time and what we've done is we put everything together in a single product and that we'll call source engine just quick demos so babel fish is that's pretty right so bow fish is the project that allows us to extract to parse any language and extract organs from it so say you have some java code and here on the left side you have the java code on the right side that's an abstract syntax tree and we call that abstract syntax tree universal the universal is just about the fact that there's a common format for all languages it doesn't mean that all abstract syntax trees look the same but rather than we have a protocol buffer definition for it right it's pretty much it and then there's also language agnostic tags or roles so for instance when you say this hello world is gonna say that's a node that has a bunch of roles it's an argument that is positional and the type is string right that type here is the string it is something that you can query for string string and now you're gonna get only the strings right so you can see on the right side those two nodes are those other strings they have why is this cool wait this is cool because actually let me copy this if now I go and I use go and this and I do exactly the same query we do exactly the thing and now we have right so what we've built is actually a way to parse primary languages in a language agnostic way which sounds weird but it's actually really powerful because if I want to say I want to use all the variable names I do not care whether this is Java PHP or go variable name is pretty well defined right so just find those and that's how we're doing it there's the project then we've integrated that also in a sequel environment so you can do cell select star from repositories and this is actually analyzing a bunch of the repositories there's 30 postures here that is all of the source code under the kubernetes under the kubernetes thing the kubernetes organization and github so in that is sequel which is pretty straightforward but the cool thing is that as soon as you have sequel you can stop using sequel and use Jupiter which is much cooler and you can start doing quite interesting things I will not get into the detail of how all of this works but you can do let me show you the one that is interesting is the one with colors so here for instance you have this query and this is saying hey I want to find all of the repositories over all of the repositories I want to find all the files that are in head and then I want to find the language that language function is something that we've added to to my sequel the custom build function it tells you what language it is so with that query after five seconds you're able to tell that that is not up to date at all because that is not Cuban area kubernetes this from a previous analysis that I did but you're able to see what are then what are the common languages did you find you can do the same thing put in a little boom little bit more complicated way using that us the concept that we talked about before and say I want to extract this thing function group slash alias clashing the fires those are the function names I want to extract all the function names no matter what prom language this is raining and then you're going to be able to generate things like this where you have word clouds for different languages write those word plus have been generated with a single query we no common language disease why is this important well this is important because the next step is going to be getting all of these data and using it to train models if in order to do machine learning you're asking the machine learning scientist to first know how to parse code they will never finish right so we have two different pieces we have data retrieval which that do this these kind of things then we have machine learning which is the second part before I continue are there any questions about this part so far yeah we we have basically all the languages that are like well-known and I same as a say visa I don't know we have Scala which is very sad for this conference but I think we do but we do have like Java go PHP will be all of those the latest one that we had that we were missing was C C++ because parsing C is really hard and it's parsing C++ is basically impossible we finished that recently so those are other than that basically any language that you can imagine is pretty we probably have it and if not it's pretty easy to add that's the whole point every every parser is actually written in any language you want which makes it makes it much easier because parsing Java and Java is much easier than passing Java in say PHP right you can choose the language that works better for your problem which we can talk about that a little bit later because it's kind of out of the scope but that's exactly what I does kai is a project that was Korean Tony at Google to create cross references across all of their code base the code base Google 3 that contains a billion lines of code right so doing cross-references over that data set is really hard and they were able to do it and now it's open source learning from Sasuke that's the next we're getting there yeah okay so let's continue with the learning side of things like so far yes we've talked about how to fetch source code and how to analyze it now the next step is learning from that source code and see what we can actually build with that so in order to learn from source code one of the things we're gonna be doing is neural networks and I'm not gonna get into detail what neural networks are because it's either I do not explain much or we spent a year here so I'm just gonna say that neural networks are a very very useful piece of technology that allows you to basically by the power of insisting on know the result should be this finally actually convinced than your network that the result is actually that right I like to use it I use them at the metaphor or the analog of it's like a puppy that you need to keep on saying no no no and eventually learning is able to do things that are useful here we're gonna do kind of the same so the this the amnesty data set that I'm sure if you've done any machine learning you've seen where we're getting an image getting off the pixels and feeding it on your network we don't really care about the structure the architecture and that's going to tell us this looks like an 8 and that's how it comes up it's around 0 around 1 not exactly 0 not exactly one because there's their prediction and this is actually something we're gonna use in a minute the whole point of neural networks and I'm gonna get too much into details is the fact that we want to learn things that are actually not necessarily doable with algorithms why is this if you want to do language analysis very often you need the full source code right you need not only the source code that you're not izing but also all the possible dependencies that is actually really hard to achieve it is very very complicated to do it / salt / open-source and especially depending on the bill system every single prominent which might do this differently right so that is really really hard what we're trying to do is using machine learning be able to do some predictions that are based on partial information that's why we're doing machine learning rather than traditional language analysis so one thing that you could do is say we I'm gonna give you ten tokens you predict the next one right so if you see four icon equals zero I less than ten after the I is probably a plus plus it doesn't mean that it's necessarily a plus plus but probably right because if it's a minus - that'd be pretty weird you would be counting down while comparing to something that is up so these things your network is able to predict quite well what if you don't want to say ten tokens but actually if you want to do something over sequences we talked about the fact that code can be seen as a sequence of characters or sequin of tokens well in that case you're gonna start using recurrent neural networks and these are very useful for a bunch of different domains one of them is language translation right language translation you're gonna translate a sentence you don't know how long it is from a language to another language and that's gonna be some other sequence there's not and the sequence that enters and the sequence that exits is not necessarily the same length right so it's gonna be much more complicated but we're currently networks is something that we use all of the time and you can use this for cogeneration and I don't know if you've seen this before it's a typical example with people do with Shakespeare I decided to do it with go so what I did is I use a recurrent neural network and I trained it with the ghost on the library and I asked it okay now you know go write some go these before training which is not exactly go it's not valid completely after just one epochs of just seeing the data set once you start saying things like this and this is quite interesting because this actually looks like go it is it will not compile it is nothing tactically correct but if you look from far away I would say this is go especially the if error is not nil return that's the must-go ish line of code ever we have go but still there's not totally perfect right and you keep on training and it does think that they actually look like go but these now go right like these will not compare will no compiled there's things like for instance I mean it does do the indentation correctly the parentheses the bracket most of it is good but for instance that if not ot close that's not go right so it's it looks good I'm still not perfect if you keep on training it eventually you get to something that is pretty decent but never good enough I the problem is when you're generating Shakespeare at least for me if you generate Shakespeare that is slightly wrong it's still Shakespeare because it's weird that's the point right but if you generate go and you generate something that is not exactly go then it's not go it doesn't compile right so it's a little bit different the NLP techniques are not gonna help us completely in this problem there's other research problems that have been presented there's a lot of research and actually I'll give you a list of them that you can check out there's why it's called var misuse and fast misuse is about this task where what we do is we train a neural network to give it a program we remove just one identifier from the whole source code and then we ask the neural network to predict what identifier goes there right is this useful actually yeah incredibly useful and it's not about predicting what is the token the that is missing but actually if you do this over every single mentee fire you might be able to predict when the token that you find is actually wrong right someone actually did a copy-paste error this happens all the time I'll show you how this works in a minute there's another one that I really like and it's much more complicated this is about learning from graphs and when you're learning from graphs there's many ways of doing it one of them is to rather than having a graph representing a graph and fitting it to a neural network is actually really complicated so you need to find defaulter native representations one of them is finding all of the paths across all of the notes so now instead of having just a if you have a sequence of paths right and then you can feed that and then you can train an integral with that but is that is what code to vectors and code to pack is very very good because it uses embeddings to actually be able to predict really useful things about code and identifiers if you want to know more about machine learning on source code we actually have a repository with all of the research we read all of the things we write everything is there it's super interesting I have not read the whole thing but this some of them are like good for beginners those are the ones that I'm breathing and they're very good very good oh okay so now we have nail networks we've trained them it's cool we might get some decent accuracy but it's never very very very high what can we do with this so this were rather than trying to predict things and generate code what we're trying to do is actually reverse the problem and say can we use our predictions to tell whether something is expected right so in the previous problem in the previous problem we were saying things like we're missing like for I for I equals 0 is less than 10 I what's going next sure we're going to predict something like things like actually this probably plus plus we have a 99% chances of this sure ok the thing is that if we see an - - we're going to be able to see that the prediction for that which is also something to compute every single time it's gonna be incredibly low so now you're gonna start using that information which is somehow like the amount of entropy like the amount of information that a given token is bringing to the program you're going to be able to see it the prediction is so so low that probably there's a problem and we can use that we've used this for the retention model for code reviews you're reviewing some code say you're reviewing this decently large change in your reading of this how do you review this well the way you're supposed to do this is lamb by line reading every single thing and paying attention full full attention to every single line of code imagine if instead of doing this we're actually able to find which pieces of the code are the least expected and least expect that doesn't mean they're wrong but they're actually more complicated there's more information in that piece of code than in the rest the rest might be public static void main right who cares about public static void man you don't review that you review the pieces that are interesting so imagine if you could do something like this and this is something we've built and it works pretty decently I applied it to this problem and here very very quick it said that - - has a super low probability of actually being correct it should be I plus plus right those are things that probably in a big code view are things that you might miss so what we're doing is trying to help code reviewers and authors write better code through tools that are gonna help them they're not going to replace them but they're gonna help them do a better job for Varmus use this is a typical problem this is how you open a file and go and you do defer to make sure it's closed at the end and I'm lazy so I do copy/paste right and now I read this program I'm opening a opening B and then coping from A to B and this code is wrong why because actually we're closing from twice right and this is where the varm is use it's gonna actually tell you hey like maybe this is okay but there's a very very low chance for this we're predicting with a very hyper the very high accuracy that this should actually be - instead of from right and we can actually provide that as even a suggestion on github reviews which is really cool so the author can be like oh yeah true and just click and fixed right these kind of things that can set some of the things we're working on go to vac I'm actually gonna show it in a minute because I have a very cool demo for code two back there's one more thing that we have is using deep learning to understand how identifiers should this should be split and we wrote a blog post about this we're using bi-directional long short-term memory recurrent neural networks super long way to say it's a very complicated thing and at the end what we're able to do is given some code given a sequence of characters that are not did not have underscores or camel no comma K is not stinky is none of those we're able to predict how we should split this and then we're able to predict to use that to predict whether something has been capitalized correctly for instance or whether the hyphens are actually the hyphens the underscores are put correctly so a couple demos the learning girl left it here so letting go I will publish these slides but this I started running it on collab before the talk started and you can see how after not very long it does generate a lot of output though so this is training let's see okay let's see so yeah there's the input sentences and the output they're generating it's still pretty bad but if I'm able to find predicted output there's a big big oh there you go so this after 30 minutes the code is generating and these things this start slope trigger it's using a TPU which by the way it's free and you should definitely get out I don't work for Google anymore but still really cool because it's free so collab is super cool for this then was very very nice but yeah this is this fin for like 30 minutes whatever the the time has taken for the talk right so it's pretty impressive code to vac this one is something that we have not built this from the techno technial University in Tel Aviv and they pass it the past program they show those paths with which pads are the most important ones are shown as being thick thicker so basically the needle network is actually trained to learn which paths are the most relevant and then use those to predict the name of a function so here this function is called F but it's saying with a 99% of accuracy of not accuracy certain certainty that this should be called sort or bubble sort or things like this you I'll let you check it out I think it's very cool and beddings also allows you to do things over and embeddings of of identifiers so what is count similar to well count is similar to size and index and some max all of those functions tend to look very similar or equals and to lower equals ignore case right so you can actually do that those mathematics inside of this really large very high dimensional space where you can start doing these things and computing whether a function name is valid or not whether it actually represents what the code is doing and the analogy is the typical one receive is to download as sanest to upload you can play with those and change them there this is actually pretty cool you will see that this works really well for the examples that are here as soon as you start changing the code things start to work not as well but this research right what we're trying to do is make this research into things that you can actually use and then you're not splitter I don't have time to do it so actually let's go through this really quick so there's the train model I am very satisfied thank you so there's the needle model we loading it this has been trained over public archive which is three terabytes of source code so it wouldn't take longer than this and then you get the things like yeah like I think this is a variable you just write like that and it's gonna actually figure out how to separate it and writing some extra Python you can actually do it so it will convert that so I think my time is over that's how you're supposed to write it in and go write like this this is pretty straightforward but still very useful so what does this look like when we're using it what we're building is an assistant cook view right this is what we're building now this is our first ml and cut product and the way this looks right now this is life on github is something like this you have oh I lost the comment let me refresh yeah so you have look at staging that's a bot and the body's saying hey like these lines of code here are actually very similar to 118 to 163 and if you look at those lines of code you will see that they look very similar but they're not exactly the same we're actually doing code clone detection over code that has changed slightly so if you have two two different functions and you change some variables and you change things like this we see able to detect that you should probably be just writing this once and not referring your source code over and over and it does many other kind of analysis so one of the things that we're working on we're working on automated style guide enforcing so learning the style of your source code from your source code and then applying it to your know PRS so the whole idea behind that is the fact that we do not want people to spend their time saying you need to have two spaces after the double bar to have a comment right like a machine can do this much better and you may write to for this there's many linters that do this very but you need to write it for every language if we were able to actually learn it from the language that is already there without having to write the tool will be much better we also want to do a back prediction and then one day maybe automated could review not assisted but actually automate it how far can we go there I think there's a lot of the different things that we can automate and then cogeneration by having conversations with your computer which is kind of like a compiler but if the compiler will be asking you questions about hey you want to sort this do you want to use the normal algorithm do you want to use something else and just to make a lot of assumptions and try to minimize how much interaction you need to have with the compiler to write a program and I think that'd be super interesting so will developers be replaced I would trying to replace the developers if you know any architect I don't think that our cadets would say that CAD kill their profession rather the opposite and that's what we're trying to do right now what we're trying to do is write better tools create better tools so developers can be empowered to do better things and we're hiring yes we're hiring if I don't say this we're hiring like product managers and like engineers and everything really we only 35 people so we're definitely hiring so if you're interested find me around and I do have stickers thank you Hey thank you that was absolutely fascinating thank you I do an LP and so doing it on code is a great idea I think we have time for like one question and then we need to move on to our final talk of the day anybody here if not you can also ask questions to my email franceska towards attack or a trance ask on twitter thank you thank you again I guess