Bay Area AI: Eiso Kant, Building a fully Open-Source stack to analyze the code of +60mm projects
thank you so before I start and tell you a little bit about myself and what we do I just want to get a bit of a measure of the room who here is a works in machine learning actively okay right who here until today or before has absolutely had very little exposure to machine learning or deep learning okay so and who here is a business person it's not a bad thing I hope so otherwise I'm out of a job so now we just to get a little bit of a feel of the audience so let me tell you a little bit about myself so let me start by telling you just a little bit about myself so you know how serious to take me today essentially so I'm one of the cofounders of source and that source that I'll be talking about today is a lot of the technical stack and some of the kind of interesting things related to building and massive data pipeline on top of code so I'm a huge computer geek as alexia already presented me although I'm a used computer geek who's had absolutely no formal education I've been programming for most of my life I've been working with neural networks for quite a few years huge passion for programming use obsession with neural networks but I'm actually not a formally trained expert in programming languages or formally trained PhD in mathematics or machine learning I feel very privileged and lucky though because I get to work with about 30 people most of which who are actually formally trained PhDs or programming language researchers or hardcore compiler guys or engineers so I feel very privileged to be able to do so so I'm going to talk a little bit about one of the core fundamental things we believe its source and all later come back a little bit about how our manifesto some of the things we believe actually play a very large role in how we treat data and how we build a data pipeline so essentially it comes from one very fundamental belief that we have and it is not a very controversial one is that we consider four grammars closer to artists or writers than we do to civil engineers essentially saying that if I'm a developer my raw materials programming languages if I'm a writer my raw material is here the English language in the dictionary and I'm a painter it's my palette of colors and and based on this raw material it's up to my own ability and creativity to create something right I can write a book or I can be Glenn Greenwald writing for the Washington Post or I can be a developer creating a web app or doing working on neural networks or etc etc right at the end it's up to our own creativity and ability to create something why does this matter so much in relation to what we do so if we look at technology pretty much since the beginning last 3040 years it's continuously grown in orders of complexity and magnitude this is one of the things that almost all of us can agree on but what's really surprising is that we've written these trillions and trillions of lines of source code but we're actually not leveraging them right we've written already trillions of lines of code but we're not using it to improve the way we program improve the way we build software we're not learning from code that's already written right and when you're not learning from a huge beautiful data set there's for sure out there some geeks like myself will get very excited about it so at source we believe that a ieung code is a field that matters is that we can learn from all the source code that already has been written and I saw earlier the question for Michael what's the difference in AI and ml it's purely who you're presenting to at the end of the day they're exactly the same thing in my opinion so you can replace a I would ml but what I'll be talking about a little bit today is our quest towards how do we turn source code into a data set that is as important as computer vision is today or natural language is starting to become right if we go a little bit back through the history particularly of deep learning since kind of the inflection point in 2012 since it became popularized again since we came out of this AI winter if we go back we essentially saw in 2010 image net being created right image net is probably one of the most quintessential datasets if today you work with things like convolutional neural networks or particularly in computer vision and image net was a data set created by dr. faithfully while she was at Stanford as AI while she was at Princeton and to essentially combine this huge data set called work net that we had of words that have semantic similarity to each other with a hundred images each set for every single word group we want to have a hundred images that actually are labeled the text is a label on top of these images once we created this data set in 2010 the whole computer vision field all of a sudden had a very very large beautifully labeled data set that they could work with now at 2012 a few guys who had been working in neural networks started actually applying some of the techniques that they had seen over the years not work to this data set together with the fact of the breakthrough starting to use GPUs for computations of gradients and essentially the the main thing that we use in neural networks so since then but five years later we're back in a full-on AI hype we're starting to see it in every single field and every single data set that we can imagine is being applied to it and but how I look at it you have essentially images today you have speech right audio voice music and you end up having natural language and probably in that order is also from ease to difficulty we're getting really really good at understanding things in computer vision we're getting a lot better than we were in understanding things in audio and we still really really suck at understanding things in natural language but code is kind of this data set that falls a little bit in between any of these things right code is not natural language code is not an image code is not a music it's essentially a very very well defined semantics that we use to instruct our machines and the only reason we have programming languages is because we don't want to deal with ones and zeros all right one of my favorite things to do is to go back to these interviews with some of the kind of like legends in our industry like Donald Knuth or Simon Peyton Jones at the early programmers who made a big impact and you'll all here like how when I started I was dealing with punch cards right like and at the end of the day we have just created these languages to make it easier for ourselves to deal with Changez so why is it interesting to start applying machine learning and particularly deep neural networks on top of code so there are several areas in terms of computer science and there's many more that we personally find interesting the first is what they call an academia inductive programming are also part of the field called automatic programming which essentially means if I have a sample of inputs and a sample of expected outputs right I have the numbers 1 2 3 4 and I have the expected outputs 8 9 10 11 what is actually the piece of code that transforms these inputs to these outputs that's pretty much at its core what people talk about inductive programming when you hear this you already start seeing this is a machine learning problem at the end of the day right it's a transformation of data from one form to the other I'll talk a little bit later about how in particularly this is an interesting area what you can do it but very practically developers we write millions and millions of unit tests write unit tests are essentially what we write to be able to see if our code has the expected behavior that we want all right in the unit test you'll say okay for this function I'm going to pass it these values and I expect this output to be able to happen so all of a sudden if you start thinking we have a data set already of expected inputs and outputs and the code that is supposed to make it happen and an ability to check if it actually does all right then we talk about translation translation is essentially a field in computer science of going from one programming language to another right and if you look over the last 3040 years particularly with us our developers kind of ability to consistently want to jump from one high bump into another right consistently changing programming languages sometimes for the better and sometimes just because it's trending or sometimes because Google is putting a lot of money behind it or another company is can we actually have systems that are learning how to convert one language to another instead of massive rule-based systems dynamic memory allocations so right so today a big part of programming has to do with dealing with memory and higher-level languages it's more abstract you might not ever realize it if you're in an interpreted language like Python until unless you're a data scientist and you run out of memory you start realizing it's very very quickly but in many cases it's more more abstract right but what's interesting we recently did some work where we took video games and we started training on the video game data while people were actually playing and using it to see if we could predict what memory we needed to allocate a compiler level ahead of time and could we predict it more accurately than what the compiler itself was actually doing rule-based turns out we've got about 98% accuracy this work we'll be presenting in the coming few months to be able to predict memory what that is not that interesting today if you're working on my Dell computer because at the end of the day training that neural network and in fearing from that model will require more memory than it saves but Apple just announced the core ml framework to start using the GPU for machine learning on their phones some soon is starting to include neural chips into their phones and all of a sudden we can start getting really really good and not just predicting memory but also predicting what the user is about to do next and actually start loading values and state beforehand all right so all of a sudden with machine learning doesn't just touch the code itself it also starts touching the way our efficiently our devices are code similarity right how in many many cases as developers we're applying patterns that have been applied thousands hundreds of thousands of times that can be simple patterns like I'm using an API or a library that is very popular that can be stylistic patterns in terms of how we do something that can be a little pattern in terms of where I put and if error nil statement if it's talking about golang and so in the end understanding similarity between code about a semantic level a syntax level and a style level can be incredibly useful for things like code search or recommendations style transfer so there's there's been this field in computer vision you might have seen like articles online or nice little tweets on Twitter where you take a painting and you take a photo and you transfer the style from one on to another right and it looks amazing it's actually one of the areas I would say in terms of kind of creative AI that has worked the best and it's having great results but in code there's also style actually you ask any single developer to review someone else's code you'll very very quickly realize their style there because we're highly opinionated people will start saying I don't like this or I do like that and the reality is a lot of this I like this or I like that has absolutely nothing to do with rational reasons right there are some objective arguments to be made in some cases but I don't know I prefer using pattern matching for recursion and so I'm wearing this t-shirt for those of you who like ice core know about it but there's lots of other ways I can do recursion inside high school why do I do one or the other just because it feels nice to me I like the syntax I like how it aligns right or even the fact that there's a lot of developers who are opinionated about languages because their syntax not because what actually is blowed the hood can we actually have style automatically transferred where is this very applicable in the world is you come in to Google you come in to Facebook or any large code base IBM and you'll start having tens of thousands of developers who all have their personal opinions and we'll all have their personal style all working on the same codebase right what if you could actually normalize that style or what if I could extract the project view it in the style I want to view it but commit it into the style of the company all right sort of interesting areas you can do here neuro compilers earlier the talk before you heard a lot about labeled data most of the work we do is an unsupervised learning but actually if you think about labeled data you think about translation I mentioned earlier programming languages at the end of the day for they're just abstractions on tops of ones and zeros and probably the lowest abstraction that we as developers like to work with or if we have to is assembler essentially this kind of small machine language the machine language that essentially is a control flow and it doesn't have all the crazy branching structure that you have in more higher level programming languages so when we write a compiler write something that turns a high-level programming language into instructions for the machine we are essentially again having a data set of two sides we have the programming language and we have the machine language and today we have these compilers that are written of thousands and tens of thousands of rules that are highly optimized there's even companies that pay to have custom compilers based on their use cases in the company but what if actually a compiler wouldn't have to be based on rules but could actually learn and change over time right neural compilers vulnerability detection security detection there's a lot of areas but the reality is today in every single one of these areas there's nothing running in production at any given company in the world or at any given tool that is really applying machine learning on top of source code when we started about two years ago this field was completely new we actually started putting the words AI on code together and now we're starting to see startups and research happening at Microsoft and Google we're starting to see more and more people thinking about this field and almost everyone can agree that in the next few years no one knows that that's two five or ten AI will influence the way we program in the way we build software but the reality is there's nothing out there today to make that happen and that's essentially what we try to do it's source can we create an open source stack that allows every single person in this field to be able to get started quickly and start building these things so mentioned data right so where are we gathering our data from at source we have a pipeline that gathers every single public get project in the world online doesn't matter if it's a self-hosted see get Colonel if it's on github which is by far one of the largest data sources if it's on github Chinese equivalent if it is on any given site personally hosted with crazy re architecture I'll talk a little bit about later and when we started just on the github data set just by far like 80 percent plus of the code there a little over three million developers who have publicly committed code online today a little over two years later that's 10 million people right so we're starting to see that this growth of code that is publicly available keeps on growing exponentially and a lot of that is credited you decides like github which are starting to push forward you know this kind of collaborative social way of coding a lot of this code that's out there from developers is not classic open source that you know right the perfectly licensed projects that are used cases in many different places but it's in many cases on millions and millions of projects of developers who just decided to play with something or put something out there for themselves so why now right why are we interested in AI own code today and happen for the last two years this is our thesis and I think in this really want talk many people the number of developers is growing exponentially besides data that you can look at in the world the easiest way to look at it is to call up all of your friends who have young kids who aren't programmers or who don't work in tech and you start asking them like I do you want your kids to learn programming are they learning something in school and you'll start seeing that all of a sudden in most cases at least in the people I know and people have asked about this they're little kids or three four years old or in some way being pushed into being a developer right and if you go to if you go to India you go to China these are countries where you see this far more right for me the shocking number is 600,000 people graduate each year in computer science in China only 60,000 in the United States right we're screwed here at the end of it no offense but so this is one of the things right this number is growing exponentially just 21 million software developers we have in the world today I personally believe within the next 10 years that will grow an order of magnitude software is eating the world right that's the phrase that essentially classifies California or at least the Bay Area programming becoming a new lingua franca and one of the nice things that you see around this is if you how you look at how people program in different countries right most variables most Commons everything is in English the code for itself can speak even if you don't speak the same language complexity of code is increasing right we don't just see this in our own code bases we also start seeing this in consistent failures right British Airways in the last 12 months their entire system went down twice right and so it's this because of hackers or is this because of they claim wasn't because of hacking claim because of issues in their code but the end of the day these systems are becoming so complex that both the security aspect and the actually maintain bility aspect and the points of failure are just becoming too many and this is a massive industry at the end of the day data sets are becoming enormous or in our case source code we have a data pipeline that goes into petabytes of source code all right so close the two petabytes of source code written that's a data set that is hard to find today and anywhere right computer vision data sets are in the petabyte but we're talking here about text right GPU computing at the end of the day I think we could all be incredibly happy at least for us who are working in the field of neural networks of the rise of computation will power with GPUs and the decreasing cost at the same time right and then neural networks are cool again all right since 2012 they're cool again and we hope that they'll stay cool for a little while but what is most interesting to me is how many people have been attracted to this field right if you look at one of the largest conferences of these academic conference in this field is called nips if you look at nips in 2012 wreckin had about 300-400 paper submissions this year was over 5,000 papers submitted right so it gives you a little bit of an idea of like what's happening here and it's becoming pretty obvious so I'll actually today ask me to talk a little bit about our pipeline so since we're not a hardcore technical audience from what I've seen I will go through some of the things we do and if somebody says hey I want to go deeper into that you can either ask me right here on the spot or we can talk about after one thing that makes us a little bit different than probably most of the companies you'll find here in the Bay Area is that we're not a big fan of doing things on the cloud when you're dealing with petabytes of data and you're trying to go to Google cloud you're trying to go to Amazon your storage cost for the high input output right the fact that you need to read and write your data quite a lot in our case primarily fast reading of our data means that you're getting to a bill of several hundred thousand dollars a month pretty quickly right when you're start-up we're lucky we're a little bit larger than you know just out of the box startup but that's becomes pretty darn expensive right so one of the things that we decided pretty early on was to go bare metal so we've actually been working with a hosting company in this case in Paris one of the largest in Europe where we built essentially 60 high spec CPU workers that cover about two petabytes of storage close to 12 terabytes of RAM and thousands and thousands of course so it gives you a little bit of an idea of like trying to do this on the cloud the equivalence is about $15,000 a month versus about $250,000 a month yeah it's a very large difference however and there's one very very big if our DevOps team and most of our engineering team it's partially revoked but is based in Madrid Spain right so we have a three-person infrastructure team and a good three-person infrastructure team here in the Bay Area or run you easily about half a million a year in our case we have that significantly less just happens to be because of the cost yeah the market rates for salaries back in in Spain where where our main offices so you always have to take both things into account and the fact of course you have a lot more points of failure however for us our data pipeline is not one that is exposed to the outside world it's not one that has millions of users banging it all the time we don't have low balancing issues right it's purely for ourselves and in that case having bare metal makes a huge difference then pretty early on we tried lots and lots of different things with all those GPUs that we need for training right so we started with a little box in the office then we spend a ton of money on Google cloud but if you're doing cutting-edge GPU computing in almost all cases you want to be working on the latest architecture that comes out in the video for us the difference between a Kevlar and Pascal architecture and chips is about 2x cost and speed savings right so what happens in this case and the video does something very very smart that isn't very publicly discussed and was anyone here work from the video no okay is that they have the line of chips that they sell into the deep learning world and the p100 and they have their box is the GTX the GX one's kind of the Rolls Royces of deep learning they have all of these chips out there but when you look at bang for a buck processing power what you want is the chips the gamers have you actually want the 1080 Ti you want the Titan X you want these cards that are in many cases three times cheaper for what you can actually get for processing and energy usage but NVIDIA does something very very smart it's that look when licensing wise cloud host computers or cloud hosting companies aren't allowed to actually rent these out so Google cloud Amazon they're not allowed to actually buy these chips and put them in their machines some of the interesting things are happening in this industry I don't know enough about because now NVIDIA has actually their own kind of cloud that they're starting to set up and it looks like there they will allow it but at the time when we were were doing so we couldn't get access to these chips at the same time because Google and Amazon they have quite a shortage in many cases of supplying the latest architectures and getting them actually from Nvidia it means that you're almost all cases working for about 9 to 12 months on an outdated architecture right so now the volta has come out and actually Google has just announced about three weeks ago to the Pascal chips are now there right about a year later or so so if you are very reliant on the latest speed improvements latest architecture changes in our case that's to do with some floating point math operations it's just something to keep in mind so what we started doing now actually it's happening while I'm here we started actually building our own little mini data center at the office we realized that in the end for us what made a lot of sense is we actually just built part of her office into a data center we start taking servers that fit exactly the same specs as we can keep them with our hosting providers we put them in the office and over time we keep on moving them out to our data to the data center literally take the box ship it and put it somewhere else in this when you start dealing with these data pipelines where you're doing a lot of GPU processing on a lot of data you hit into this very stupid practical constraints one of the biggest practical constraints you get is energy consumption for most data centers that you'll find a rack is allowed to consume about three kilowatts of energy so any hosting company that's out there prefers to stuff that rack with you know 15 20 cpu machines that don't have GPUs in them because an average machine with GPU will take about 4 or 5 600 kilowatts right so you can put as many machines in the same physical space right so there's a lot of these silly constraints that you get to when you start dealing with at least processing at skill and you got to start making your own decisions of what makes sense for you the only thing I can say is that it's not that necessarily bare metal is better than the cloud or putting a being crazy enough to put things in your offices is the right decision or co-located hosting or these things but at least spend some time figuring it out what makes sense for your use case right in our case we have this essentially processing cluster that can prepare with SPARC all of our data sets by the time it goes in a petabytes of data the data set that will come how's that goes actually in deep learning is usually the terabytes or in the hundreds of gigabytes which is relatively easy for us to transfer and then actually transferring that data set in our case literally to our office makes most sense right but this is very specific for your use case we are big fans of containers probably every single developers these days it's a bit of a hype but in our case it makes a lot of sense with the way we actually manage your architecture right the fact that we have that we will still sometimes go to Google cloud if we need to process something in time for something that we will switch between our bare metal processing and the actual kind of mini server rack back at the office it makes a lot of sense to be able to have a format that you can easily move things around with also at the same time there is an interesting trend happening or starting actually it's not happening yet because I know maybe if two or three companies are doing so but a lot of the people in the container industry are talking about this is instead of moving your data from worker to worker why don't I move my container images from worker to worker it's actually keep the data on the machine and actually start moving the containers instead so you can build processing pipelines where you essentially keep state of what's happened in the container without moving the data right if you're dealing with very large amounts of data this can actually save you a lot on the i/o operations or they started networking operations that you have in between and if someone's interested in this stuff we can talk more I'm not a hardcore DevOps guy but I feel like I know enough to talk a little bit about it and then I won't go into the basics too much today but we've run pretty much plain vanilla spark pipeline it's not the best spark it's not the greatest but it works it works very well most of the tools and I'll talk about the things that we build we build as binaries and we run on top of that and then we use tensor flow we use tensor flow not because it is the best library not because it's the most performant library we use it because there's about 4x more developers who use it than the next-best library right tensor flow has essentially one in terms of size of community and when you're looking in terms of particularly the work that we do in deep learning what you want is you want the things from the latest research papers as quickly as possible to end up in code and then this from code you quickly want to ended up in actually distributed way of doing it and then a distributive performant way of doing it right and so often you're working in a vacuum you're working on top of your own stuff but a lot of the architectures that are coming out are coming out from other people other researchers from other groups from other companies in most cases we see that those implementations hit tensorflow first and we can actually go work on and improve them so we spent a lot of work together with Google actually improving some of their models into tensor flow models repos so you can actually see as a company we dedicate a lot of time doing tons of pool requests in the official tensor flow repos both on the core and on the models because we see hey look if it's coming on in research paper we can put it there and together with the community we can build it and so this is why we've done so it's actually I think the reason a lot of people are going for it there are some extra benefits in tensor flow as well but the reality is for many specific tasks there are other frameworks that perform a lot better at least in my opinion for now right you kind of have to think where do I bet on and I personally prefer to bet on the mm deep learning researchers that Google is hiring right now and the other 27,000 engineers that they have so what is talking I won't go too technical into a pipeline but when we look at a pipeline we look at it both from an actual how is our organization structured how is our data structured and how is it actually technically structured so for us our raw material is always code and code starts with data retrieval where do we actually get all these projects and code from so we have a data retrieval team and source it's about a seven person hardcore engineering team with guys who actually it's a very very team the lead on that team one several times the AI vandalism competition for Wikipedia has worked on crazy large data pipelines in-memory query engines and probably the most junior person on that team actually worked for worked with the with Dennis Ritchie on planned line so that gives you a little bit of an idea in terms of the different type of people then we have the language analysis tools I'll talk a little bit about that later but it's essentially how do you how do you take code and turn into a format that you can actually work with it now we have the machine learning team which in our case is a little specific so one of the things that early on we realized since we were over a relatively small company about thirty people we didn't have the ability to go and take the top researchers from deep mine or top guys from Google brain and so we decided something different we said let's find the people in the world hence we have the code of nine million developers so we could look at the code so let's find the people in the world who follow essentially this kind of profile they need to have a hardcore math or physics background at the end of the day if you're doing things in deep neural networks you can go very very far with getting the stuff working but if you don't have a strong math foundation you're going to hit a roadblock at some point so let's find people who have that strong background who then after for whatever happened in their lives actually decided to go from math physics to computer science and became really hardcore computer scientists many cases compiler guys or people who are really like doing low-level programming and then around the Year 2012 also woke up to this hype and moved to machine learning and particularly deep neural nets so in our case that turned out actually based on our data set and a few coincidental occurrences that we found everyone who really fits that profile to be based out of Russia and so and and that had to do with the fact actually later speaking with people I was just giving a talk in Moscow last weekend weekend before last about about this I was talking a lot of guys are like look why do you think this is right is it well many cases you know we started we studied math or physics and we couldn't find interesting jobs that we wanted to go into and you know my dad was doing something with computers I came across this book annette aside and i got excited about computer science and that actually ended up going into Olympiad programming Olympiad Math Olympiad programming and from there I ended up later on a machine learning so we made a pretty conscious decision as a company early on to focus a lot of our efforts in terms of recruiting and an attracting talent pretty much specifically on Moscow st. Petersburg Tom's get a few other cities inside inside Russia or we found people who really have this back and also in many cases fitted our culture and today funny enough I would say close to one-third of our team is actually Russian we're from nine different countries from all over the world and doctor originally were based out of Spain and my co-founder is German so with people from many different places but turns out that what made a lot of sense for us and allowed us to hire people that didn't actually want to come in many cases here or how to offer some Google or Facebook puts it has too far from family I want to stay in Europe I want I don't want to go that far and actually allowed us to find really great people on our team and then data sets right at the end of the day of pipeline along different steps that could be on the data retrieval and it could be on language announcer machinery at the end you brought you spit out data sets and I'll talk a little bit later about models and demos so just to give you a little bit of an idea of some of the things that we have in our data pipeline and I'm going to keep it very simple we start with needing to discover where there's code since you're discovering where there's get projects in the world right so we have Rovers Rovers is a project that's Rovers and voyages are actually being open-source next week everything else that you'll see is fully open source but these are essentially crawlers and API connections to start building a massive queue of list of every single git repo in the world so that we end up with this massive list of git URLs and with Borgias we start get cloning and storing every single source code project that is out there and we don't just do this once we do this continuously right essentially every few days that's the maximum time of projects allowed to be out up outdated we keep on get cloning again and gathering this data what's interesting about version control data at least in particularly of source code is when you do all of this you get about a hundred terabytes of code it's a lot it's not that much but then when you actually start differing commits right going back through the history of version control you end up with more than 10x more data you start ending up into petabytes range of data and this is very interesting I think the same thing probably would apply to documents and other things but if something is version controlled you can probably discover a pattern like this for us what was interesting is that along the way we needed to build a lot of stuff that hadn't built yet or haven't been built in a way that was useful to us we needed to actually modify the implementation of get so today we have the world's most popular implementation of gift it's being used by tons of different companies and all over the world and we actually build get from scratch it's a little crazy we needed to realize that the way we were storing data the way we were archiving format we needed to be able to have a C Keable index so need to be able to take a piece of data compress it have it indexed compress it have another index compress it and be able to go through those indexes right because when you're dealing with this much data and you're on a budget right you don't want to go and spend hundreds of thousands of dollars in the cloud you needed to be able to compress it and effectively manage it turns out that we actually need to create our own archive format tar and the others didn't do this so you find yourself very often even though your whole focus is on AI and on deep neural nets you find yourself with a big part of our team in our case almost 70% of our entire team focused on actually the data pipeline itself not on the fancy models on the fancy architectures so just be aware of time yeah I think we're good so code is not the same as language it's not actually this linear sequence of tokens right well we when we look at writing or we look at code we essentially see okay it's one token after another but in the end when the compiler gets code right before it starts turning it into machine language it gets an abstract syntax tree essentially it's a tree structure of code right if I have an if statement I will have the two conditions as nodes below and the values of those conditions below that and at the end it's essentially a tree and one thing that holds true in almost all actually which everything in machine learning is how you represent your data next to the integrity of your data is crucial right sometimes you can represent your data just as a text in our case representing it as a tree makes a huge difference one thing that's interesting for those of you who are really want to experiment and do some crazy things sometimes taking your data and representing it in a format that you don't expect as natural to it will work better like I've seen people be able to take things like text and right a representative soundwaves and for their problem that allows them to actually get much better results right and I've seen so there's a lot of interesting things you can sometimes do with transforming your data into different data structures in our case it makes a lot of sense that code is an abstract syntax tree so I won't go too much into this but essentially we built an architecture that has a server and I send the file to the server any for any piece of code could be on a crypto PI it will detect its language it uses a machine learning based approach detected about 400 languages nothing too too crazy or fancy and it will find a container with and in that container is a language driver a native ast parser and something called a normalizer that annotates it and turns it into a universal standard for those are your programming language freaks like me or researchers we're taking every single programming language in the world and turning it into a universal abstract syntax tree an actual representation for that and it's a massive undergoing project start about four months ago we're getting a lot of people involved and at the same time it needs to perform incredibly fast right trying to take code and parse it into this tree structure is not an easy task so I'll talk a little bit about some things a little bit more general you see here three libraries and all three of these libraries were created from us by scratch not to solve new problems see all of them are classic computer science problems but to solve them for the first time in a performant manner on GPUs open-source I'm absolutely certain that we've replicated the work that for sure is inside Google inside Amazon but in all cases is closed source and we haven't seen in the public domain so start at the bottom one of the most probably early things you on the end up doing a machine learning is clustering k-means clustering k nearest neighbor search alright this is one of the things where I have a data set and I say ok based on several features cluster it into ten clusters 20 clusters when you're dealing with code you in our cases and many cases have per project tens of thousands of features we use language embeddings I'll talk a little bit about that later and we have not 1,000 documents or 100 or 101 million documents we end up with billions of documents so how do I take this very high dimensionality vectors a vector of 512 dimensions I have a bag of hundreds of millions of it and actually clustered them based on their distance or similarity to each other well I can tell you exactly how much that cost on Google cloud cost about $50,000 to try to do that with a bag of 16 million documents and for every single document have a vector of 512 dimensions that's not feasible and pretty much the moment we saw that cost after we ran it because we needed to run it we said okay what can we do that actually avoids that and what we saw there's a lot of low-hanging fruit today in actually writing direct implementations on GPUs and I'm essentially advertising this video in the video here because we write a CUDA library that is highly optimized to do k-means clustering and k nearest neighbor search on top of GPUs and now I can take two ties index cards or two gtx ten ATT is that will cost me 1,200 bucks to put into a machine and i can run that entire 50,000 dollar experiment in the course of about 15 minutes on these two GPUs right so there's a when you're dealing with these kind of problems which essentially at the end of the day our matrix multiplications then you should really consider what if we write something that works on a GPU if you are please open-source it because nobody is open sourcing their stuff that's on GPUs in many cases so it will help a lot of people this until two weeks ago or three weeks ago was by far the world's fastest implementation hundreds of times faster we just got beat by Facebook we were a little bit upset we're like wait well we're not that bad like how did they beat us turns out the guy who originally invented k-means and all of the implementations the last 15 years have been working on optimizing it and actually be us it's called fat FA is s the only thing is it's a research-only license so if you're dealing in commercial environments we're still actually the fastest and for some use cases we are as but essentially kudos to them in many cases when you're dealing with very very large datasets and you're trying to do fancy stuff with neural networks you are finding yourself in a position where you need to start reducing your data set into something manageable right I have 1 billion documents but I want to start checking them pairwise to each other based on some features that's not possible right it's a billion times a billion it's essentially an np-hard problem so one of the old-school techniques that still works really really well and many people try to do lots more fancier stuff today is just hashing right just using hashes and there's particularly a technique called weighted min hashing which essentially creates a hash of a document or of any piece of content that you have and allows you to start comparing the similarity of that hash based on something called Jaccard similarity essentially I can say give me this huge set of documents create a hash for every single one of them and now show me all the documents that are 70% or 80% similar in terms of their content so weighted min hashing again this was a problem that on CPUs would cost us tens and tens of thousands of dollars to be able to do comparing software projects if they were Forks of each other but actually not marked as Forks so what we decided to do is say what happens if we actually create an implementation on GPUs pretty much the same thing as I mentioned earlier takes 15 20 minutes to Train on top of two or three GPUs versus and we'll run instantly after with inference versus actually doing it on CPUs and spending tens and tens of thousands of dollars on the cloud so there's a lot of problems where GPUs don't make sense but if you're doing doing floating point math if you're doing matrix multiplications anything essentially at the end of the day with numbers put it very bluntly have a look if it makes sense to write libraries directly on GPUs and then we have something called W and D relax W D relax doesn't have a very sexy name but it's in my opinion one of the most applicable libraries for almost any given problem that you face today with any type of text content no matter if it's code or if that's not language so earlier I only got to see part of Michael's talk but as far as I understood from his talk is that they have a massive amount of documents and they want to be able to understand the similarity between those documents right in many case or make recommendations so what wmd relaxed does actually out of the box is it says for every single one of these documents whatever you want to use try to create some embeddings for the words in them so there's a lot of interesting work that's become very very highly popularized over last few years called word vectors or word Tyvek or word embeddings and essentially what that is it's a model that has learned the co-occurrence of words if I have a given word horse how does it colloquy with the word stable right if horse and stable are most likely close to each other in its vector distance and it's Euclidean distance or whatever distance measure you take horse will be closer to stable then horse will probably be to computer all right and these word embeddings have turned out to be unreasonably effective in many many use cases it works very very well to be able to understand semantic similarity and context in our case we do some crazy stuff on top of natural language and code I mentioned earlier guess programming languages are very small they'll have like 80 reserved keywords and with those 80 keywords and sets of tokens you can program anything you want but then as developers we name things all the time throughout our code whenever variables anymore functions or classes and essentially what you end up with when you analyze 60 million projects like we do you end up with two magnitudes more vocab that you have in the English language you end up with 20 million Cokin we're actually unique so what we decided to do is that we want to understand natural language in source code and the ability to understand natural language in source code is to actually extract everything that we've named ourselves and try to find a similar type of embedding for that so when we have a project we will take all of the unique names out of that project all the names that are essentially that developers have named or have used from other libraries and for every single name we'll look up the word vector we'll look up the embedding that we've trained in another model all of a sudden we have this bag of embeddings it's thousands and thousands and thousands of Victor's we have a bag of thousands of vectors and I want to compare it against sixty million other bags of thousands of vectors right and those 60 million bags of vectors in our case are representations we have on code but that could be things you extract from natural language it doesn't even have to be natural language embeddings in most cases when you're dealing with things in machine learning or dating science you have some form of vector representation and you'd love to know the distance so wmv relaxed is an implementation of the word move or distance problem in computer science which is proven really really effective to be able to actually start taking this bag and seeing which are the bags that are most similar so what that allows us to do in very fun cases is I can take the vector for the word beer and I say compared against 60 million projects and I will find all the projects that are highly related to beer or I can take earlier I mentioned we have an implementation of git I take the bags of vectors for the project go get and I will find all the projects that are actually trying to do the same thing as gogit right and this can apply on documents this can apply on text and by other things so right now all of the links you see in my slide are essentially individual projects one of the things we're working on in the next should this month and hopefully done by the end of the month is a fully new site with documentation examples that it makes it easy for every single person who wants to get involved in terms of applying neural networks or any type of machine learning on top of source code to be able to start using these things out of the box so I'll just talk today quickly about I talk to you about word embeddings just this is something just to write down if you're expecting that you want to do something on top of text you want to have some form of embeddings for words and you've heard about words effect and you've heard about these fancy things and you're dealing with a lot of documents go to swivel swivel is a model created by Google it was a paper released about a year and a half ago but Chris Watterson and it's essentially a distributed way to compute word embeddings that is insanely fast and skills with the number of vocabulary and not with the number of documents this is pretty much the only way today you can create word embeddings on top of air be very large corpuses of data so it just makes sense to know and I'll show you a little example of a small skill one that's fun so every single dot you see here there's about 10,000 dots because after 10,000 my browser blows up is a name of a function in the Linux kernel my Linux kernel is probably what we're is one of the largest software projects that is open source in the world and what I've done is they are in this case embedded in a hundred dimensions that means every single one of these functions has essentially a a hundred row long vector in our production stuff we use a lot more and I've it's good here I've essentially decided to represent it in 3d with a technique called t-sne see sne is a really nice technique to visualize high dimensionality data in 2d or 3d and i in my opinion it's by far the best technique today what is interesting about this is let's start looking up and I'll also take suggestions from the audience but I mentioned earlier cryptography let's see what's happening and here I see when we go to woods crypto I see a little cluster let's get it back there I don't know how well you can see this but what you start seeing is that all the functions that are in this case understood to be semantically similar in your crypto are des cipher as CHC are Keys authentication encryption and once I go in this I also always like finding the random number generators which are here somewhere to look at your RNG and PRNG which is right on top of it the random data is all the functions that actually share semantic similarity in terms of cryptography right you can imagine not just doing this on code you can do this in natural languages with us on any corpus of text data for code it works very very effectively to start understanding what actually is similar what is also interesting and anybody here the developer wants to give me a name of a function they think is in Linux kernel No seek seek okay let's look at secrecy seek here so let's find it okay so this is interesting I've never looked up seek but you can see it hold on is soom out a little bit so you actually start seeing near seek which makes sense right if you start seeing file names directories reading directories sinking you see file maps you see actually printing nearby you see search here very relative relatively nearby reading so this is essentially a representation of embeddings of something that's learned why is this useful for us right when we're dealing with petabytes of data we don't want to deal with petabytes of data there's no dealing particularly with neural networks you don't want petabytes data more data we also yes more data is better more data is great but when you're dealing with the kind of networks we deal with particularly LST M's recurrent neural networks that we want to learn and remember something having more than even a few gigabytes of data many cases will blow up what the use case is that we want it will not actually make it better or more effective it has to do with the limitations of the architectures the sizes in some cases with hardware so what you want is you want to be able to take that large data set and create small artificial data sets that are representative of what you're looking for I want to train a neural network to be able to understand security vulnerabilities I want to start taking the vectors related to known security vulnerabilities and I want to then start maybe taking all of the functions that are semantically similar or co-occurring in that space and build a data set specific to that and I just make this one up on the spot but at the end of the day you want to be able to use whatever you have early on to narrow down your data or to filter it at some moment so just for the sake of time I won't go too much into where was I in two three based Ellis TMS this is just what I showed you so that's a little bit my talk today I want to thank you guys very much I hope it wasn't too specific on code but I also hope to show you just a few general trends right consider doing things on GPUs look at your infrastructure not just how everyone else is doing they'll go straight to the cloud make sure you make a consciousness for doing so if something seems really really hard or expensive or impossible in most cases it can be solved and don't always think that more data is better because it really in many cases isn't and in many cases also you don't need label data almost everything I showed you today did not use label data so yes it is more hard it's always better to have label data but if you don't don't necessarily think that machine learning can apply to you so thank you very much [Applause] principle other people have been thinking about the thinking is to get different experience this information I mean I know I know exactly what you mean local experience yeah yeah so location link to experience essentially right do you so I must correct so our team is fully based in Madrid right now we are now just starting since the last few months allowing people to be fully remote that has more to do with the culture of our company so we actually everything about our company's transparent you can go on our github and find out how much stock I own or what our holiday policy is or how much options people get pretty much everything that we do is out there in terms of location and experience so I can personally comment on the fact that I've lived out here I've got to live in Asia for some time and I've got to live through Bridget from Netherlands and I've got to spend some time in different hubs in Europe I'm still young so my opinion it's not remotely as valid as somebody who has more experience in this field but what I've seen is the following is that in many cases today at least in the field of machine learning or or let's call it AI or deep learning there's definitely certain hubs so you have Montreal you have San Francisco you have Paris being one of the main hubs in my opinion and those are essentially the academic hubs right the fourth year the hub's formed around certain form of academia or in SF they are in this case round industry what I've seen in the field of deep learning is that it's pretty much not location dependent in terms of finding experience as long as the people in this field are out there reading the latest papers going to the conference's making connections alright some of the guys on our team we have this concept of open source Friday which means every second Friday the whole company only focus on working on something that's open source and that's the only rule doesn't have to do anything with the company it can be about anything you want and I've summarized in our team we're working with guys in LA or in San Francisco or contributing back to comma AI right so what I've seen is that this has become relatively democratized another thing that I've heard quite often from CEOs out here particularly European CEOs out here is that they don't want their hardcore deep learning research to be here in the Bay Area for two reasons one is the talent pool is not that large and it's highly concentrated at companies and places that can pay a lot Facebook Google Open AI it cetera so in many cases they end up having their hubs in Paris sometimes in London is one of the hubs for exactly that reason but to be honest I've seen great talent all over and I think as long as you're connected to it it works okay one of the things that helps offset this and this is what we actively do is fly people around for conferences right so I have our team here different people coming here all the time we are yeah pretty much all over the world to be able to get some of those insights or just relationships that you can't get unless you're here in person but it doesn't feel at least for this field in other fields that is so that it really matters that much yeah so right now we've given a network battery problem inside a cluster or between clusters so between clusters there's not much you can deal with besides making sure you get the fastest possible line from your local provider it's literally as simple as in our case of literally calling the local provider and say look we want a direct link 1 gigabyte a second link that makes a big difference in terms of inside the cluster we heavily rely on match box which is a project bike or OS for the networking we've been contributing to it and in most cases one of the things that you can do what I mentioned earlier is if you're if networking is an issue inside your cluster is to move containers instead of moving data this helps and trying to process as much locally so one of the things that we optimized spark for internally is the pro and it does it by nature but we try to go quite far is to move as little data around as possible yeah it's a very good question so yeah it's probably like half the questions answer I should have mentioned this up front we're a little crazy what we believe that code will become as big of a field as natural language or computer vision and when it does we want to be the company that has built the entire stack below it and can come into Google or into Facebook IBM and say look let us train custom models on top of your code bases right let us take the entire tooling infrastructure everything that's out there and let's be the one to do it so it really is a big bet on this becoming a field I will say we're very privileged and lucky the fact that we have the funding and the investors who are willing to invest so that actually happens the analogy we often draws to the field of containerization right doctor went a small company out of Paris moved out here created the field of containerization essentially and now there's an entire industry with an ecosystem that makes it a viable commercial entity today it's not yet but to give you an example when you have things like automated testing or vulnerability detection all of these things require custom models on top of private code bases and to be able to really get value from that you need to train inside and we feel that right now and hopefully next few years we're best position to be able to make that happen you [Music]