data.bythebay.io: Anthony Goldbloom, What Kaggle has learned from 2MM machine learning models
so guys I have a favor I'm going to be one of these annoying speakers that makes a request can everyone on this side of the room come over to this side of the room I think we can all fit here and it will be a much much more uh fun and intimate talk um and that'll allow us to make it a bit more of an interactive session thanks very much okay um so just while while everyone's moving over uh uh so just a quick show of hands who's uh uh got a kaggle account so quite a few who's made a submission a few less all right well hopefully by the end of this talk I can motivate those who who haven't to get get excited about the um uh getting more involved with kaggle um so what what I'm going to do is I'm going to cover a couple of things today first of all I want to talk a little bit about uh uh just to give a very quick introduction to kaggle for those who aren't familiar uh it'll be very very quick um because it sounds like most of you have uh then I'm going to talk a bit about um some of the unusual things that kaggle gets exposure to as a result of we we're quite a unique company uh in the data science space in the machine learning space and so we see some unique things and so um I want to teach you or share with you some of the lessons that we've learned we've now had over 2 million machine learning models submitted to kaggle competitions it's a lot of machine learning uh so you would think there would be some good lessons learned from that um and then some of the exciting things that we're working on that you might find interesting and useful um so first of all just the very quick introduction to kaggle so if you come to our website you see all these different problems uh they range from uh trying to take MRIs and diagnose heart failure through to uh predicting the stock market so there's a really wide range of problems um all the problems that supervised machine learning problems they follow a very simple structure uh which is roughly that we split the data into training and test so people uh train their algorithms on the training set and we evaluate them on a test data set and we actually have a few different configurations but very often two test most we we always have two test data sets one test data set where people get feedback as they submit on a live leaderboard and another one where the actual uh competition is scored and so you can see that the uh everyone can see in real time how they're performing relative to each other and typically you can make up to five entries in a competition per day um we now have a community of uh 550,000 signed up um and then the the nature of the cackle Community is quite biry so people have time they'll come in and do a competition then we don't see them for a couple months then they come back so it's not 550,000 that are active in any given month but we have a very large community who who who drop in and out depending on their um availability and so that gives us um that gives us the chance to so so what we we maintain a ranking system so anyone who competes on kagle is ranked from 1 to 550,000 based on their performances and really the ranking system is is more a fun thing than a a sort of a true uh data science rank uh but it gives those who participate act regularly a chance to um uh to to you know to show off and to have a have a cool credential um now we've when when kle started a about six years ago now I I never would have guessed uh that we we could that there would be 550,000 people in the world who are interested in machine learning and data science and statistics uh it's really surprising um so a couple of things have happened one is I think we've been very lucky in terms of timing so machine learning and data science are a much bigger thing today than they were six years ago and and I suspect there'll be a much bigger bigger thing in in uh in six years time but what is it about kagle that engages data scientists how have we been able to build build such a a large community well the vast majority of people come to kaggle to learn that is by far the number one motivation so um it could be that uh you know somebody in their job focuses on a certain type of data science and they're interested in getting exposure to different types of problems it could be that somebody's new to data science and you know maybe in the first competition they finish 3,000th but then the next one they finish 2,000th and it's sort of a way to for somebody to mark their progress um when you're competing in a competition you see you you get to a certain score and then you you often get exposure to the techniques that won that comp competition so it ends up being quite a powerful way to learn at least the at least some some of the some of the things that other people are doing uh that that and and get exposed to new techniques that's by far the main motivation um also as a credential so we show up on a lot of CVS now not so much our Global rankings but if somebody does well on a competition um they'll often put it on a on a on their CV and this is something that we're doing a lot of work on at the moment uh having kle be more of a not just kind of a credential not sort of a fun and frivolous credential but actually a real and very um important credential that that data scientists uh that more and more data scientists can use on their CVS um and then the final motivation is uh research so um it's I call kaggle at the risk of being a little bit crude a really big BS detector so every time somebody publishes a a machine learning paper it's always amazingly good and better than all the other algorithms right that that get published well um kaggle is a very efficient to figure out what what works on uh on different types of um what works on different types of data sets so uh it used to be the case that when kaggle first started six years ago competitions were won by a huge range of techniques support Vector machines uh we even saw self-organizing Maps we saw a really large range of techniques and then by about 2011 competition started getting dominated by random Forest um so it wasn't that random forest was new in 2011 but the community sort of settled on or realized that random forest was the most powerful way to win the vast majority of uh machine learning competitions um then in 2012 uh some researchers from the University of Toronto so it was uh George Dal and and Jeff Jeff Hinton won a competition uh hosted by MK using a new well an old technique rebranded called Deep neural networks or deep learning and so that kind of got our attention a little bit and then another of Jeff hinton's grad students vad men one another competition uh using deep neural networks and so all of a sudden we were seeing um deep neural networks work on a certain class of competitions so text speech images and I'll talk more about this later um and then in 2015 we started seeing um so we we had random forest and deep noral networks winning competitions then in 2015 Tian Kean who's a researcher at the University of Washington uh uh created a really nice implementation of gradient boosting machines called XG boost and now that's supplanted ROM Forest so it's it's a really nice uh CLE are very um it's a very it's a good way for researchers to spread if they have a technique that really works it's a it propagates uh very quickly through the kaggle community because what happens is uh you are competing in a competition you finish uh I don't know 800th and you want to know what the first person did you pay a lot of attention to their their techniques so it's it it works out so if you're a researcher and you have a new algorithm it's a good way to to get attention for it um just a quick I'm curious quick show of hands who's heard of XG boost here yeah quite a lot okay um by the way I'm happy to be inter interrupted so just stick your hand up if you have a question I I think unlike Carlos I'm standing out of the light so I can actually see if you have a question so feel free to put your hand up all right something is Frozen all right um so um kle is so just to give you a sense of the I mentioned that we've had 2 million machine learning models submitted to kaggle uh we're at the rate this is a little bit uh needs to be updated but we're now at the point where we're consistently hitting over 100,000 machine learning models being submitted to kaggle competitions each month there's a lot of machine learning that's happening on on the site um if you think about it every single machine learning model takes around about our estimate is in the order of 5 to 10 hours so it's a huge amount of hours going into to competing in these competitions um okay so as mentioned uh we see a lot of machine learning what what are some of the lessons that we've learned so the first one is that competitions end up being a a really powerful let's see if I can get the click at a work so I don't have to keep returning to the laptop no oh good okay um so competitions end up being a very powerful way to get everything there is out of a data set and I want to give you an intuition for for why this is and and how this works so one of our very early competitions was with na NASA's jet propulsion lab uh in Pasadena California and what they wanted to do was they wanted to uh algorith algorithmically detect the ellipticity of galaxies so they wanted to figure out um given a given a Galaxy can you measure very precisely using image recognition algorithms what the ellipticity of that Galaxy is now they use simulated data in this case because it they needed a real ground truth um and they had techniques that they had been working on for they had quite a sophisticated Simulator the techniques that they' been working on for quite a long period of time uh but they W they weren't accurate enough and they needed to be very very accurate because the the use case of this problem is if you're um if if there's Dark Matter between the Observer and the Galaxy it will distort the ellipticity of the Galaxy and so uh the utility of this algorithm is being able if they can measure very precisely the ellipticity that they can back out of that the Dark Matter distribution of the universe so they needed very accurate algorithms and what they had wasn't accurate enough now very interesting the first entrant in this competition was a glaciologist by the name of Martin L uh and so what Martin did was he uh he used algorithms that he had built for his glaciology research he did a lot of image recognition in his glaciology research what he would do is he would take satellite images and measure algorithmically where the edges of glaciers were so Martin got into first place and uh had actually in a relatively short period of time less than a week had outperformed the best of NASA's uh approaches um and so Jason rhs who was the investigator at NASA's jet propulsion lab po posted this on the NASA on the White House Office of Science and Technology policy blog and he said um the world's brightest physicists have been working for decades on solving one of the great unifying problems of our universe in less than a week Martin oir a PhD student in glaciology uh had outperformed the state-of-the-art algorithms so this is a fabulous result if you look inside the the text of this uh blog post um you can see they referenced Einstein's theory of relativity they mentioned Newton's law of gravity they're explaining the concept of dark matter and white this problem's important so Martin has both a Twitter account and a sense of humor and so what he did was he got on Twitter and he said not that I'm bragging or nothing but the White House has just compared me to Einstein and Newton so he was he was quite pleased um but he doesn't stay pleased for very long because what happens is then somebody passes him and then Martin passes them and they pass him and he passes them and eventually get to this point where nobody is able to get any there's no no more improvements that anyone's really making I mean maybe there's there's there very slight improvements but really there's only so much signal in a data set and you get to the point where you've really extracted all the signal out of the data set um so that's the it's it's one of the things that competitions are very powerful for now what we've noticed though is that different competitions converge at different rates and sometimes we don't run our competitions for quite long enough because you can see these are the competitions Broken Out by desar we've normalized the evaluation metric so it's it's always the higher the better uh and you can see this is a bottom desile competition um you can see that it's still going up it probably hasn't converged uh and then these are some of the easier problems where they converge very quickly any questions before I go on yep um we're getting better at it but it's it's um it's really quite difficult so the question I don't know if everyone could hear the question but the question was how do we estimate how easy a problem is um in some ways we we the only reason we need to is because we need to set a time right we want to host a competition for the shortest time possible uh because most of the work that's going on through here is is just like adding complexity Without Really adding much performance so the ideal is to is to cut off a competition pretty close to to when the that limit has been reached um or conversely making sure we don't leave a problem we don't shut a problem down too quickly um which you can see we have done in a in in a bunch of cases and one limitation we Bas is it's not we can't change the rules Midway through a competition it's really unfair to participants so we we will not do that uh so we do our best upfront um uh to estimate um and so typically we run competitions between two and four months uh two months for the easier competitions four months for the harder competitions and I think what we're going to do as a result of in analysis like this is some of the really hard competitions will extend out to six months um you know there's obviously a bit of a trade-off between maintaining people's attention uh you after a while certain amount of time on a competition becomes less interesting and people want to move on to another competition but also making sure that we're not cutting it off too soon any other questions y so so so the question was is there any what determin what's the theoretical limit the theoretical limit is really um in any data set there signal in this noise right uh so there are things you can predict uh using the state-of-the-art of machine learning and things you can't um and so the limit really is like data science machine learning is not magic you you have richness and you have noise in your data uh and if you've competitions are a powerful way to get to find the signal out of um out of data sets now for a lot of the more numerical data sets my sense is that the limitation is um is more on the richness of the data so the techniques are quite strong um it's a case of there just isn't a lot of you can't do better because there's there's more there's not more richness in the data some of the text speech um and image uh based competitions I think in a lot of cases it's actually limitations of the state-of-the-art of machine learning so if you think about it like this um for image recognition competitions before deep learning uh the the the limit was was much higher um and so as as we are developing newer and newer techniques that you know what what is the the limit of what we can do um uh moves actually interesting um we have this opportunity we haven't done it but we something that would be interesting for us to do is to take a competition that that deep learning or deep neural networks is very good at and rerun it uh now that we have these new techniques it's a good way to give us a sense for kind of what the what the Delta is on the on a problem any other questions before I move on okay yep yeah yeah so people people use uh deep learning wins all the text image and some of the spee sorry all the image competitions computer vision competitions uh all the ones with speech and and audio and some of the text competitions people can use whatever they want and so that's why I say kaggle is a good detector because um you know an academic paper can say this is by far this is the world's best technique at doing XYZ well if it wins competitions you have sort of an objective uh uh sense of what that whether that's true or not so people are not restricted in what they use um okay so this opens a question how do we know that we've hit the limit of what's possible uh on a um on a data set or on a competition well very often when we've hit the limit you start to see correlation maps that look like this so what is this first of all I should explain this um this chart so the top left is the correlation between the first and second placed entry the first and third placed entry the first and fourth placed entry um so what what we've done is we've taken the submission file and looked at you how similar the submission files are and you can see this is pretty bright red right the correlation between first and second all the way through the first and and 19th or 1 and 20th is is pretty much um uh almost at one and so what this is telling you is that people are getting all the same things right and all the same things wrong um and you know when people getting all the same things right and all the same things wrong they probably have reached that that limit right um so it's it's more corroborating evidence that given the state-of-the-art of machine learning and and uh and what you can pull out of the data there is a a limit um now I'm going to show you a competition where we didn't converge this is a competition for the Allen Institute for artificial intelligence and what they wanted to do was predict which uh uh basically build an algorithm that could solve an eighth grade science quiz so this is an example of a question so the human eye has receptors that can detect detect which part of the electromagnetic spectrum so show of hands who thinks it's a b c and lots of people think it's a d we all right um and all the algorithms got this question right as well um but this the colors aren't showing up here but you can see this correlation map is a very different color people are getting dramatically there's a big difference in the things that people are getting right and the things that people are getting wrong for this question which is for this competition which is a sign that it it it did not converge um there's a little bit of there's a little bit of an art in in inter like a lot of machine learning and and data science there a little bit of an art in interpreting this because um this competition also got a lower overall score so the actual um the actual overall score was dramatically lower uh than than um the than the NASA competition so some sometimes people get the things that people are getting wrong are they're getting or or sometimes people are getting if you get a lot of things wrong sometimes you get them right by accident it's a multiple choice qu test so that will naturally lower the correlation but not to this extent um so this is an a question that half the half the um algorithms got correct and half the algorithms got incorrect so you can see that change so this is a a motion with accelerated speed and motion with constant speed represented by different equations uh which of the following best described motion with accelerated speed so half the algorithm said changes in velocity over time and the other half said time over changes in velocity right so you can see what's happening here this was uh the the winning techniques used an information or retrieval based approach and information retrieval based approaches are basically looking for words that kind of correlate acceleration changes in velocity time um uh those words correlate with acceleration and so but the algorithms weren't smart enough to really understand the meaning so they they were just basically word matching right so you learned something about the limits of machine learning um or the techniques that we used in this competition from the things that people get wrong question I I would say it's so well here's a way to think about it so the winning algorithms in this competition scored 60% on the test um and so good humans obviously you would expect uh I don't know how difficult the test was but um I don't know what the average score for humans is on the test but I suspect certainly good students are getting a lot higher than 60% so the algorithms are not not performing that well on um at least when we cut the competition off the algorithms were not performing that well any other questions okay so I'm going to show you one that all the algorithms got wrong and this one's hard uh in many types of bacteria Offspring receive all their genes from a single parent these types of bacteria I'm trying to figure out whether to try and embarrass the audience and ask for uh take a poll for this one I I won't uh so a produce very little Offspring pass on RNA rather than DNA do not have different Sexes receive DNA from from either the father or the mother so the actual answer is C and most of the algorithms said either a with some saying D and again it's something about information retrieval uh based responses so you see the word parent Offspring kind of correlates with parents or father and mother correlates with parent um you know that to to get to this answer you have to have a deeper level of understanding than these algorithms had so the the message here is it's it's quite you learn a lot about um machine learning by starting and what what what machines are good at and what they're not good at by looking at um across all the submissions to a competition what are people getting right and what are they getting wrong um there is also this opportunity to on so if you haven't reached the frontier what you can do is you can start to combine because they're not getting all the same things right and the same things wrong you have this opportunity to Ensemble top models together and get a better performance so I said to you earlier that the best algorithm single algorithm was actually 59.3% so not quite uh uh 60% if you Ensemble models um 1 to 5 you get 61.6% now they're much smarter ensembling schemes than just a flat average but this was just a naive approach uh to ensembling um you can't really see this this is not showing the color is not showing up here but if I do something a little bit smarter so I just look at first and second and the models that they're least correlated with and Ensemble of those you see I I get an even bigger boost so 62.7% so when we have hit the frontier there's no opportunity to unble when we haven't hit the frontier you can start to Ensemble the models together and get even better performances because some of the models are doing well in some areas and some are doing better in other areas um and then finally this is a competition that I'm particularly proud of this is to take MRIs and diagnose heart failure so you can see this is a this is a heart it's beating um in order to measure heart failure what a a cardiologist will do is they'll measure something called ejection fraction ejection fraction is the ratio of the um volume of the left ventricle when it's small versus when it's big um and so what what the algorithm is doing is you can see it's marking when the so d stands for diastolic s stands for systolic I think systolic is when it's small diastolic is when it's big um and so what the algorithm is doing is it's identifying when the heart is full and when it's empty uh and then it's it's got a circle around the left ventricle and so it's measuring the volume and then it's it's giving us a a measure of ejection fraction so this is something that is very tedious for a cardiologist to do it takes them 20 minutes and it's not very fun um I forgot to mention the takes them 20 minutes partly because this is one angle but they the the MRI takes photos at 30 different angles because it cares about volume right uh so this is something that a machine learning algorithm was able to do as accurate as a human now one of the really interesting things about this was um if you look at the correlation map there are two different approaches that seem to be doing well right so first second third fourth are all pretty highly correlated with each other fifth eighth and 10th are not very highly correlated with each other uh and you can see sorry not very highly correlated with first but they are highly correlated with each other uh so it turns out that um uh that the those teams used a technique called active appearance model so it's a it's another technique um uh uh that could do almost as well as the winning technique on this problem so it tells you that there were actually two solutions um lesson number three so two approaches are really dominating compet kaggle competitions I've already mentioned them but I'm going to go over them in slightly more depth so the first is um is what I call the handrolled feature approach and this is using the X XG boost almost everybody is using XG boost so what they're doing first of all is exploring the data so looking at correlation maps and and Scatter Plots and histograms and exploring the data every which way to try and get an understanding of what's in the underlying data set um then they come up with a hypothesis so the hypothesis uh that I like um the example I like is the the uh orange car example so we hosted a competition to predict which cars sort of secondhand auction would be good buys and which would be lemons so which would have a warranty claim against them and it turned out that the winner found that um unusual color cars were more likely to be reliable than standard color cars and the way they found this was they took car color and they group dark color cars with light color cars they grouped um unusual color cars with standard color cars and they tried lots and lots of different ideas and then they they looked at the ones that really did have a have a genuine relationship and then once they come up with their features they' put them in XG boost so most the vast majority of the work is actually exploring the data coming up with hypotheses testing those hypotheses and then actually tuning the hyperparameters is really not it's not where the majority of the benefit comes from it most of the benefit is in exploring the data coming up with clever ideas about the data testing them um the other technique that's doing well in competitions is is deep neural networks which is quite the opposite so you spend very little time uh doing feature engineering and much more time constructing your nor Network so uh convolutional nor networks are dominating image recognition competitions uh you can see on the left we have uh image of the eye and this was to diagnose an eye disease called diabetic retinopathy these are the eyes that have been Pro processed by uh convolutional new network this eye has a rating of zero so it's perfectly healthy this eye has a rating of four so if you have a rating of four on your diabetic retinopathy scan you are blind uh and you can have a 0er 1 2 3 4 if you have a one or two you're you're developing diabetic retinopathy you can have surgery uh either a laser surgery or an injection and it will be cured um so it's it's quite an important disease to catch early um and you see what convolutional nor networks do is they basically do a sort of complex Edge or they do um sophisticated Edge detection so it's it's they sort of pull out the the features of the eye um the other type the other flavor of deep neural networks that we've seen do very well are recurrent neural networks so these work well on time series problems or problems that have a sequencing Dimension to them so we've done quite a few competitions on eegs so grass grasping or lifting um uh is one example so can you tell whether somebody is grasping or lifting from their EEG signals we've also done competitions around can you predict in advance whether or not somebody's going to have a seizure based on their EEG readings so recurrent nor networks dominate these competitions um interestingly with the text so natural language processing text um we see a bit of a mix so we sometimes see information retrieval while the handrolled features approach winning Tex problem and sometimes we see neural networks but it's not it's not clear like there's not it's not nearly as clear as it is for images uh that that deep neural networks are dominating so to give you some examples um I don't have an example but a uh a deep neural network won a competition to take the text of job ads and predict the salary right so this was for a job classified site in the UK they wanted to take the text of a job ad and and get a sense for what the Benchmark salary would be um and a deep n network won that on the flip side that 8th grade science quiz uh example that I gave you earlier um it was won by an information information retrieval based approaches so we really are seeing a mix any questions before I move on okay um and then finally I want to talk about the attributes uh of top kagle participants so really they have four characteristics one is creativity so being able to come up with lots and lots of different kind of ideas features um uh is um is something that uh you know being able to come up with dark color cars versus light color cars orange unusual color cars versus standard color cars they coming up with lots of ideas as an important characteristic um secondly tenacity so you're in first place and then somebody passes you and you pass them and they pass you and it's it's quite you know not being not getting dispirited um when when others are passing on the leaderboard but but going going and and and reimagining what else you could do takes a lot of tenacity and so we notice that's a characteristic of a lot of top kagglers um the second is very good with Statistics so one thing that we see a lot this is a the story of Gregory Park so he um remember I said we have two test data sets one test data set is the test data set that we show you how you're doing on the Live Leaderboard as the as the competition is going and then we throw that away and we retest you to make sure you're not overfitting and so a lot of firsttime kagglers what they'll do is they'll they'll they'll think doing really well they have they'll be on the top of the leaderboard um and they'll be they'll be very excited and and what happens is we flip the when the the deadline for competition is midnight UTC and so what we do is we flip from the public leader board to the final actual leader board at midnight UTC and so Gregory Park would have been uh staying up and he's hitting refresh refresh Refresh on his keyboard he's very excited he thinks he's won the competition and he's not in first and he's not in second and he's not in third he keeps scrolling down the page he finished 50 yes well he overfit to the uh to the public leaderboard and I tell you what he will never overfit again it's a very very good way to learn the lesson of overfitting um but it happens a lot um and the amount that it happens on kaggle makes me it just makes you wonder as you read research papers as you um look at algorithms in production this is a case where we have the test data set quarantine so it's actually not really possible to overfit most cases that isn't the case so it makes me wonder how much o how prevalent overfitting is in the real world uh outside of this world of kagle competitions where we can actually control for it so it's a really big problem and so one of one of you know I think about the contributions to the world kaggle has made one of them that I think is actually quite a Val valuable one is uh people who compete in kaggle competitions know about overfitting and learn techniques for for protecting against it and I think that it's a it's a it's an incredibly valuable um lesson for people to learn uh and just more generally being able to really test uh when you have a relationship and when you don't and then finally Version Control and and good software practices more generally so I would say most top kagglers are using good software practices so Version Control being an obvious one um not everybody but the people who do well in kaggle competitions and aren't using good software practices are probably spending about five times as long on the competition um because what you need to be able to do is you need to be able to keep track of the things you've done that work and the things that that you've done that don't work um and and dismiss the things that don't work and and keep the things that do work um so this is the weakest of the four uh characteristics of top klas but but uh if you're not using it competing in competition is going to be a lot less fun and a lot more work um so I think I still have a little bit of time and I'm going to end off with something that I think I hope uh you all find interesting and useful so um we next for kaggle so kaggle is currently the place where data scientists come for their side projects and historically it's all been around competing in competitions um we we would Al we would like to uh be much a much bigger part of a data scientists uh daily life and so we've been developing a set of uh an environment that that's um that is aimed at at at helping data scientists in in more ways um than we currently do so I'm going to show you a demo uh I'm going to start by motivating the problem so one thing that we notice is that hang on we have forums so this is a competition very this is a simple getting started competition for so for anyone who's new to kagle we have lots of these getting started competitions they're very easy um and they're a good way to get going and uh and then you'll get addicted and then you'll before long you'll be diagnosing heart failure um okay so you'll notice here this is one of our getting started competitions lots of very uh friendly well often friendly Community sharing lots and lots of uh tips and tricks um and one of the things people are sharing is they're sharing code so this person is sharing their neural network code uh in order to get it running you need the same version of python it's it's written in Python you need the same libraries you need Panda py learn mapai no learn and lasagna you need the same versions of those libraries maybe uh you need the same data and so this person has generally really shared their code but no one's really discussing it no one's done anything with it it's just like a hassle is probably maybe half an hour's work to get his code running you don't even know what what you'll get once you run it um so what we have launched is this nice little environment C scripts we also give you a notebook environment where oops you can see you get an r a python or Julia environment the data is automatically read in and you can just start coding immediately you hit runscript uh and we don't charge for it but the condition is that all the code that you uh that you share is shared publicly so just taking a minute to load you can see all the examples of code that people have shared so the highest I've ordered by votes the highest voted script is um a tensorflow deep neural network so a lot of people tensorflow is the Google deep neural network library so a lot of people are interested in it um I'm going to show you a different one though this this one I particularly like so this is somebody has written a random Forest to visualize the digits and the proximity the digits to each other so you can see the sevens are near each other the NES near each other the sevens are near the nines so this algorithm sort of working as you would expect it's classifying the digits and grouping them close to each other in red are the ones he's getting wrong he's consistently getting the sevens with the dashes through them wrong you see this one that thinks is a nine it's right near the nines let's say I wanted to improve on his code I just hit Fork script I get his code running in an environment that that will execute his code there's no faffing about with the environment and I can to start editing his code immediately so um if you compare this so I showed you the example where somebody had shared their newon Network code before if you compare this with the other example um this has been forked 32 times so it's been just making it friction so two things are going on here one is we're making it frictionless to build off what somebody else has done and the other nice thing about it is you see what you get you see what the output is before you spend time uh playing with it um and so it turns out that just lowering the frictions in being able to build off what somebody else has done makes a big difference now most of our competitions have scripts enabled and so it's kind of like a way for people to share tips and tricks and code as they're um competing in a competition so if you're newer to data science you'll see lots of getting started examples of how to get started in this competition where we can just Fork the code and edit it immediately so it's easier to start with somebody else's code than with the blinking cursor and if you're more advanced but you want to learn a new library or a new technique you a lot of the T an Chen for instance the creative XG boost spends a lot of time in here you know creating examples of how to use XG boost for this competition or that competition and so you see you get to see how experts use libraries that you um you may not be familiar with uh so it's a good way to learn libraries the other thing we're starting to do is we're starting to allow people to use scripts for things Beyond uh just um uh competitions so we've we've launched public data sets on kaggle now so this is an acknowledgement that we have a community of 550,000 people not everyone wants to be competing in competitions not everyone has the time people want to do more exploratory type analysis or different type of analysis that is not that competitions are not useful for or not not not encouraging of um so I'll show an example one of the data sets we have up are Hillary Clinton's emails so if I go in you can you can see all the scripts that have been written on Hillary's emails and the one I like is this person did camon analysis as you can see his code he's done camon analysis on Hillary's emails and you can see uh Hillary turns out not to be that Keen on Iraq and actually surprisingly not not so crazy about the United Kingdom either uh she doesn't like Libya um so um and again this has been forked 14 times so at the moment uh we have probably about 25 data sets up what we're actually doing uh by the end of the probably by the in the next couple of weeks um anyone will be able to upload their own data set so let's say you're um you're working on a research paper and you're working with other researchers or you've just finish your research paper but you want to make the the code and the data um public and reproducible by others uh you can you can load it on here and and you'll have a very large audience of people to play with your uh your data set um and further out this will become something that uh you know people there's a we have a lot of work to do on it but it'll be something that um uh we'll probably make available to companies as well that want to share among their data science teams so it's it's at the moment a really good resource for getting to getting to play with lots of different data sets whether they be public data sets or competitions see other people's code learn from other people's code um teach other people uh tricks that you know that they don't know uh so it's a a really become a really popular addition to kaggle and a very good Learning Resource um so with that I may have time do I have time for one more question or are we up yeah I do okay I've got time for a question yep uh you mentioned R of Al yeah so for that um it depends so uh for different competitions there are different rules uh and it depends on the competition host so for the eth grade science quiz competition the Allen Institute allowed people to bring in any Source what they think they packaged up Wikipedia data the science science articles um from Wikipedia it's actually interesting most of the win Top performers brought in other data sets and the data set that were most useful it turned out were not like the Wikipedia Corpus but like flash card data and textbook or I think it was flash card data was very popular and and very useful um which was quite quite interesting and probably an endorsement I think one of the companies for from uh the one of the companies whose data was taken or was used extensively extensively was a company called Quizlet it's probably actually quite a good uh endorsement of their um their material at the back uh and it also seemed like there was sort of like a bifurcation of domains where those uh applied like the neural Nets worked with unstructured data and the ex like worked with ultimately day though was like a little bit more structured like like would you say those are actually Alternatives that come up in problems or it's more like some problems get dominated by strategy one and other problems get dominated by strategy 2 I actually think the way you expressed it is perfectly right so XG boost tends to win um the structured data problems and and the feature heavy feature engineering approach and deep neural networks when images speech uh and then the one kind of murky example again is text so text is you know sometimes text problems uh like an information retrieval based approach is is really um a handroll feature approach uh and we see that winning some text problems and in other ones we do see recurrent NE networks so that's that's the one that sort of fits a little in the middle I think I'm out of time now um but uh thanks very [Applause] much okay the next uh talk will be at 10