Reproducible Machine Learning at Scale

Reproducible Machine Learning at Scale

Recording: Reproducible Machine Learning at Scale

and take it away uh i think you can introduce yourself but let's just uh say briefly that lucas uh is a veteran founder and speaker uh he started crowdflower i think like one of the original crowdsourcing platforms and that transformed through several incarnations and he basically hosted a bay area ai meet up and spoke at it so it's really great to have lucas back and just one thing i'll mention is that uh you also speak at scale by the way conference which is in october october 28 and 29 also online but with a lot more immersive tools than here we'll have you know special charts will have discord and so forth so we really you know encourage everyone to to uh join um skill by the bay and the information will be sent to the top so with that lucas uh take it away uh okay awesome so um so yeah so i'm lucas um and i guess i hope i didn't miss the mark here i was actually intending these slides for um an audience of kind of engineers who didn't necessarily have a deep background in ml the ironic thing is that you know normally who i talk to is ml engineers so i can kind of tailor this i feel like i could have used one of my stock presentations but i made a custom one for you guys sort of assuming um less knowledge so um so anyway it'd be fun i mean it's a small group which is fun i mean if if actually if people um you know want to jump in and talk and interrupt and stuff i think that'd be more fun for me as a speaker just because it's a little bit it's a funny feeling speaking out into the void of people that you can't uh see so um if people do want to you know interrupt and ask questions and talk about their experiences um you know that would be that'd be fantastic from my from my perspective um you know the things that i wanted to cover in this um in this talk is you know like i'm saying here so basically you know first of all i want to talk about you know basically how ml like what's really working in ml today and actually you know you guys have some interesting um you know perspectives here i mean a lot of your companies that that i work with and companies that i don't work with i'd love to hear kind of your take um on how it works and then i also wanted to sort of talk about you know my my perspective on like why um ml engineering is is kind of different and why like new tools are needed um you know i spent most of my career making machine learning tools and so you know i have a strong perspective that new tools are needed but not everyone shares that perspective so we can talk about that and then finally i wanted to basically um show you the products that that i've been making recently um you know less is a sales pitch and more just that's the thing that i know the best is the products that i'm actually working on so i kind of wanted to walk you through um the products that i'm making now and kind of how i how i think about that and you know maybe get some feedback um from all of you does that sound good yup i want to get you i want to get you talked a little bit um yeah so like alexi said you know my background um i started a company called crowdflower um crowdfarm became um renamed itself as figure 8 and sold to a big australian company called appen um a few years back and we basically were a training data service um for machine learning teams so i'm not going to talk about that today but you know i spent probably um you know over a decade working on the kind of training data human loop active learning problem um and then after that after figure it sold i started weights and biases which i think of as a developer first um ml ups platform um i'll say you know i think we work with some of the folks um here we work with um with with cisco and we just actually announced a partnership with um nvidia um yesterday but nvidia's been been a customer of ours for for quite a long time and we've had a lot of conversations with the um the ai the various ai teams with this big company about a child have you actually heard of weights of biases did you did you come across them i said in video uh so i haven't come across it uh as a part of my work but i did watch an interview that you did with nico uh he yeah yeah uh he is my director so uh is it uh are you actually working with with my with my org or you know we we we kind of do a bottom-up strategy so we don't always know but we haven't we've i think at this point we've sold into several hundred licenses so i see i don't know where the the teams live um but maybe not europe because you sort of make a product then for car companies but um i love the interview with uh with nicole um cool um so yeah so so um you know that's just that's kind of where i'm coming from here i thought it might be fun to tell you um you know before i did all this actually back in like you know 2004 2005 i worked at uh yahoo which was kind of a big deal you know back then in the you know this was uh quite a while ago but um you know i was basically running the um dml program to make the search engine work better um and and so we were actually switching from um uh kind of a essentially like a hand tuned system with like a little bit of graded descent on the parameters to like a full-on ml system and i just i thought it might be fun to just i was thinking about my my workflow back then so back then we did um we built the grading boosted tree system so we had basically human labelers label search results as good or bad and then actually use an r package um called gbm back then i even you know at that time it wasn't obvious that python was like the language of choice for ml i think r was pretty um you know the new research algorithms would get would get built in r first um and so i actually remember i wrote code to sort of take the trees and generate c code like literally c code with a ton of go to's you know because it's um it's a red it's such a random forest and then um what another it's a gradient boosted trees i guess it's slightly different um and we would actually um would actually check them into um the production code that would run live and i remember um the the engineering team like asking me like what the hell i was doing because the dips were so enormous back then um but these followers were like small compared to what you would see nowadays by the time i left yahoo i had checked in i think as much code as every other team combined because the code was being automatically generated um and then um you know we we basically deployed into production and then tested an evaluation set which like is not really best practice in my view but um you know it was too hard to to test the search algorithm in any other way it was too expensive to really build a test cluster at the time then we'd do an a b test and then we would um deploy it into ga and the whole process took about a month or two um you know kind of from you know original training and seeing that a model was better to um deployed into production and and we ran into all kinds of problems with this process i think like kind of every you know ml malfunction that i'll talk about um we hit uh we hit with this so anyway um i have i have worked as an ml engineer um now mostly serve um ml engineers um so i kind of want to first like i guess ground things i mean you all have a perspective on it since most of you are actually working in ml um but i you know i do feel like there's like a lot a lot of um hype out there so kind of want to talk about the applications that i'm seeing i actually think like you know i don't know how you all feel but to me if i was graduating with a um a background in ml this would feel like a really exciting time i think like i remember when when i um when i graduated in 2004 i really felt like you know i loved ml i loved doing ml but you know there weren't very many companies um out there it was sort of like you could rank advertisements or you could rank search results um or you could work for a hedge fund where it was kind of like the career opportunities for actually doing ml um and now there's just a crazy number of options which are um all very cool i'll say um i think like one thing that people um i think one misconception people have is that there's these like massive breakthroughs and i think like mostly what i've seen you know over the last two decades working in ml has been more sort of steady iterative improvement than these sort of like step function changes of course like i think when the application starts to hit a threshold sometimes you get like a step function change in adoption but sort of interesting i just pulled this there's a lot of graphs like this but this is like actually in the economist they were trying to um you know sort of look at uh speech recognition error rate and you can see it is on a log scale but you can see it's kind of a steady reasonably steady improvement but then you suddenly hit this threshold in speech recognition and you know everyone is is willing to like use an interface right like i feel like most people these days you know if they don't care too much about privacy they you know they use alexa or um or siri or or um or things like that so you know it sort of feels like a step function change in adoption of speech recognition but i think from a algorithm performance standpoint it's a much kind of smoother um curve um i'll tell you something i just kind of want to give you a quick tour of of the stuff that i see that i think is is really cool um one of my one of my favorite examples i don't know if any of you have seen this before um this is a subdivision of john deere called blue river and this is actually deployed in fields in in california this is a vision application um there were actually customers of my last company figurine and weights and biases so i know this really well they they basically look for weeds in a camera facing down in a tractor and then they they try to spray just the weeds not with pesticides actually but with fertilizer and so the fertilizer burns the weed and kills the weed but then the fertilizer is useful for for growing the plant so i just like i love this example because this is the thing i really would have loved to work on you know coming out of grad school um because it's like you know it really um you know it uses pesticides there's sort of less impact on the environment it's actually like more efficient right it saves money it means you can grow more plants and um it's also like a really cool um use case and it's it's really only powered by ml right like if you if you mess up if you if you um you know spray the lettuce with your fertilizer and burn it you're really gonna cause a problem so it's like a really high value application um for the farmer um i'll say like we actually got them to do a really in-depth um case so say case study it sounds sort of like it's going to be like this like kind of businessy high level thing but actually we were able to convince them to publish um a lot of their code that they used and actually their whole tech stack so if you really want to get deep on um you know how ml's working at uh at john deere for this application um i put this link in here um wmb dot me slash uh john deere that i i would totally recommend um checking out i'll say that you know their stack is is pretty typical um of what we see these days right it's it's basically pie torch on top of um slurm which they use to manage the workflow and everything runs on kubernetes and docker this is all in training and then they're actually using jenkins as their ci tool which you know is typically um uh you know that's actually kind of not necessarily ml only but but that's what they used to deploy into production and then kind of onyx to do the um the edge deployment um have any of you heard of this use case before am i am i using this too much is this news any of you a child have you heard of this use case before yep uh this is this is not not not new but yeah yeah yeah yeah i guess i guess it's fairly fairly well known at this point um and uh and can you tell do you uh i guess can you say something about the use cases that that you're working on yep uh so we i i work in the data set development team so basically uh we build the uh so when when when you're training a machine learning model that goes and sits in the car you want you want to know what data that model was trained on so my team is responsible for giving out data releases where we have some standard set of data that's collected uh over you know through different mechanisms uh ingested indexed and you know for you could have a data requirement that's like uh i need 3 000 frames uh of of nighttime data in japan and and my team is responsible for for fetching that data uh basically driving the cars and uh and and packaging it in a way that it can be consumed for training well so when you say fetching the data you mean literally driving it's not like looking at a database for that data it's driving the car it's it's a bit of both actually so we already have you know data that's ingested and indexed so if if part of that data requirement is met so we uh we if so we first check if do we have the data that meets this requirement and then if we do we we figure out a way to package it version it and send it out to the perception team uh if if we do not we figure out okay where do we die where do we drive to to get this data that's basically what we do well that's super cool does anybody else want to share the the use cases they're working on i'd love to hear these i'm trying to remember who said they were an mlg gita you said you're working some of this stuff right ah actually i'm not actively working on it i did in the past i worked on the system for the u.s post office to sort mail really wow that's so cool really to like to decide yeah this was in the uh actually in the 1990s i would say so it was deployed by the post office to sort mail back there were they using like a um like a learning algorithm to do that in the in the 90s uh it was using uh neural networks and different but neural networks were kind of not practical to deploy them so they were rather small not multi you know very deep ones just a few layers but still neural networks in the 90s for the post office yeah but we also had a lot of other algorithms more traditional ml algorithms and we also did a classifier combination kind of approaches and there's a lot of image processing as well in that application oh cool yeah um and anyone else want to show i mean neil you're a bit vague about your applications do you want to tell us about any any that you're working on oh i worked on a bunch of stuff on search and worked on infrastructure at oracle you know we use machine learning for search at google uh you obviously don't want me to go deep into what we did over there um yeah i've worked a lot in machine learning i mean well we did it huh what about at oracle like what were the kind of key so oracle the main thing i did was build a runtime platform for machine learning and what what but like what were the models doing that you were using the runtime platform for let me increase the volume what were the the the models that you you ran on your runtime platform what were they what was the purpose it was mainly for sas applications things like you know resume ranking or you know which inventory item to order next that sort of thing so pretty much every area i mean oracle has a very large sas group about 16 17 000 people at least so lots and lots of different use cases so this was horizontal infrastructure not critical stuff um cool cool well these are cool things anybody else want to share um in smart integrity for our company we are using the human level data and and the pipelines to reduce integrity reduce harms all sort of problem for facebook search we have some cool applications as well like in arvr for oculus cast and also there are some interesting other applications um other than the voice assistant and all of that that is to actually automating the process of reviewing prs really yeah like things that can really mess up the production you know like can create a save or something so finding those interesting scenarios in automated way with machine learning signals um and generating features for that so there there would be there would be some interesting cases i would mention wow that's really cool oh man i want to hear your talk on that that's a i did not know that was really working in the real world um i guess i'll just mention a couple more you know i think like you know i would say one big area where we see quite a lot of um [Music] real energy uh going so my last company um did not do hardly any work with pharma companies and now at weights and biases we see a ton of drug discovery work both that kind of startups like um you know and citro and and others and then at like the the pharma companies like you know genentech um and and roche navy and others and and um you know uh there was a really exciting announcement i think um about a week ago google kind of finally open sourcing at least the the sort of runtime of their alpha fold um algorithm so we actually have we made a little collab where you can run alpha photo fold on your own protein sequences if you want to um you know go to this shortly but i think um i guess gregoscopy maybe we don't have any drugs in on the market yet you know that are that are fda approved that my feeling is that um this is this is definitely coming um down the road um i'll say you know just other stuff that we see quite a lot of um you know there's a lot of interest in robotics and vision i think the vision working more than the robotics right so you know for example we see um you know robots going up and down aisles to check prices in the supermarket i think here you know the the navigation like reliable navigation has kind of turned out to be harder than doing the vision um recognition of what items are actually on the shelves um and so you see like standard ai you know building um stores that you can go into in in downtown san francisco and basically like it'll automatically watch the stuff that you take off the shelves it's been kind of interesting to see the vision i think is you know feels a lot more well a lot more kind of usable in the real world than um robotics at this point um you know other like medical applications i think medical imaging has been like a huge um growth area lately especially with like you know kind of deep learning helping vision algorithms um work better so we see you know kind of precision medicine um in in lots of various forms i mean people often talk about cardiology and radiology but i think um you know maybe where we see this the most is in identifying cells in um uh microscopy right where you're looking at exactly what the cells are which are the cancerous cells kind of what shape do the cancer cells have and and what that will do is let you know um both sort of like diagnosis of exactly what type of cancer does someone have and then also prognosis of you know um you know how how severe um is the cancer this is this is definitely um this is getting real world adoption and then um autonomous vehicles i mean we have some people you know here like a child that know more about this um than i do um but there's obviously in the last couple years there's been just an explosion of um of interest i mean not just um you know not just cars but like delivery robots and and drones and and um many different takes on um this kind of semantic segmentation problem um and then i guess i wanted to sort of like play a nod to like i think the vast majority of what we see are maybe these less um less sexy applications like um you know demand forecasting um credit scoring we have a little um demo here um and want to be w x g boost to sort of show um kind of a very real world inspired uh credit scoring situation but um but you know like most of the stuff is kind of less you know beautiful for slides i couldn't even figure out what would be a good image to put for credit scoring but um i think nearly all of fintech is doing some version of this and then there's this the adjacent kind of like fraud detection um which is is basically all moved to um to ml i sort of imagine that um these are pretty similar to the types of stuff that um that neil was talking about at oracle i'm sure you had some of these these applications um so i guess you know i want to talk about how like kind of machine learning engineering is sort of different than um you know kind of like software engineering and i'd probably be preaching the choir here but i will say you know when i um you know when i started weights and biases you know a lot of the pushback that i got from people you know i wanted to start this is about maybe five four five years ago starting a um mlaps um company making like developer tools for ml engineers um a lot of the pushback that i got was like hey you know do do we really need different tools um for ml than we do for um for software engineering um and actually have a great interview um i think you'll especially appreciate this with um this is with um anantha who was at the time the person running the autonomous vehicle program at uh at lyft and and he was basically like actually his background is not an ml but his perspective on ml he's actually facebook now um but his perspective in in my interview with him was really like um you know ml is not so different people act like it's different but really it's just a subfield of um of of software engineering we just hire people that we kind of use mostly best software practices um to to get this deployed in in the real world um so if you want to counterpoint what i'm saying um i definitely recommend um going to this uh this little link that i have here at the bottom and checking out what he had to say um i think like there are there are a bunch of clearly a bunch of differences though i think the first one is that um you know like all the software abstractions get broken when you switch to deep learning running on a gpu right like i feel like just that switch from a cpu to a gpu or tpu it sort of breaks the whole stack and i you know i feel like um i mean i remember in like you know 2003 2004 kind of feeling like i encountered my last linker error but you know in the last couple years i've been running into like a lot of linker errors right because you know i'm using kudo which is sort of less um mature and then like i think like when you break the very bottom of the stack of the gpu you end up with a whole new um stack and so like just actually setting up an environment um to to do deep learning um is like a huge ordeal for a lot of people and requires like a lot of knowledge of um hardware and software and and it really matters right so i think like you know another thing i'll just like note um is that you know kudanan is this library or coup dnn um is essentially a library made by um by nvidia and it really is just like a kind of a compilation layer from like hey i want to run a cnn and then compiles it into cuda code that then runs on a gpu and it is amazing how much performance gains you can get from optimizations in that library i don't know if any of you are kind of hardware people but this always feels uh magical to me that you can get like a 2x speed up from a new version um of such a low level library right which i think shows there's just like a lot of room for improvement um you know across the stack right and so um yeah again this is maybe for you know someone who's not deep in in ml engineering right but i think like you know you'll have like you know on top of kunan you'll be running python then typically be running um pytorch but you know even that um is often you know too low level of uh of a library like typically we'll see somebody running lightning or on top of tensorflow they're running keras right and that's actually the main interface that we typically see our customers engaging in deep learning and then finally now you see this explosion of interest in even higher level libraries like you know i'd sort of consider hugging face in this category where people are basically starting with these um these these frameworks right so there's quite a deep stack now um and and the stack is totally different from from software engineering um you know another point that i'll make and i'm kind of preaching the choir here but it's kind of worth noticing is like you know the the um the ml model looks a lot more like machine code than human generating code but there's like some other key differences here you know the the ml models these days um are essentially the size of the ram on your um gpu so these are like you know typically like gigabyte size um things that you're deploying we see your customers making you know thousands of them they're not diffing them right so like they you know you train the same model on two different days in the same training data it's function equivalent but literally every single character in that file is um is different right whereas like you know if you if you have like a v2 of your software stack um you know you might change 10 of your code um would be a lot and it would be sort of like in specific areas where you could look at a meaningful diff right but you know the song the ml code just does not um default at all um and so i think what happens is um you know you see that like um like like i'm saying the debugging is really more like observing what's going on in your model and then um you know making adjustments right so andre kurpathi um gave a talk at our user conference a few years ago and he talked about you know amount of lost sleep um over over data sets or models and algorithms right but like you know notably here i think if you're a software developer you wouldn't be losing sleep over data sets or models and algorithms right what you would be doing um is kind of is looking at you know like requirements and stepping through your code and modifying your code to like fit new requirements i mean here you really either you have this option of sort of like changing your model algorithms you know what it does or changing your data set and seeing what it does and it's it's it's much more like observing what the model does and then um making adjustments so i'll give you an example you know everybody talks about like explainability problems with with neural nets but it's kind of interesting you know where explainability has gone right so this is one best practice that we see um for explaining what an image algorithm does and it's basically blanking out different parts of the image and seeing what changes right so you know this is image of a dog playing an acoustic guitar and if you run it through um you know like a resnet you basically get sort of three labels um electric guitar which is wrong acoustic guitar which is right and labrador which is right right and so in this case i guess electric retards the highest probability um at 0.32 and so you want to look at why does it think that it's an electric guitar and you know it's interesting right so like you know if you if you basically blank out ranges of pixels you can see which pixels are contributing most um to the the decision that's electric guitar you sort of see it's the fretboard and it's the person's hands um kind of pointed upwards which makes sense right that looks just like a um an electric guitar and it needs to kind of learn to you know also see the the you know the body of the guitar makes it an acoustic guitar but i think what's interesting about this right is it's not stepping through the model at all right i think like when a lot of software developers get into deep learning they really want to look at okay like what are the weights in every layer they want to like walk through what's happening and i think that was like hard even back in the days of sort of like you know two-layer multi-layer perceptrons or svms it was not really feasible i remember it back at yahoo with trees people always wanted to step through the trees but it never really um led to insights but i think now it's just completely infeasible um to walk through what the models doing so instead what people do is they sort of treat the model like a black box and observe what it does right so here we're kind of modifying the input image into the model and seeing um what changes so the explainability it's a funny notion of explainability it's really different from software where you're not really explaining what it does from a kind of first principles approach you're more treating it like a scientist might and so you know what happens is i think like you know debugging ml is incredibly hard there's actually i don't know if any of you have taken the full stack deep learning class it's a free class online it's really really excellent and um lecture seven is one of my favorite lectures on on debugging and you know i think like one of the issues here is that you know you can get all these different um problems and none of them causes the systems to throw errors at all right and this may be an opportunity in ml tooling um to kind of you know suss out these problems in advance but basically you know bad hyper parameters um bad data set like you know bugs in your implementation they all look the same which is just you know there's no error it's just like your model performs worse right so um debugging is a real challenge actually um i'll note that they they actually um this is a bug that i've had um and this took me a week um to resolve remember for me which is i don't know if any of you know this you know glob is not um deterministic so if you run the glob function on a big set of files um you'll get a different order um each time did any of you know that this is beating into my head because i ran into this what happens is basically if you use a glob for the features and you use a glob for the labels where those are in files they'll mismatch right and so what will happen is you'll basically get um it'll be like kind of impossible for your model to um perform well um so it does look just a maddening bug but anyway i recommend going to this link where they have tons of tons of materials and debugging but again all these things are really more around making small changes to the way the model trains and observing what happens versus kind of stepping through the the model's code also um you know i think like one another really interesting thing and this is like maybe for like management problems but you know i'm mostly a manager these days but like we just really don't know what's hard and what's easy in ml and we never really have good intuitions at all and then like you know i go back to the the movie um 2001 space odyssey right where uh there's this amazing scene where the astronaut you know banters with the computer um and also uh plays it in chess and apparently they wanted to take out the part where the astronaut beats the computer at chess which is so interesting right like you know because chess actually you know computers got better than humans in um i think like 96 with um with deep blue and better than than the average she was better than the best even 96 better than like the average human you know maybe in the late 80s um but in the 70s when this movie was made they want to actually remove the part um with the the the chess playing because that seemed just so impossible that a computer could win at that but then they didn't you know consider the fact that like bantering with a human is almost an ai complete um challenge so we should be very um you know cautious about you know our predictions on on you know what how am i going to perform um over time and what problems are easy and hard i kind of came across this running a kaggle competition which tackles this site i don't know if you all know it where you can basically submit um problems and then you can kind of pay a reward for the person that gets the best solution it's actually kind of amazing um if you are if you want to sort of see how hard a problem is because you'll get a lot of people like thousands of competent people engaging with your problem with like a prize of say like twenty thousand dollars so i put my i put a task in here um there's actually a search relevance task so you know neil you'll find this familiar um and um basically predicting the um relevance of a document to a query and you know the accuracy basically doubled in the first three days right so um you know i was like thinking wow this is going to be maybe a too easy of a task maybe look it's like 99 um accuracy and actually over the next like 90 days it just completely flattened out right so you can imagine as a as a manager of a project like this that had this kind of character it'd be incredibly frustrating right because you'd sort of be predicting um you know great success for your um modeling and then um you know you completely run into a wall and this is like you know the best teams would run into the same wall because this is thousands of teams across the world i'm competing and the line here is their best accuracy so you know there really is like you know without like a you know change in like the state of the art of um relevance this is kind of the best accuracy that they can eke out of such a system and you know we see the same thing i mean the self-driving cars i hate to pick on it but just i think they're the worst defenders here where um you know you see like in 2015 this graph of like the miles per disengage that google published right and so you see this kind of exponential curve you totally imagine extrapolating from this curve like wow you know self-driving is um you know going to get solved in the next few years and of course there's one guy who keeps doing that right so elon musk in 2015 says you know we're gonna have full autonomy and um you know two years from now it should have been 20 i guess he's predicting 2018 then he changes the prediction of 2017 kind of looking at that curve and is is um you know clearly um wrong there um so the the forecasting is super hard um i i mean software you know you said you say like um you know what's the rule of thumb like for any project like double it and add one or something um i think you know for ml it's just like you have no idea like if you know if if uh you know here you're probably yourself tomorrow maybe solve never um and i think we just really don't know how to tell um what what things are easy and hard you know there's another thing that's kind of related which is this what we call the reproducibility crisis and you know pete warden had like the initial blog post on this which i think was really good i mean he's sort of like talking about a problem that i think has kind of plagued ml for a long time which is like it's actually getting harder and harder um to reproduce the work in um you know the the say their papers and it's actually more and more important right like most teams these days are actually reading um the papers and really trying to like you know get right to the state of the art which is i think different than other fields there's just a rich um uh i mean the academic work is incredibly relevant to teams you don't necessarily want to take all the complexity of it but um you know for example when transforming google published transformers um uh or the when google published attention is all you need um that got its language systems you know in six to twelve months it was deployed in a lot of the state-of-the-art um language systems that you that you'd run into so um i guess like you know for someone that's not in deep learning you might wonder like why is reproducibility so hard because like software is like pretty deterministic and pretty reproducible i mean there's like threading issues that are maddening but other than that um you know not so bad right and so um you know the um there's there's just like actually lots and lots of places where stochasticity creeps in i mean one is just literally the order of operations in your gpu is not necessarily deterministic right so like you know you sold like everything you like nailed down um you know your random number seed you know kind of all the upstream data these days reproducibly still might not be possible just because of the um scheduling right so like i think like um the really when people talk about reproducibility there's like many different levels of it um and what i would say is that um you know there's sort of like the more constraints you put around yourself the more reproducibility you could have but truly being able to like run the same learning algorithm twice um and and getting out the same the exact same model um i think at this point is basically impossible um there's this interesting checklist i think put out by mcgill and you can see that it's a pretty detailed checklist like pretty tough i mean it's probably worth doing um but uh there's quite a lot you need to get right to get um reproducibility and then you know andrewing um you know put out this made this point that like basically deep learning is continuously sort of increasing our ability to consume more data and get a better model which causes us to build bigger and bigger models and you know as the models themselves just even training them you know get something out of magnitude of days or weeks or more um reproducibility just gets really infeasible there's like a great uh like a great summary blog post from open ai which talks about you know kind of the breakthrough algorithms and their resource consumption and clearly that's just exponentially increasing and you might say you know maybe that'll stop right like maybe you know we'll kind of figure out ways to um train more efficiently and there's like tons and tons of work just like there's lots of work on um uh model explainability but there's lots of work in model explainability because explainability is hard and getting harder right so the fact there's always work and explainability doesn't mean that explainability is like on the verge of getting solved and then you know with um uh with with resource consumption it's kind of the same thing where there's a lot of work on um on improving the resource consumption kind of getting more with less um cycles because these models are so resource heavy actually i think this blog post is kind of famous this graph you might have seen it before but i think this graph maybe more important and got less attention which is that basically the um the resources needed to train models to alex net level performance have also been decreasing exponentially so while the overall um uh resources going into the biggest models is increasing exponentially the efficiency of training is also increasing exponentially right it's just kind of increasing exponentially at a lower coefficient so i think this kind of shows you and i really recommend going to this blog post i put a link here um what this says to me is that basically um as things get more efficient it actually becomes more rational to deploy even more resources at a problem because you can get even higher level higher levels of performance from your model so um i don't think that the resource consumption problem is going away anytime soon ever i think actually resource consumption is going to continue to grow at a um at an exponential rate and then i think um uh charlie you were talking about this a little bit but you know the dependency graph is getting more complicated for real-world use cases because typically the data set that goes into training a model um some of that data is generally generated by upstream models and then in a way um you know there's there's things that you might call model training by not but like you know kind of data preprocessing steps um can be critical for model performance and typically happen um asynchronously from the model training so you know a lot of our customers just want to know um hey you know i deployed this model production there's this one piece of data that i got a takedown request from i want to make absolutely sure that that request you know that data that someone wanted to remove is not in my deployed model seems like a simple question actually really hard um for even the best companies um to answer right now um and then kind of a big effect that's happened is essentially um transfer learning i think the first i mean transfer has been around for a long time um this is like where you basically you know you have one place where data is cheap and one place where data is expensive and you do your training um on the thing where data's cheap and then you fine tune it or mod you kind of do the last step of training on the more expensive data and i think the really amazing application of this was the dermatology paper in nature where um you know basically these scientists trained a model on imagenet which is like pictures of um you know stuff on the internet and they fine-tuned it on um images of cancer and built a state-of-the-art um cancer detection system but this isn't in imaging and it's i would say you know transforming is like table stakes in vision at this point but it's also i think at this point table sticks in in um in language processing and this is like you'll think of word embeddings there's like one way to look at that is kind of a transforming so there's you know there's bert there's glove there's gpt3 in fact um you know people talked about prompt engineering which is kind of a new concept where you don't even i mean with bert you would actually kind of include the embedding into your model prompt engineering is kind of something anyone can do i think it's like a very engineering approach um to this but the idea here is basically you take like gpt3 and you literally just sort of like give it a couple pairs that you find tune on where it's like in english you say give it a sentence and then you give it like a translation of that sentence and then um what will start doing is just replicating that right so here someone made you know an emoji translator by just like putting in a few examples of emoji and then um gbt3 did did an amazing job of building a emoji translation system and this you know works in some cases not in others but i think we're going to see more and more of this um this type of thing um and then you know finally i'll say you know um i don't want to get like too deep into the the ethics and ai in this talk because it's a big topic but you know i think you know we obviously have seen a lot of cases where um people have been worried about you know models um deployed like they you know they sort of exhibit bias you know often the bias comes from the trained and it can also come from the embedding these days so you know it's hard to imagine what even an unbiased trained data set would mean but you know even assuming you had that bias can creep in from embeddings and and people have been embarrassed in lots of different ways and and um i think is only going to increase in scrutiny is going to increase there's a great book um real world ai um that was written by um actually my old my old head of product alyssa um who works at blue cross blue shield now but um you know kind of goes through the different cases and gives some suggestions especially for um yeah this is kind of especially good i think for um people more managerial level to like how to kind of organize organizations to to avoid this but there's a lot of case studies around stuff that might happen um so anyway i want to talk about um you know i want to make sure that i get a chance to talk about the stuff that i'm actually like working on you know right now and and how i'm thinking about that and i think like you know the thinking behind um my company weights and biases was something that kind of bothered me for a long time where like you know i feel like there's like this developer workflow where you um you know you basically um we kind of know how to deploy code into production right you'd never um build a system that didn't include um version control that didn't include ci cd that didn't include um you know monitoring production there's like all these companies that actually do these things for you and open source projects and you know sort of best practices around it and it felt like you know we're increasingly deploying um ml models into production applications and the workflows are totally ad hoc right like most the people that i talk to um don't use git to version their models and i think the reason is like i talked about earlier um you know that the models don't diff well the model files are enormous um the diffs even aren't that interesting and also there's sort of like um you know multiple different projects happening at the same time in a sort of like you know workflow and so you actually want to have different version controls and lots of different stuff at the same time but anyway where people end up typically they're putting their models into um uh you know s3 buckets or um you know gcp buckets and that's crazy right i mean like that's really dangerous i mean for some applications it's dangerous for some i mean for some applications it's dangerous and that it could cost the business lots of money but i think there's clearly room um for improvement here and so you know these are the kind of developer tools that that um i've been working on and i'll i'll kind of walk you through some of this stuff in a second um also say um you know this is kind of in contrast to other mlaps um things i think are also really interesting right so so like you know there's also kind of like automl stuff where it's sort of like automate the entire process i think data robot is a great company does this google automl is a really cool product that works you know phenomenally well in a constrained environment i think the issue with automl really is that i think that companies are not precise enough about what they're trying to optimize for like in the real world you're not really optimizing typically for one metric unless you're like a hedge fund and then maybe you can optimize just for making money but even then there's like risk as a factor and um you know but for like a typical application there's actually a lot of factors you know kind of at play and so i think like you know automl can really be kind of like dangerously over um optimizing um and i'd say in the real world mostly people don't do pure automl and everyone's been saying for the last like 20 years that like you know autumn l is a couple years away and my perspective is that you know we don't see like software getting automated away and so i kind of don't think we're going to see like ml getting automatically of course we'll get more and more automation but then i think it'll lead to new problems that you'll still want an ml engineer for for a good long time um there's also kind of these pre-built um models and a lot of companies do this right where it's like you know hey i have a credit scoring model for you maybe you can fine tune it i don't see a lot of this a lot of traction for this it's a really good idea it seems like really useful but it does seem like every business has like slightly different variations on exactly what they want we i just don't see a lot of pre-built models getting used in um you know production if if if you see differently i think differently i'd love to know about it um and then finally there's like this this whole genre of company that's like more about this kind of data pipeline and giving you like a place to train your models and bricks stage maker paper space all these products that i really respect they're all very good in different ways um and and they some of them call themselves um mlops but it's much lower level definition of um mlaps than than i think about that at um at weights and biases um also say you know like you know i think like people probably get like a lot of entrepreneurs coming here like pitching paperwork we have something a lot of people use i mean you know facebook research um uses quite heavily um you know in in citro is like a big bio companies heavily nvidia you know like i was mentioning use this a lot and i feel you know i feel really proud like you know making software um for people to use and so it feels really good to see teams actually um find it useful um and i'll show you what it does in a second but i kind of wanted to say like things that are like important to me in the product i mean the one one really key one here i think is interoperability which is actually a really hard thing to do in the ml space because it's changing so fast right so you know i think if you're trying to make an ml tool um i mean one challenge that you have is like two years ago probably tensorflow was the most popular framework and and it's probably switched to pie torch you know in just the last year or two we see like lightning getting a lot of momentum um i mean things are changing super super fast i mean hugging face kind of came out of nowhere to become like standard and like feels like a year um so kind of in this changing world where people have like lots of legacy systems lots of new stuff i think you really have to be able to work um with everything um out there because because most real world teams that have been doing this you know for a while um they have a bunch of different different systems i mean it's funny at facebook you know you think like they make uh or they're like famous for making pie charts you know you see a lot of um cafe there you see a lot of experimentation with lightning and fast ai and lots of different um lots of different stuff and boosting algorithms where it makes sense um and um and i'll say like the the kind of way that we wanted to design the product was sort of like you know these kind of three principles mine's like one is like a really simple integration um just because you know everyone's so busy like we want people to be able to try it really fast and then we want to make the ml developers life like individually really easy but then you know once they're once their life is really good we wanted to add um you know collaboration um on top of that so i'll kind of show you uh very much left here um i'll show you i'll show you a quick demo of how how our stuff works so yeah typically you know we see training code in python that's the vast majority of what we see um and so you basically would import our library um into your training code would be a typical um starting place so so um you know you put a library if you're using keras or lightning or pytorch you have kind of specific integrations for those libraries so in keras it's it's a you know 1db callback that you add to your training function and each library is kind of a different natural integration but the important thing is like two or three lines and then what happens is as you're training your models it's reporting um as much as it can about what's happening so you get these nice graphs of like the accuracy here so this is an example of training actually a semantic segmentation algorithm um trained on on roads and so um you can see the gradients you can see actually here so you can see the actual task where there's an image from a car like a car and a road and a sidewalk and then the goal here is to label the car and the you know the road and the sidewalk and the bicyclist um here so you can already kind of see you know why this is like a complicated um task right so like you know road accuracy here looks kind of flat if you look at you know car accuracy maybe that's going up i mean human accuracy you can imagine is like you know extremely um important um traffic accuracy may be less important right so you know here we have a case where the the traffic accuracy is going up the human accuracy may be flat um you know hard to know if this is like uh if this model is really like working for the use case we have in mind um or not even though here the training loss overall is going down as the um as the model trains um you know we also try to capture as much as we can of of the of the model so you could reproduce it as much as possible and sort of like the more best practices you include the more reproducibility you get right so you know we'll save the git repository and the diff against the latest um get state i think my my strong recommendation for ml practitioners use docker everywhere so if this person was using docker we would have um you know the docker file saved so you'd have like you know the state of all the libraries and things like that you know we also save the hyper parameters and and um and basically as much as we can to make the model reproducible um you know then um i think like one of the things that we realized um in the course of this is that you know looking at like an individual run is not as relevant to an ml engineer as looking at like a whole bunch of runs um together right because what you're really doing here is an investigation of sort of what data what hyper parameters kind of what strategies lead to um to better models and so here we make this big table where you can look at you know kind of all the different runs um or all the different experiments that you have and the input values the hyper parameters really and then the output metrics so you also like you know you kind of generate graphs you can generate these things where you can explore the space um you know try to find the regions where maybe your traffic accuracy is higher you can also you know do things like um you know look at what parameters are important with respect to some metric actually what we do behind the scenes here this was kind of i think a long-standing best practice among researchers but but um you know no one ever really published paper on it we basically train a random forest where the input is the hyper parameters and the output um is the um metric right and so then we look at because random forces have a great um they're really robust and they also have a great notion of future importance we can say like okay you know here the um the config parameter num valid is probably the highest importance even though the correlation is smaller and then using probably an alex nut encoder has a strong negative correlation in this case um the correlation is a lot higher but the importance is about the same which means that it interacts less with with other features and so this is like typically the place where some like an ml engineer would live in our product but the thing we really want them to do because we really want collaboration here is to hit this create a report button and so what you do here what happens when you hit that create report button is you can actually save a snapshot of something that you did and this is almost like a pull request for for an ml engineer where you can say like okay it looks like um alexnet has worse performance right and now i can save this report and and the nice thing here is that like this investigation that i did where it's actually a negative result right so i probably wouldn't publish this and probably wouldn't talk about this too much um easily could get forgotten um but now it's safe forever if someone else comes in they're thinking about alex and they can find this report they can see that that was kind of the case you can see my conclusion but importantly they can also investigate it right so they can say okay these are the 224 runs that led to that idea you know what was the state of the code at the time what were the hyper parameters considered and they can decide if it's like relevant to you or um or not i should say like one thing that i'm really excited about is that a lot of folks have um a lot of our academic users and and you know make it free for for academics is is basically um uh uh they will publish um kind of many results they have kind of in the the tool here so if you actually go to www.ai slash fc you can see a big curated list of um different results that have actually um you know been been published on our platform so you actually find a lot of kind of like quick little nuggets of like what works and what doesn't on different um different use cases um so the next thing that someone like typically wants to do in this case is do a kind of automated hyper parameter search and like there are a lot of tools out there to do um hyper parameter searching both like companies and open source packages there's like optuna is a great one ray tune is a great one um uh there's a company called sigops they got bought by intel and you know i would say like actually using all of these they're a little bit different but the truth is like they're all really good they all will actually save you a lot of time and trouble um but what we saw when we interviewed our customers is that actually most of our customers are using none of those tools right if they're doing anything that's kind of like a hand rolled optimization system we're kind of asking why and i think that there's like a real barrier to entry so what we tried to do is basically wrap these open source libraries and just make them really easy to use right so what you can do is you can basically like once you have you know a set of um runs that you did you can basically just create a suite and what this does is actually guess is a set of um configuration because another problem with hyper parameter search is there's quite a lot of hyper hyper parameters right of exactly how do you want the hyper parameter search to work so we tried to make this um safe for you so we use um bayesian optimization by default we use um gaussian processes with small amounts of data um with um as you get to like bigger sets of data we switch over to um tpe which is a strategy that kind of scales better we guess the parameters that you're looking at and we guess what the distributions are so again we try to make you know if you want if you really understand hyper premise version you want to like you know customize the the meta meta parameters here you can go nuts the thing we really want to have happen is just like i feel like everyone at some point should be switching over to beijing optimization because in almost every task it'll save you um compute and you'll find a better set of um of hyper parameters so the cool thing here is like you hit initialize sweep this works on any set of infrastructure because what you then do is you just run this wb agent command anywhere it can be inside a docker container it can be in a collab it can be um literally like in in any kind of infrastructure and that's just actually going to ask the server hey what set of parameters should i run next so this super simple system it kind of works anywhere and we try to make it so easy that like anyone um you know would would want to would want to do this i guess some other stuff you know beyond that that i can show you is you know the kind of next big request that we always get from people is hey can you version my um data sets and and and and models and this is actually um kind of a complicated problem we looked at a lot of the the stuff out there we ended up rolling our own system with kind of again like an eye towards simplicity so um so we we version your data set but we also keep track of like who uses it because that's like a big question so like so basically um you know here's like five versions of my training data and actually each one is used by a different um a different run here so i can go into the run and i can actually see exactly you know what the input artifacts works is typically a data set and the output artifacts here which is results um and a model right so i'm kind of housekeeping not just the data set which needs to be versioned in a smart way you know because you're often making incremental changes there we're also sort of simultaneously versioning the outputs and allowing you to attach like metadata to that and i'll show you just one more thing and then i'll be quiet here but but um you know like another huge issue that we saw um and i think utah was kind of talking about this a little bit but basically just even finding um images in a large data set or finding like examples of training in a large dataset it can be like a real challenge right so we've kind of built a system where you can actually see so here we're like seeing your data right we're seeing um you know the semantic segmentation and then summary statistics and you can filter it down into ways that are kind of like you know useful to you as a um as an ml practitioner so maybe i want to look at just um you know roads where like a certain fraction of the image was um was a bus right so like you know maybe places where at least one percent of the image was a bus right now i'm like seeing all those and i could say well maybe i want some percent to be a bust and i also want to find images where um there's there's some prediction of like a bicycle in there um also right and i guess that filters down to that case i guess wasn't in my data set maybe if i make it a little more open-minded so now um there's a prediction of a bus and a bicyclist um in here these expressions obviously can get really complicated you know because a lot of what you're doing as an ml engineer is just sort of exploring your data sets and trying to figure out um what was going on i can also kind of group this so look at like what the dominant prediction was so here um you know the are the data sets where my main prediction um was a was a truck right and here's one where my main prediction was pull i guess void means maybe there's not like an obvious main prediction but they can also see okay for the data sets across all the differences where you know there's like a lot of truck in them you know what was the propensity to predict building and wall and fence and so on again all this kind of in support of like a fast understanding of what the training data is because that's such an important um constituent to how well the um how well the the model is going to work and you know i'm i'm over time now so maybe i'll i'll um skip some of this stuff but i did want to leave you know anyone who is interested in ml an engineer but not um you know ml engineer yeah i kind of wanted to offer like it's something that i really think is great i think when when when engineers you know get enough ml to be dangerous i think they they that's actually really really valuable um to to companies and also like you know can lead to a great career we see this all the time so i'm super passionate about engineers kind of retraining in ml and i think the best resources are one is fast ai which is a great resource great community um and um and and totally free another is kaggle which is a place where you can actually kind of get practice doing melon you know the cto of open ai actually learns ml he said entirely from entering capital competition so there's no substitute for this kind of hands-on learning and they now have a lot of new programming around just tutorials on ml a super friendly community and finally we've been putting out content um based on popular demand um on uh kind of getting engineers into ml and so you can also follow like www.interim um and then um you know if this is interesting to you or you want to talk about more my email is lucas wmb.com um are there any questions i realize i went a little bit long there all right yeah i have a question lucas so when so you you showed me the data that uh searching within the data set that's super cool by the way but [Applause] how do you make that a generic thing so is that specific to image data how can i use the same thing to train say uh a model that that does natural language understanding or how do you how do you make that a genetic thing is it yeah no totally it's totally generic we actually so we um uh yeah so you know we often use image examples just because they're the most um beautiful examples you know yeah um but most people don't um use that with with images so basically we have kind of like a type system um that um so we can kind of like know if like there's sensible you know they can show you an image or a graph or a number or or whatnot and then um basically the um the the the math that i typed in um gets compiled down to a graphql query um to store everything in a um in a graph database um underneath the hood so we can kind of do that fast um querying and it's all kind of lazy right so like you know our goal is that you can like see um your data live so that you can kind of do that like that exploratory use case on a really large data set i could put um i can put in actually you know the best the best place i think um to learn more about this is probably just center dock so let's put that in the um you know but but um but yeah that's that's that's the idea that the two use cases really considering here it's like um input data and then also um evaluation data right so like you know when you run a lot of models um on the same data set or the same like set of data sets you know being able to see like which ones you know where are things working where they're not working yep that's really cool um yep other questions i think there was a question earlier about the slide so if lucas shared the slides but she graciously uh uh said yes uh we could just post them under the meet up for for all the folks afterwards hi joseph here uh i had a question uh so yeah uh like this is like i work for a banking client uh and i'm located in pittsburgh where the client is right now and one uh challenge uh or one project uh that we work on is like trying to see if any sensitive information about a particular customer um is like not leak is properly redacted uh in for example in a bankruptcy case for example or in a foreclosure process so there are i know there are ocr tools to do that but uh can i use like machine learning models like cnns or neural networks to identify those redacted information in those documents instead of like you know manually scraping through those long third 20 thirty pages start for pdfs and and redacted sort of means i'm just picturing this it means like there's like a is it like in the movies where there's like a big black block over here yeah yeah i mean i would imagine that's a really easy um vision task um so i mean yeah i think with a pretty small amount of data you could you could train a model to um to identify like redacted regions i mean it almost seems like you could almost do that yeah but i need to look for specific redacted information like for example account numbers or date of birth or like license um plate number for example i see i see so so it's maybe a kind of a combination of a vision and like a language processing task possibly yeah yeah i mean it's an interesting question you know it's i would say that i mean this kind of reminds me of um kind of structured data processing which i think is like one of the most um common real world tasks that we don't see as good of a set of like kind of open source libraries for for like you know handling structured text data i don't know if anyone um wants to post in the comments like anything that they're seeing that's um that's useful there i feel like that's a case where people tend to roll their own systems because there's kind of a dearth of um of stuff but i do think um you know you can view it as a vision task where you're sort of like looking for um these images or you could you could ocr and then view it as a language task and you know i think it's hard to know a priori it kind of depends on how accurate your your um ocr system is um like how well that's likely to work uh but you could try both approaches uh i have another question relating to the uh the data set uh visualization stuff that you showed me uh so what what debugging uh what tools for debugging do you offer and did i miss it or is there a way for me let's say i identify uh my model is performing poorly because of a bad label is there a way for me to identify during using some debugging tool to figure out which label was uh was was the root cause and and which data set which version of my data set was it present in that's a great question so i wouldn't i would say right now there's not an explicit um debugging tool but i do think there's a couple best practices that people we've we've seen a lot of our customers use our table um to find mislabeled results so we we might um what i would recommend actually is is basically if you look at your um first of all simple thing if you look at your training data so your model running on your training data and you look at where the highest residuals are the highest error those are often mislabeled it's a great like place to start right is to look at the the training data that your model has trouble classifying even after it's trained right so just like empirically that's like a really simple query that you can do with our tool or any tool um and and find miss labels and then we actually have um we have a plot that i didn't show you where you can actually you can look at across all of your models the error on your training data and also the variance of the performance and the training data and so basically things with low variance and high error rate those are most commonly the um the mislabels so case where your model's just like always getting it wrong it's either a mislabel or there's something like funny about it right so like you know you'll often like you'll find them like you know imagenet there's a bunch of like completely black images that get labeled i think with tardigrades so you end up like black and you'll it's the easy way to find funny stuff is like the low variance consistently high error um uh things which which is like you know and also relabeling those is the most important because those are actually affecting your model the most right because you know typically the models are optimizing for loss which is typically something like the squared error or um categorical cross entropy even worse right where like you know the models are emphasizing the um they're really hard to classify examples um so that's definitely a place to start the other place to look is the um you know ones where there's high variance and a high error rate um tend to be just like difficult ones or maybe less um consistently like errors or miss labels um but also um worth worth um looking at because those tend to be like the really hard example so you know once you've trained multiple models you can kind of get the that's the best lens for finding mislabeled examples got it thank you um these are great questions i really appreciate them i i'm sorry i wish i had done a more i'm technical talk with this audience i uh i'm sorry i misjudged you guys um i'll do i'll get more in in the weeds and and scale by the bay what any other questions going once going twice all right so there is no more questions let's all thank lukas really really you know really great group today and thanks for all the questions and i think you know the the topic is right on for this audience because first of all you know uh we have a large uh audience on functional tv which will see the video of this right and so that i think the core of the community uh more into data engineering i think software engineers move into data engineering you know eventually becoming kind of all kind of encompassing engineering right so i think that's really the really hits of the nose but i i would just say you know producibility uh kind of tuning debugging right these are all kind of engineering questions and i think you really hit it like all the questions how do you deploy it right this kind of a perennial question and a lot of people like there is a whole like a system of startups around this so i think it's really helpful uh uh to to kind of see i just uh lewis just shared a link to um fully connected uh on your site for nlp and that's i i just went it's really amazing updating uh set of news and topics so that's you know i think it's really helpful so thanks again lucas and we are looking forward to seeing you in the community and let's get by the bay so thanks guys thanks lexi always great to talk with you with your group have a good thank you thank you