SF Scala: Manasi Vartak Interview

SF Scala: Manasi Vartak Interview

Recording: SF Scala: Manasi Vartak Interview

[Music] hello everybody my name is Alexa crabber from the founder and organizer of the era AI and we have a joint meet up tonight at D 2aq a well-known company basically doing data orchestration and I think it's very fitting we have a jury made up of multiple meetups and the topic of this meetup is machine learning models and specifically versioning and monitoring we have two talks and the monitoring is with hundreds upon pushkaryova and the versioning is with mono severe talk who is the CEO of Verta III recent startup so welcome mama C thank you thank you for having me tell us a little bit about yourself why did you decide to start Verta and what this is all about absolutely so I am a technical founder I did my PhD at MIT worked on data systems all throughout grad school so one of the systems I built was modeled TB we started looking at how data scientists work and what were challenges and this was around the time one Alex net and all of the deep learning models were really becoming significant I realized that the things that we take granted for software I like get type versioning systems deployment debugging were all missing for models and so we ended up building model DB which became a popular open source project and then further evolved from that nice and so I'm curious so you were a researcher to me right was very soft regime before that how the specific topic came up yeah so I did I worked at Microsoft between undergrad and grad school um my PhD was also in the database group from which came post grass and verdict also like we built systems and so it just we write software all day it was a clear and natural direction nice but obviously machine learning model also very different beasts do you have specific issues in machine learning how did this come about yeah absolutely so my interests are between data systems nml my under with CS and math so I also write algorithms and things like that I used to be part of the feed ranking team at Twitter also worked on dynamic ad targeting at Google so that was applied ml and that gave a good balance to the systems building and advert up my CTO actually comes from Twitter and NVIDIA we both work together Twitter is like it also brings a lot of significant chops in the production of asian of models very cool very cool so the you can have brilliant this two worlds right song I'm know I'm curious right because you're clearly in the camp of software engineers which most of our meetup members are right so we kind of so the the theory which kind of most I see working in in the area if you have working company you know machine learning is a commodity you take it off itself you're not you know working Geoff Kimpton next deploring iteration you are a grunt deploying models making sure that make money or don't lose money and and your job depends on them being you know in production right and so like versioning is beautiful it sounds like music to our ears because if something bad happens you need to find out and so can we understand get and and continuous integration like so but but a lot of data science development is done by folks who do Python and do it in a notebook they're not in production so how do you address those guys like how do you interact with engineers versus the designers versus business owners right I will see all these metrics in the hundred person um so for us the fundamentals of our product don't change for us a model version consists of as I will talk about the code the data configured environment together now if I'm a data scientist I am in a Jupiter notebook or our studio and the way that they access our product is via very easy to use api's or gooeys mm-hmm now if you're a software engineer and or you're a DevOps person then you care about your SaaS throughput how's your model behaving then you can access it directly through kubernetes sort of coop control use the yeah no files integrated into Jenkins do the software route and then if you're a business owner then you're only looking at our part of the product so there's the same fundamental very similar to say github if you only look at the UI that maybe a business owner or a project manager can understand it versus if you're a software developer you have the command line name you're doing merges and pulls all the time so here's kind of an EVE scenario right so I'm a business owner and I run an online shop right now my data scientists are kind of doing the upselling customers on shopping carts right or they're basically like I'm watching it they're coming back and so you know the the model basically does it and so I have a metric how much do they upsell each checkout and and and so so can you kind of can we is the field this is necessary reach the point where I am the shop owner I have no idea what that assigns as the programmers are doing I wake up in the middle of the night I wish how much every checkout is upsetting and suddenly because the previous night some somebody messed up and the models really bad and like I'm something really plunged can I push a button to rollback like is it at the level where you can give this tool to a business owner who which is a clear business metric and he will be alerted or is it the job of an engineer I mean are we gonna involve somebody who understands the business metric should it be a pet religion level thing for an engineer or can you let like reach the point where a business owner will be able to to say no this doesn't work because you know he cares he wakes up and he does a rollback is it possible I would say that in this sense our models are not different from existing software if I'm a product owner and say it's my mobile app and I see something wrong with it I don't go to our dashboard and roll it back I hurt my engineer to do it yes so I don't think ml models for any different we would escalate it via page or duty they can provide input into why they think a metric is off but we don't think they're the right people to roll back a production system okay all right cool and so I'm kind of very curious about this specialization right so not only we get a bunch of em elopes startups dealing with other BOTS now have you know monitoring versioning and tuning as specialties of different companies so do you think and obviously if you if we go this route and standardization when you can really agree what these models are yeah because right like if we really want be like do you think it's going that way that you know we have or like language used to describe it and models are we gonna go this route of kind of thousand flowers or isn't gonna be polite platforms which their own clothes tools because like for if the a standardization and open market them it's really good for star laws if it's all kind of inside things and then say sure Amazon and Google like what do you think this is going I'm going to see my specialization or not I would say we're going towards standardization and we have been involved in some of these efforts through model DB through verda where we're trying to find what does a model need to have with it what kind of descriptors doesn't need to have how does a model package get defined and things like that so I don't think just as in software we now have reached a conclusion on what a Jenkins file should look like so we are going to approach that stage with models we're just getting there slowly and the startups I think are providing a lot of impetus for that there's gonna be a bunch of llamó [Laughter] so we can check in yeah that would be wonderful very wonderful yes yes but I think Corinne that he says doesn't really got a good job in popularizing yamo as well as having api's and we've pulled some of those designs and sort of ideas from there we seem to be working well so what is the hardest challenge for you and you know left onto the next year were like what's the your current you know what keeps you up at night what do you want to solve with this kind of coming year for Virgil when you're a small start-up there's lots of challenges as you might imagine this move this field moves very fast as you might imagine and so there's always either a new algorithm you'd have of doing things or there are new companies out there so for us it's always finding what is the thing that we can do especially well compared to other companies and at the end of the day for us are we making a difference for the customers that we work for if we're moving the needle for them then that's a huge one for us and that's what we'll focus on sounds awesome we wish you all the luck and they're looking forward to my talk thank you very much thank you [Music]