Bay Area AI: Varant Zanoyan, Zipline at Airbnb

Bay Area AI: Varant Zanoyan, Zipline at Airbnb

Recording: Bay Area AI: Varant Zanoyan, Zipline at Airbnb

[Music] okay yeah as a Knick a very appropriately introduced time around I am here today to talk about zip line a little bit more about me I am a software engineer on the machine learning and for structure team before that I was kind of an engineer on the data infrastructure team and a data engineer at Airbnb and during that time I like worked with a bunch of different teams to help like build and deploy and launch their their models so a lot of the ideas that we had for ML infra came directly out of kind of the pain points that we had and the things that we saw kind of go wrong during that process before I move on a fun fact about me when I was really young I got very sick on a flight from London to DC and made the plane land early in Canada so that's why it's my fun fact on to business so my agenda is to start with just a bit of context I'm actually glad that I went third because I you know I don't have a definition of feature store so I'm glad that the first two talks did I'll pretend like that was intentional that I knew I was going third so but the context I'm going to set it just kind of around like when a person would want to use zip line what sorts of projects would it help with like kind of roughly where it fits into the whole workflow and then more specifically like exactly what the painful things are that you know zip line addresses and then kind of like how we thought about the solution then if there's some time at the end and I might just skip that part we can go a bit deeper on some technicals so context I think one question or what I've given talks about ziplining before sometimes it clicks and sometimes it doesn't and I find that often people have an idea of like an m/l project or an example ml projects in their head and depending on what idea that is rather kind of like what three categories that falls into like this either clicks more or it clicks less so this is a little bit experimental this is the first time I've kind of presented this part of it but you know the first thing that you might be trying to predict is like something like image classification you have this unstructured entity and basically all the features all the information about is is like kind of encapsulated in that entity right so like if you have an image and you running a cat or a dog model you know your data is there like it with every every observation every example whatever you want to call it the features are they're kind of all packaged up I don't know if I'm structure identity is actually the best title for that but then you know the other thing that you have is another sort of entity prediction situation where for Airbnb we might be say you know predicting a listing or classifying a listing as whether it's good for families or not and the important distinction here is that we just have these entities and we're just kind of classifying them or making some predictions based off of them and the important thing is that there's no inherent time component there's nothing temporal about this they're just listings or they're just people or they're just users and we're just kind of running some prediction based off of them this might be better more clear once I contrast it with a third with a third case which is events right now we have something like network traffic and it may be you're a cyber security provider and your model says like given these network you know your model is based off of network traffic and classifying them as a malicious or not now we have a situation where every single one of these observations or examples happened in the real world it was an event and it had a timestamp and this kind of gets back to the conversation we were having just before in the last Q&A or the gentleman in back that's a good question about backfilling and we had a bit of back and forth about how important it is this is the case where it gets very tricky because now we have these events that happened and it's very important to whatever feature values we assign to those observations for our machine learning whatever training data we make is as of that timestamp and you know the next question and again I think this was talked about just a minute ago is does your model run in batch only or does it serve traffic in real-time right this is kind of the other case which is relevant to like whether zip line is useful for you or not or whether the service have provides these back fills of feature data is useful for you or not and so kind of like breaking it down in that graph basically as we move from left to right for my first one of like kind of image classification to my last example of network activity zipline goes from like kind of maybe not very useful at all probably to extremely useful as it is very hard to do ml without something that solves these problems for you and then if you take my last question on top of this which is basically like are you trying to launch your model in real time are you trying to serve traffic or are you just doing some batch prediction if the answer is yes we're trying to do online stuff you know that goes from a lot to like a lot a lot and the middle one might go from somewhat to like very much so so yeah I just say this is kind of like set the scene that like most of the things that we talk about are most of the problems and I'm going to talk about for hipline are you know in this third column our fourth column rather it's the events driven ml and on top of that it's when we care about online serving of this model and yeah I think yeah I think that this is a pretty common confusion when talking about like backfilling of features and so on and so forth and and yeah and then actually that discussion that Nick and the audience had was pretty good about logging right a lot of companies they just logged their features that way we kind of just like you know guarantee and that this is what we saw when we would have at this point in time because we logged it at that point in time but it brings up the issue of like backfilling new features okay so motivation more specifically what did we see at Airbnb and you know this was a very very common sort of sentiment amongst data scientists and we were talking about this just before and I forgot who it was but somebody said that 60% that's not bad we've seen that member go to 80 or 90 so and that's that's certainly true so like you might have seen this comic about a bunch of the other scientists slacking off while their model trains but for us it was more like you know like it's like my back goes running like I messed up my training data I had to rerun it it's like taking me forever this was one big symptom of a lack of infrastructure that we wanted to address with zip line so let's look at why it's hard right like why is feature engineering so hard for ml right like you know data processing an ETL frameworks are generally pretty good why is it lacking for machine learning so let's say let's say it like for example we're training a model that's when a user views an experience I do so let me set some more context I guess at Airbnb you know you can book listings places to stay on your travel you can also book things to do which are experiences so in this case what we say is like when a user you know views this experience so immediately we're onto the case of events right this is a model that runs upon page view we want to make some prediction as in we want to guess that this user has a high intent to book because we might you know show some buttons a bigger something smaller or whatever purely hypothetical example but let's say that's what we're doing and so we might have an example feature here the sum of prior booking is in the past 7 days so now you know we have this timeline right and the prediction time is deter is dictated by the product right the product says I need to make predictions when a user views of listing so this is the timestamp as of which you know my prediction happens the label comes in later it might come in hours later I might come in you know a weeks later in this case you have probably come in relatively soon if it's part of the same session but it comes in in the future then we have a bunch of feature values so feature one in this case might be the example that I gave a minute ago it might be the number of bookings that this user has made in the past seven days and these kind of change along a timeline so what you want and your training data is your label plus all of the future values as of that point in time exactly and if you have another row that other row has another point in time right so generating a feature value for a given point in time is pretty easy it's pretty expressible in sequel but when every single row every single observation has a different timestamp and you need these aggregations and these things express exactly as of those times times that's when it becomes a very difficult data engineering problem what a traditional data warehouse is good at doing for you is this basically daily sums right like right now with just simple hive SQL air flow you can set up some jobs I'd say some of the bookings by user every day and pretty write those daily partitions right but it give you those intraday values not not at least any technology that we've seen exist for traditional data warehousing or ETL so then you might ask you know like why is it so important does this really matter that much just take those day boundaries I'm like call it a day right like just to generate your training data and like you know be done with it and the answer is sometimes that might be okay but a lot of times you get into some risky situations here let's say you go forward right you take the end of day value it's very easy to leak a label in this case which means you the thing that you're trying to predict you accidentally slip that into your training data then when your model learns it learns that this feature is great you know I'll just like predict mostly based off of this feature and it has a very high evaluation accuracy but then when you go to production the issue is in production your data doesn't look that way your future values look different you're no longer you can't make the label in production because you can't you know go for it in time and then your model doesn't do as well in production let's say you go back right okay now we've kind of solved that label leakage problem we've gone to the prior days midnight and we're gonna use those future values well the problem here is that you've lost some really good features maybe like let's say you're doing your predicting something based off of user activity and you want to know the number of times they did something in the last five minutes right you're gonna lose all of that basically if you throw away intraday accuracy so this is why we cared a lot about intraday accuracy and this kind of gets back to like that Q&A that we were just having like it does this matter why does it matter these are the cases in which your matter is kind of it's the cases where you're trying to do events driven ml and you're trying to do online serving it makes it very important to get this right it's a complicated thing I don't know I've tried a lot of different ways of explaining this I hope I hope this kind of makes sense I guess we'll see when we get to Q&A if this cleared things up or not but that's that's a big part of the motivation was there the other kind of half of the motivation and this is another symptom we saw a lot is like you know and I kind of talked about this a bit my model perform as well or least I thought it did until I got to production and so this kind of gets to this second so the point in time correctness is one example of a broader thing which is online/offline consistency I the model the data that you give your model at production time for you know scoring and prints prediction whatever you want to call it has to look exactly the same as what you trained on otherwise you get these funny situations where it looks great here but it doesn't perform well when I need it to a common cause of this at Airbnb you as people basically training their models off of ETL off of stuff that's in the warehouse so you have all these like processed cleaned up good data tables that like data engineers and other teams have built and people use that for ammo like why not you're a data scientist you have access to it that's what you're used to accessing for data and so you use it right until you take them one user age and there's an integer and it's clean and it's been like like outliers have been removed and fake users have been removed right and you use that for for training but then when you go online you know you're just like oh like how do I get this user thing from like the hive warehouse and you know what people do is like oh well whatever there's a prod DB field that has like more or less the same field but there instead of being an int it's like some other you know timestamp thing and blegh it throws off your model I mean that's a pretty contrived and like silly example hopefully that exact thing wouldn't happen but the point is as soon as you have ETL in your warehouse it becomes hard and you don't have access to that warehouse and production scoring like how do you now guarantee feature consistency right what one solution is logging right you can't log anything that you don't have access to in the production logging environment so you're guaranteed to have those features but that brings in the issues that Nick was talking about a second ago where you're depriving your data scientists of this huge warehouse of feature of potential future sources and processing clean data so that's kind of like the main like those were the main symptoms that we saw of a lack of a zip line at Airbnb and like that's what we saw people struggling with was like those two things like hard to do my ETL and hard to debug my my model looking good but not doing well in production so that kind of gets down to like you know the like now we move on to like the solution so one way I like to describe zip line it's like it's the time machine that you stick on to your data warehouse so you know you stick zip line on top of your data warehouse and now you know these features that you are compute previously supplied can compute that for any given you know point in time it can take your sum of user bookings in the past seven days and it can move that forward it can move that back I can backfill your data with point in time accuracy the other thing that it does is you define your features once okay so sometimes I like to call this zip line travels through time and space so you know you define your features once and it can make them available in your data warehouse where you need access a training data but it can also provide you feature values in the online features store when it comes time to serve your model so yeah it's kind of like you define at once and it makes it available in the places you need it and it can kind of move move the value of the computation forward and back I'm going to go into a little bit more specifics here when I kind of show you what it looks like so this is kind of just like a mock-up of what we want our oh and I haven't talked a lot about sharing and discovery but one important piece of you know the future just the future repository is like you should be able to find features so that you can leverage other people's features I think this was talked about much better by the previous talks probably than mine but you should be able to kind of you know identify features and like do some visualizations and like understand at a glance what this feature is what it looks like does it fit into my model so you can kind of leverage other people's work in an organization but yeah so that's so like that's kind of it's a quick note on the sharing thing but now I'm going to kind of jump back into you know what it looks like to define a feature and kind of how we actually enable that time travel to work so here we have a simple feature definition the source of this feature is a hive query so what this means is that it runs in batch right there's no streaming component this is kind of like the simplest thing that a user can do so in this case you know they're selecting some field from a data warehouse table now these fields don't immediately become the feature but rather the raw data that feeds into the future computation so on top of defining the raw data you define the operation that you're going to run and the time windows which it runs so now the zipline knows how to get your raw data and it knows what aggregation you want to run and it knows your time windows the other key thing to note here is that it knows the primary key and the timestamp for each of these raw things like these raw event rows are going into your aggregation as the result of a zip line being able to identify these things independently I can now come to zip line and say hey for user one two three four timestamp yesterday at noon please give me the exact seven-day 30-day blah blah blah future values for these features so basically we just like gives a plan a little more information that we would a traditional ETL framework where you might just give it a query that it runs every day with a group by and accounts an account and a summon an average you know function UDF's because they're playing kind of can separate these things out and now has a bit more control over how it generates data that's kind of like the key concept here so this defines a feature like what happens next the answer is kind of well nothing until somebody comes along and says hey zip line I need these features for these timestamps for these primary keys so we're a little different here I guess and other than other frameworks that we've talked to because you know some people like I guess I got branch for example they you pre compute a lot of things which is something that we've thought about - I think this pros and cons to either side and I think you can make it work but we do we do relatively little pre computation of actual future values until users come along and kind of make this request so this is what the API looks like here the first thing it is is a list of features which you can imagine is a important part of your training set so here's the Python four you know saying I would like these features from zipline but then the second thing that we asked for is this query that we call the driver query this defines the primary keys and timestamps for which I want those features computed so another way of phrasing this is the user gives us this and zipline fills in that where that is some arbitrary set of features if that makes sense so the idea is like you know what zipline needs from you to make you your training is your primary keys or time stamps and the list of features so like you give it that and it fills in all those values and I guarantees you those features are correct as of those time stamps for those primary keys okay the other cool thing that we can do now that zipline knows these things it knows your features is that in production we have simplified the workflow of scoring a lot so let's say we have a scoring request know for some model and the model is a model I you know gave an example of prior is like the booking likelihood model so before if I was going to score the booking likelihood model which I've already uploaded to my prediction service which I'm not talking about today but we have other information on that if you want I would have had to get that whole feature vector myself and send that future vector to the prediction service this is where that you know opportunity for inconsistency comes in is during that feature fetching instead now all I do if I trade my model and zip line is I tell the prediction service here's my primary keys right this is the user this is the listing or this is the experience or whatever and then what the prediction service can do is talk to zip line because it knows your model is a zip line model and it talks a zip line and says hey here's my stuff here are my primary keys and here's my zip line source that was used to train my bucking at likelihood model zip line now makes your feature vector out of your primary keys and if we think of the features as still being time traveled the time here is always now so the online future stores never does it like a back in time thing but it's always as up-to-date as possible so it says hey here are your future values you know as of now and the prediction service is great now was indeed the time stamp thousand of which I'm scoring because I'm the prediction service and then it generates your probability and returns that back to the user so we've taken away the need to like kind of wrangle your online features are you wrangle your feature data at prediction time which was really part of the the main engineering overhead that we saw going into deploying a model so in summary zip lying is basically focused on time travel and consistency there's some data quality and monitoring stuff which I didn't talk about very much but we've have kind of built into our like front-end UI stuff and we're still kind of working on that searchable and shareable and integrated with the rest of the ML workflow and that's about it I don't know how I did on time but I think we might have some time for some questions yeah we got we had a few minutes yeah okay yeah yeah [Music] producer see all the users and how much they spent in the past seven days of experiences and the second question would be what happens when it just happens to be yeah I'm gonna can I answer in reverse order no we don't currently handle that but it's a very interesting idea and we've thought about it and we kind of know what we would need to do to extend to support that to like kind of pipeline features it's a pretty interesting idea we've seen we've seen some people have requested so I might think about it some more we just haven't prioritized it yet on to your first one I think maybe I explained things poorly but that group thing is I think I often say like a primary key and a timestamp but really it is kind of what you said like if you wanted all users you know and you wanted their some of Luckies in the past seven days you would just write your driver query to select all users the other thing that I might have over emphasized is one primary key you can multi key these things so like for your user for your device you might have some count of views and for your you know so if you wanted like mobile and laptop you could separate out those two but yeah it kind of is what you said like look at the group is the more standard thing I just kind of gave the row-by-row example yeah this questions this question is related to the previous one above backbone yes if you go back to the timeline slide yeah you weren't using an example of backfilling there you're just using that timeline slide to talk about consistency yeah so they're very related I guess so the reason why backfilling is hard is because it's hard to be consistent about backfilling I guess is my point so the point is when I backfill okay so if I was back filling this training set for this model and every every online prediction corresponding to a listing review writing my driver query in this case would be like selects user listing ts from listing views table right for example and then once I had that driver query once I had those primaries and time stamps computing those future values exactly as of those timestamps for each row that's what's hard exactly yeah yeah yes so we are trying to open source this you know people always ask when and I think it's pretty hard to estimate because you know if we could all just sit back and focus on open sourcing it we'd have it out pretty soon but unfortunately the company has demands of their employees you know outside of what those employees don't want to do but it's coming this year exactly when you know sooner rather than later I think - I think maybe like summer ish would be a reasonable estimate I just said on my camera so I guess I'm I'm gonna be held to that but yeah we are kind of open stories are pretty soon as far as like I guess maybe Nick would be a better person to answer that but I think that you took a lot of these concepts to your design a branch and maybe your implementation there were just more suited towards kind of what the demands were at the time but I think we talked a lot about our like compared and there are some differences but by and large they solve kind of the same problems I think zipline and gondola right so you know I think the answer is here Nick side is a good opportunity to troll me and via that's primarily why they yeah but no I mean I've been asking him probably once a week when they're in here in Star City I think that there are 20 pieces here that's all a lot of the problems that we had to solve thank you thanks [Applause]