sfspark.org: Peng Ye, Building Machine Learning Pipeline Using Aerosolve

sfspark.org: Peng Ye, Building Machine Learning Pipeline Using Aerosolve

Recording: sfspark.org: Peng Ye, Building Machine Learning Pipeline Using Aerosolve

hello everyone this is Tom from Airbnb thank you all for coming thank you a census mo for inviting me here to the talk it's a great primer to give a talk here so today I'm going to talk about how to build up machine a pipeline using airsoft so Arizona is open source a machine in the library there was a developer to answer pricing and availability at Airbnb so aerosol was started to support the pricing modeling tasks at Airbnb so therefore before we dive into aerosol I want to spend some time talking about Airbnb pricing so our team is called the pricing and availability also known as a pint a team so the primary goal for our team is to build the tools to support our hosts to help them to site and be triple price so they so that they can be more successful in the competitive marketplace so let's take a look at this demo so if your host on Airbnb so this is a calendar view you can say on the manage listing page so basically instead of studying of flight rate for all the nights you can sell customers the daily price so by smartly set in your daily price you may be able to gain more bookings and candle money hotels usually have some professionals who help them to set the price but most of our RDP hoster don't have don't really have the resource to do so in a very organized way so that's why they need our help so the demo here is showing you how the price tips work so as you can see we provide a price suggestion for every available nice the host calendar and then the host can choose to adopt our price tip for one night or multiple nights so this is great so this is actually helping the host of were using on price tips a lot but the problem here is that it's very difficult for the ultimate updating their calendar to make sure their price tips are always updated using our latest priced hip so to fix this problem so we recently launched something called Smart pricing so Smart pricing basically makes the whole like selling price process much easier so basically the host only need to set mean price max price and hosting frequency and then once they turn on the Smart pricing button their whole calendar will be controlled by our algorithm so we will always keep their price tape updated using our lady surprise tip and also the tip will always be found in the Pisan min and max set by the host so basically a soos may be testing we found this is helping our host allows there are actually a khadeem own eyes and earn more money so the problem here is how to make sure our price tips are ultimate for our hosts so this is a main task of the machining engineers on the panel team are working on so so I was talk a little bit about the pricing model so the very fundamental problem that we need to solve for pricing modeling as demand prediction so in particular we want to protect the booking probability for releasing night so we do this in two steps first we build a binary class classification model to predict whether a listing night will be booked or not and then we as Windows calibration so we estimated a curated booking property based on the binary classifier output so the model we use the four buildings a binary classifier is generalized additive model so the model is defined in this formula here is this X 1 to K are basically features so and each picture corresponds to a function so their function Maps the feature to its weight and the model output is nothing but the sum of our feature weights so yeah and then in airsoft the generalized additive model can support two types of functions spline which is a piecewise linear function and then also linear so feel free to interrupt if you have any question so even though that I will go in Sao Paulo piecewise linear so it's actually an arbitrary degree kiss my sleeves they're actually comodo highly nonlinear shape each interval will actually linear but because we have a lot of pins so it is actually highly nonlinear so here is some interesting features we learn from the model so this is the day of year feature for different markets in this case we are looking at the city and Beijing so as you can see from the first part so feature wait for the summer time in Sydney is very low and well for patron is very high so this means our model actually captures the per market seasonality of the driver friend so next it is a global model but this feature is a market across a price Adele gear feature so next I'll talk more details about airsoft so first as always from machine learning just to have a quick overview of the machine learning problems so basically you can consider machine learning model is a function and then have some input some output and there are basically four K components clear so first is representation you need to choose what has a function you want to use for a model like even have linear function decision trees or in the pricing model case we are using generalized additive model and then you need to define some evaluation metric so for different problems the metrics are often different for the pricing binary translation model where you are using a UC for evaluation and then you need to have some optimizer that can help you to learn the specific parameters in the function so this is often done by minimizing some loss function or training data and finally one additional or optional a very important component is Fisher engineering so if you are using like like dick Nunez type of model not need a feature engineering them much but for us in Arizona really focuses on very simple interpretive model their full feature engineering is very important for problem so next I will show you how these different components are implemented in Arizona ok first I want to acknowledge Hector so Heather is a my architect of Arizona but unfortunately he left Airbnb a few months ago so now he is back to Google so currently our song intensifies remaining email engineers on the Panda team so thank you Hector so and so basically have two parts so the course is implemented in Java between and training is in Scala in Korowai defines models transforms and some basic data structures so in models way really focuses on interpretable and human typical model which are your a very simple model like linear model and the generalized additive model I introduced earlier and it is also worth noting that the scoring function in Arizona thought is actually also implemented inside the model in Java so this extra mix is very easy to use a model for an for all time scoring so the second part is transforms so this is for feature engineering so every so provided a very powerful feature transformation language for for the users so users it allows the user to to very sophisticated a feature transformation without actually writing any code they just need to write the config files and also there are data structures those are all defined by okay so the second part is training so in training we implement a trainer evaluation and the something called generic pipeline so currently aerosol we are only supporting supervised learning and our training complements our spark therefore it can allows you to clean your model and a huge amount of data in a distributed way and for pipeline so generally it's just a tablet our set of tasks so with genomic pipeline so our data scientists time train evaluate and the popular model without writing any Java Scala code they just assume a config file that's the idea so any question okay so next I want to talk about why we are focusing on interoperable and human development model in herself so first when we started the pricing group operating group is actually pretty new so I told her maybe in early in March 2015 so that gene was about maybe two months old at that time so we don't really have a existing training day at all so the data is new not fully defined therefore we'll leave some - for how much to understand the data better and seventh all features are mostly sparse and interpretable features so by interminable I mean those features actually has have its meaning like the number of rooms location multi-country or price so every feature have its own meaning it's not like in computer vision we may have some image pixel as a feature so the pixel and stop by the solve does not really make any sense it's not so we that have a feature has no incredible feature and also a typical model can help us to depart our data because we also have some data quality problem and also this can also help has to discover something like limo leaking so yeah we had some level even problem when first the Train the model and finally with this hyper model we can discover some relationships between different variables and our target prediction this can help us to get some business insight alike for example we can say how different types of amenities like parking Wi-Fi or reviews are affecting the booking probability for listing demo so this time I will show you how we're doing feature tip parking okay okay so this is a my internal tool we built for our cell model departing so I'm loading actually a model and recently trained it's a very it's a toy model it's not an actual pricing model but there are some simple features I can show you here so as you can see you can see it's the most important feature here is review so the more review you have the more likely your listing will be booked and then there is a dailyrx feature so this is now crossed by market therefore it is only reflecting the current it's as you can see here like overall arrogant be summer is a high season and since there are the price feature where the lower the price the more likely you will be booked so yeah you can also solve the future by like playing with y one thing so in this way because we also have hundreds of no vision but like having a tool like this it's easy for us to debug feature to identify the most important features so this is let me show you another feature so this is awesome cross feature okay so from here is do not find something very interesting clear for so okay let me first assault by the weight so so Y axis is weight the x-axis is a feature value so this is us plan so this is how the future value is mapped to the vision weight so after I do the sorting so I can identify the most important features here so as you can see here a private room Crowl surprise feature it looks like this feature is more important than the entire room cross price feature so this means for every Modesto there are more price sensitive when they are trying to book a private room the like they're less sensitive there are less sensitive to price and when they are trying to book an and her face okay so this is the tool we're using informal debugging okay so next I'll go back to the talk so why do you want to use yourself if you have a problem that is similar to our pricing problem like you want some you want her interpreter role model and you have sparse and interpret well feature you may want to try a result so a second motivation for you maybe so if you have some modeling tasks that is really like feature engineering Haley you may want to consider use the visual transformation errors or because it provided you a way to control the features by writing some simple config files so sometimes maybe you don't want to lose the modern assault by you if you can still use a feature transformation part in aerosol to help you to do feature engineering and yeah because the transformation are defined by a config file and implemented in Java so this makes sure the consistency between training and scoring so another motivation might be distributed training on spark with this you can trim on a huge amount of data so another advantage of aerosol is it's very easy to use it for real-time scoring so because the Union stays model scoring and feature transformation code are we in Java so there's no spark dependency and basically the model freedom sparta can be directly loaded in a java service for the real time scoring and so it's really good at like the thing we're trying to optimize for basically but it's not designed as a generic solution for all machine learning problems here are some cases where you may not want to use airsoft for example you have a really small training data you can fit all your data in single machine and you don't care about real-time scoring so in this case I'm feeling you can you may be able to iterate faster by just using like secular no art and secondly if you have you're having you're doing like limited classification or speech recognition type of problem so we may not want to use aerosol because your features are actually not human interpret more feature I think in that case you should just try out like the cutting edge deep learning thing yeah so next I will introduces the underlying feature representation aerosol this is an ingredient that is making aerosol really flexible for feature engineering so feature vector is basically crystalline to one clean example so if you are familiar with other version elaborate it is just the feature vector it is just a one row in your training example so there are three types of features those are all defined by math so the hero's math is featured family name so Fisher family is like a very unique concept in yourself it is basically a set of features with the same type so organizing relational features into the same family makes it really easy to my operations on a group of teachers so yeah so it's worth noting you should output features with different types in the same family so let's take a look at an example of a feature vector that is representing our pp listing so for the string features we have to feature families location and listing so location maps to a set of strings like San Francisco say USA and lip-sync match to a set of strings that are representing of listing properties and in float features we have three family the first is the location so this max to another map so the new key of the internal map is the Fisher name and you disc in the occasion face were looking at two features latitude and longitude you may know histories of feature that we colored label so this is actually the label for example so in this case I'm putting this is the positive sample so you should not never use the label as a feminine for other features and for dense feature like you can imagine that so we may have some image equality feature to representing to like to describe as a listing quality so for example and the you know dense feature the Fisher family well back to a little double so it's a dense feature in this case we have image of a surveyor and RGB feature and the questions okay so that's feature vectors so additionally also provide a way to organize different feature vectors into an example so an example consists of a list of feature vector and a context the feature vector so the contacts the visual vector is some are features that are in common over the whole group so for example you can maybe put all the feature vectors representing these things from the same air DB host into one example and you can put all the features that are in common for this listing like the host feature into the context so there will be shared among this group and of course if you want to treat each twenty sample independently you can just put one feature vector in this example so that's a feature representation part the next awell describes feature transformation so Arizona transformations are always applying operating on feature families so it can be applied one pigeon family or to different future families for the first time acceleration imagine that you may want to apply some element wise function for Google features for example you want to do some memorization for some feature in one group and then you may you can output another feature to a new feature family or you can choose to overwrite the original feature so in the second case we have applied transform to two feet on families this will generate across features from two families to capture some feature interaction so here are some examples so in the first example let's take a look at a string cross flow picture so this allows interactions between string literals and the flow features in the pricing model we have as we have a feature which is a market cross price feature so that feature is generated by this type of feature transformation so the right hacks sphere are the config file config that you need to write to apply this transformation so basically you define the transform name which is string cross flow first and then clear the field one is a feature family name for the family that consists of the market feature and then this flow field who specifies the feet the floating feature family that consists of the price feature and then the the crowds the feature is output to a new feature family called and cross F so and this is how we are like a modeling so our market price information the second example is Monas co-creative quantized so this is used to construct the multiple 90 degrees for 2d coordinates this is very useful for modeling location so we also use this in our pricing model so in this case the transform name is monika great quantized and this transform applies on the feature family called allow States basically location and the two features were operating on our latitude and longitude those are specified in value 1 and beta 2 and the parkings defines our parking size we want to use to do the quantization and then those quantize the features are out to output to a new feature family called a few IOC ok so once you define those in color is very easy to apply those transform cool example so basically you just in two lines of code so the first line defines the transformer so as you can see the transformer acronym is loading the config file and in the second line of you like my combined contacts an item to your example example is example introduced in context the transform I'd help transform and combine the transform so for this is a example in aerosol here we have contacts and some independent item so you know transform in the transformation we will first apply the contacts the transform and then we apply item transform and then internally also work on phasor transform the contacts to each transform item and then we apply combined transform to come to final transform the features okay that's the transformation next I will talk about briefly about model so all the aerosol models are extending the abstract model so basically in the model we define model weeds so mono is basically are like it's a parameter C on model it array on that so if you have like a forest model me have trees and here we also define the scoring function so for different purpose we have different sculpture score items go hiking for multi-class and score ability we also provided some departing utility functions in this model it will help you to do Model D parking and of course we have loading and saving model wave functions so we looked at airsoft there are three many inside those are the four models that are used in production and Arab in being each other some are serving different purpose we support regression federal education and what I class classification so in particular I want to say this these are the two mostly used models inside our B&B so first the end model I will say this is a most venerable model things it has been used extensively by the pricing team but it does not support multi-class classification and also it requires a lot of feature engineering to achieve good performance so another model in forest model forest model is not really a very human interpret Milano so the reason we are adding this model is basically there are some other teams at our VP are requesting these features of Isis so the advantage is that it does not require heavy feature engineering but the problem is so the courage finger implementation is not so good so it's actually extremely slow to use this model also if you have the patience to wait you can also give your medical model components so finally I will talk about pipeline so this is a typical mission in Python and there may be so like the first step is like we know data from hype or some sense we foul so for the pricing problem where you know we're seeing are all features in half and then we converted features to example and then we apply transforms defined by the user in the config file kind transform the features and then we train the model and then we do model evaluation the parking will repeat these steps for several times so maybe I'm some point we can shave or visible a good model and how do I use this model there are every two different ways so you can use this model offline you can just a run your models calling out line on our cluster and then you upload your model output to some other storage like we often use elastic search and each pal and then the scoring stories I'm just accrual rate of like a mass storage to test your model scholar so another way is like using the model for real-time scoring so one way to is we upload the model to s3 and there will be noted by the scoring stories to do real-time scoring so so why do we wanted we know time scoring so for the icing phase actually most of our modeling a scoring job are running outline but there are some situations that we want to UM as for me for example when we have new listings we don't have any previous information about this listing the listing does not the information does not even exist in the higher table we have so that were why we want to do uh not only in this situation in aerosol we have something called generic pipeline so basically it defines a set of common tasks for building machinery in pipeline so it allows the user to train you and departmental model by just writing config files and this is a link to the high pass so you can see detailed instructions on how to like build our pipeline using generic pipeline in this head pad okay so finally I was some time talk about the current status of our aerosol at Airbnb so for the development it was mostly supported by Imagineers on the Panda team it is actually under very active development but because we're not like like machine learning infrastructure infrastructure team we're a product team so most of our development our agreement by the pricing model in need so this may be very different comparison so like the development of other and our library and also although this Court have started at pricing it was actually used by many different insight and we'd be like booking teams or race teams or I product accidents put business travel so it was so it was used to it it was quite popular this is mostly because we have a common scoring service to support Arizona model so what does this mean it means but then a scientist can't deploy their model very easily without writing any java code so previously dinner scientists also often have to like fund some engineers who helped them to deploy the model but with this like Commerce foreign service so it makes their life really easy okay so that's all here are other main contributors for airsoft so and yeah thank you everyone for coming to the talk that's all from me so we'll have any questions now this actually implement in in Shelley is a shiny dashboard our yeah our shiny - repeat the question for video the question is at the time of the - I showed in the demo open sauce run so the final answer is is now there's a internal - and it is like implemented in using art as you said it's a shiny dashboard we don't do that we have only one global model but a model pixel on features like hundreds of thousands of feature yeah the question is will have had great overdramatic soup an Ori feature so the answer is worth in something like mathematically equivalent but we don't do that as a introduced earlier were operating model all the features into a map so the categorical feature would be responding to string features so it will be a set of strings internally and so so but mathematically it is the same answer like one hot encoding yeah the question is with her what are the price I try on so trip the price is not a label clear what the model is doing is to predict the weather eliciting well people are not so price is a feature in the model the price will be whatever the price and the horse to have standard for that particular night so the question is do we have any metrics to show that whole model is actually helping the host to get more bookings say also a CMS because we do a bit hasty so we know that we're by using this the wholesome artery I have no bookings yeah but but I'm sure that our model can be improved further so it's better than the price tied by the host yeah so the question is do I never try to predict the best price the question is yes or no because no because we don't know the best price right so we don't know the optimal price we know what I the host to have the list in our box but that does not means as a booking price is equivalent to optimal price so we don't but yes yeah so the question is can we refer to the reverse engineering to up whose optimal price equivalent booking probability so the thing is how do you define the booking probability so you're a if one like a hundred percent booking so that means your price is already just a zero so the goal is to optimize not just for booking for parity but also for the revenue yeah yeah we do something like this this place for max revenue strategy but we're not doing exactly the same thus testing but we're trying to balance like you have few things Oh yeah so the question is if this is a pride you've always Northside our hosts are using this this is helping them to get more booking but all the hosts are starting to use this how do we make sure they're getting booked here the question is the answer is we are actually not only competing in center pimpy right so we're also competing like with other alternatives so but you bring up a very good point so there I trick we observe some cannibalization in fact that's why I say the relative cultural like measurable impact so say we seem to 2% again um nice so you reality may not mean that huge because of the cannibalization in fact the bookings that you're Ragosa to the the controls goes into the feminine boost so that will exaggerate and the thing we have yeah but we don't have a really good solution for how to measure the cannibalization effect so the question is that do I find so host of our using a tool time to lower myself increase our price right so the question is lowering the price yeah so yeah so this is where so oftentimes hosts are thinking our prices are too low but in reality from our AP testing result will they lower the price they can do nice and the same time they also can move revenue because padorin is and the price a little bit they can move okay so overall the revenue I love yes yeah that's a good question the answer is we also have demand they like supply information and features so we know the demand we also know the supplier yes the two types of features are out there so you give customers any insights into the future weights so if I have a listing you know where should I invest that get my price to increase interesting where I first think about this thing but we're not doing this so I think it's a possible like action item for you so for example we discovered something very interesting from a model basically we found these things with a red pillow often have a higher looking for details because of like it's it is having higher people that is more attractive I guess I don't know so that is some mysterious similar name oh yeah I was gonna say I do use the tool by the way oh thank you and I will buy a red pillow every floor about how you use spark to information water is there going to be what are you - how do we so basically so so I learned how a little bit on the Alamodome so basically I outed her are no living from high sorry R DT right so hands-on way to meditation and those are data back to different petitions and then we train different are different models different foundation and then we do some aggregation so that's how the item always trained it's like a packing algorithm so actually we have an image quality model that is an additional model so that model output is the input term like pricing model so then although actually captures an information because we found that into quality features have heavy like positive correlation with the booking probability and the red pillow also like they say something we just identified so we're always adding new features I think where when yeah we may be adding these type of text features later so we actually our model developed cycles pretty fast so we iterate very faster so every time we add some new features or I have categorical features of each other hundreds and hundreds and hundreds of categories right and I mean what is your intuition how it should be okay because we also have this about how this type of feature like market you imagine there are thousands of market in the world and we also do like the multi secure colonization will quantize the latitudes allow me to filter into different market all those are different are categorical feature so the model can automatically capture which feature are which features are important so the model training and free will job some less important feature like by themself yeah that's it we don't really yeah we I said we can't deal with I saying because we we know how many least listen how many bills and this thing is getting so for the event and like that and so the listing bills not really also spiking did you see the same challenges so wait wait don't they say that have a heart for forest not okay I'm serious wait we don't want forest model for pricing problem yeah sort of one way I have my service and we don't really have a feature engineering I have not always often performance the past we actually try this all night XG boost and I put stuff to like improve model but it has all the original visor won't always like and these are all so far yeah so that's the first question of question is do it actually uses a natural light coming spine the answer is we is a yes because sometimes we founders of splines actually looking like a cubic spline we actually do some smoothing to make it actually matching the cubic spline so smooth thing is the potholes or you know the training is doing that so the question is like yeah yeah so I think when we started this project that one does not exist because spark analysis based beta frames then you can only do that it's very hard to take that for presentation then applied in real time okay so no questions thank you