scale.bythebay.io: Neville Li, Featran, Featran77 - Generic Feature Transformer for Data Pipelines
well thanks for coming to my talk this one will be our future and our future transformer and my name is Neville so let's get started so let me just introduce myself first what cats qualify in the New York office being there for 6 under more than 6 years and before that ID load bit machine learning on Yahoo at Yahoo search and boy if I mostly work with music recommendation data and machinery infrastructure so I work with the team that built discover weekly release radar some of the features that you might have cut off and I woke mostly with Scala and big data infrastructure libraries and tools we build our own Scala API for Google Cloud dataflow and Apache Ben coccio we used to run a lot of scouting jobs a little bit of sparking storm and basically the the Apache open-source in ecosystem so mandatory PR slide about Spotify we have a lot of users and we collect data from the users every track you stream so and we have a lot of data stored on or used to be our Hadoop cluster but we're also moving to Google Cloud so we actually have fair a fair amount of data petabytes and we have a lot of Engineers solving problems actually have at this point close to 300 developers writing scholar pipelines on a daily basis and we also have around 300 people in the ml slack channels those are the people interested or currently working with machine learning and for the ml stack we use mostly a shield for pipelining future engineering waste featurin and then we use tensorflow heavily for a lot of machine learning stack so that's a little bit about Spotify now let's get to the business about feature engineering so it's the process how many of you have done any email sort of work or taken across of course so basically part of all the first step of doing machine learning is to engineer features you basically take categorical numbers labels whatever and and massage them into format your favorite machine learning algorithm assets like vectors or just matrixes and things like that so it is the process of applying this domain knowledge to to convert your data into features that work with search and machine learning algorithm an algorithm might have requirements like the values need to be normally distributed or normalized to a certain in a certain way or form so if it is a very hard part of machinery it's actually a the most probably the most time-consuming part because you're basically writing a lot of data pipelines to transform data and it's pretty tedious so coming up with features also also hard it requires a lot of excellent knowledge both in the tools that you use like the data processing framework and also in the knowledge of machine learning and algorithms so yeah it is a very hard part and we are trying to solve it so let's just take three very basic feature engineering transformers example so I'm not just gonna switch to the console that's more interactive so let's say i'll just generate a sequence of double numbers let's say point five point zero swing point twenty two point one oh no fifty nine eight numbers very simple so the first transform we're going to look at is binarize ER so we have a sequence of 8 numbers that's continues feature and we want to binarize into a binary feature which is either 0 or 1 depending on where the number is above the threshold or not so let's say we pick up a search showed 42 how do you do that super simple right just map the numbers and then you have greater than 42 one point zero otherwise converted to zero point zero right it's simple suggest the mapping function so this is the first transformer that's super simple and has just input of double and an output of double and there's a mapping function between the double and double easy next one is slightly more complex called me max kala so I want to normalize this these numbers to between a mean and minimal minimal and maximum so in this case let's say between 0 & 1 how do I do that first I need to know that many of my maximum of my current dataset right so I can do something like this right this is easily doable but what if I have huge amount of data like billions of numbers and I'm using something like spark or RDD API or sculling type safe API probably don't want to do that because that will require like to reduce steps so I can do it in a single step right how do you to college in a single step take a guess so I can basically compute both the minimum and maximum running minima or any maximum at the same time right so I can map X out into two for two for a single number this is the minimum maximum it's the number of self is post minima Maxima I can then do reduce I'm gonna use reduce option but explain it so reduce takes a binary function and left and right so I'll generate a new tuple to masked or me arrow to underscore one that's the fullest field out of underscore one I really need rainbow currencies to be able to do this correctly but let's try I mean this is a pretty trivial problem but it's not trivially you know coded in a even in the languages I'm pretty familiar with do I have them correct number no no one okay so now I can compute both me a maximum at the same time that's also pretty trivial it just takes a lot of coding especially with tuples it's you know arrow prom luckily there's a library called LG but I'm gonna use two LG board which provides a lot of increases or category theory constructs like semigroups and monoliths that does these kind of operations automatically so I'm gonna actually increase its class rich I'm gonna enhance the sequence object data structure with some option master so that I can do this in a more magical way so there's some option what does it take increase a semigroup for the type T so that I know how to sum the type yep a semigroup has a plus Messer which basically defines how to plus certain scenes some certain scenes let's see alright so when I have a rich sequence so I can some scenes magically in parallel how do I do that let's revisit exists on map so from our G but there are these two data structure cause God what two data types call me and Max so I just wrap my number in it all right reduce option actually some option that's the reason I have to call it some option because there's already a some on the on the sequence trait define all right so it sounds them up and if figure saw a summons to increase a semigroup and there's the plus operation and finds out too many maximum the next step our extract these two numbers let's say a Cheeto underscore one docket you don't score to go get all right I know it's not known so I can just get it out so basically I perform the same thing but without having to like manually type the underscore went into a still it's still a little bit of manual coding but it's cleaner it's more generalizable and now I know the minimum X how do I map my original sequence to between 0 & 1 I can just normalize them by rescaling right so to rescale in each specs I have 2 minus X X X me and then normalized by the range which is x max minus X me right I have the correct number of current yes and you see some weird artifacts from the you know I triple ye double float number but that's another story anyway so this is the me max kala it's still pretty simple let's go back to the slides input is a double we map a double into a tuple two of me a max each of which have a has a semi group defined so I can compute them in parallel I summon a semi group of double minimax and then computes the minimum maximum which in this case I called abrogated or summary of my data as you go through every record to find out a minimum a maximum of my dataset and I used that to map the original data to rescale them so the three type of operations Map Reduce and another map one for encoders slightly more complex so let's define a new data set called Wiest I'm very lazy so just using very short variable names and short name label names but in reality this could be no Hashanah of a music or country or some kind of code categorical information so let's say I just made up some labels so the point of one hot encoder is basically converting categorical data into binary vectors so in this case I have ABCDE F maybe you see the ef6 different labels I want to convert each individual label into a vector of dimension 6 so if it's a the first dimension should be one everything else zero and speed the second dimension the second column should be one simple as that so to do this I first have to go through my dataset and figure out all the available available labels right obviously I can do this to set but it's kind of cheating and it's not how a lot of the distributed processing framework works so I'm gonna use the same Map Reduce map paradigm so I'm gonna do map each y2 set of single elements right I mean you can also shorten it you just this the sum option the map what does that give me that gives me a set a sum or a set of all the labels and I want to sort the labels alphabetically nicely so I'm gonna to another map afterwards to seek sorted something like that get okay now I call this labels again this is a summary of my data this is all the available levels in my dataset right so how do I have the original YS so I have to map each individual Y with the labels on map so every going through every label this arrow if if the label is the one I'm wops observing right now then that's one point zero otherwise it's zero right I'm gonna print it out nicely so it's easy to read I must have missed nope means the France is somewhere yeah there we go does it make sense so I have a map to one and five zeroes be map to the column B is one everything else zero and the next one C and B EC this is t EF so now I have a one hogging coda these are fairly simple to to code up for a single transformer right the problem gets more complicated if I have let's say except Y X what if I want to do feature engineering on two columns in parallel generating three three features I have to do a lot of like to pole underscore one two three we actually have algorithm with maybe two thousand features or something that just not humanly you know possible can craft like this so we wanted a better alternative to that go back to the slides what we observe so far for one hot encoder again it's the same you have a input string you map each string into a set apply a semigroup over the set to sum them up and then map them into array or sorted sequence and then use that sequence the summary to to remap your original input data set so we can abstract this into what we call a transformer so from a record T in this case my record is a tuple T of double and stream it could be a much more complex record it could be a case class with 50 fields it could be a verbal protobuf record we want to extract a single value from the record I can transform on in this case like underscore one underscore two and then there's this concept in algebra called the aggregator which is exactly what we've been doing so far Map Reduce map so prepare method takes the a input map into something that's summable in parallel the type beat with someone a semigroup on type b and then do the to the sum operation after that we present B into the type C which is a summary that we can use to remap data in the case of the one hard encoder it's the sorted array of labels in the case of mimic scalar just a tuple Mia maths right and I can transform the original data points a with this additional information the summary called see that's where a future income Z so let's take a look at this data data set instead of doing the manual mapping reducing a mapping I'm gonna use feature on so implants modifying future and okay and I am gonna import a bunch of transformers alright so I'm gonna have let's use a place mode so it's more readable well feature ins feature specification features spec tore off my input type original T as a double and string right that's my except with YS alright and the first transformer I want to apply is underscore one which is a double field I want to apply a binary so let's call it B and I think binary that takes an optional argument which is a threshold that's C in this case make it 42 okay quarians I want to reuse the first field at the end but applying a different transformer call me max scalar okay me max all right and then the third one and let's go - one hot encoder yep it's control-d to finish ah it's required sorry and call again he's got the history to help so required this is the first one second one and so right now I got my feature specification this is a spec saying how to transport data and I can extract from a data set in this case just data the sequence of two photos and calling it everything is lazy at this point nothing is being computed I'm calling this feature extractor alright now from this we can actually get our valuable data out feature names everything has a name and has a column name just imaging if it's a two-dimensional table or the columns mapping mapped to the original transformer name plus some other information in this case you can see that there's a binary feature from the first underscore one second feature is a min Max and then we have actually six columns one map into each of the original labels string labels in the world encoder so actually this is taking a tuple two mapping into eight vector of dimension eight let's see what else can we do with this feature extractor we can extract values out and values actually takes a second takes a type parameter because it's generate you can extract it as any data type you like we can make it a sequence of double so that it's more readable for each Perin genuine these are my vectors everything is concatenated together everything done in a single pass which is great I can also transform it is sometimes you want a spot seizure for example so I can do it's a map of strangers double-fist case essentially a lot denser because it's thrown away all the zeros from the one hot encoding some some features transformers a spots by nature so there's no point in coding all these zeros alright let's go back to the slides this is some of the basic functionalities of featurin I also call it a future and seventy-seven get you get it no f77 yeah anyway so we talked about feature spec you take your record abstract method two extra individual fields the Transformers that takes the original input a transformer to a some about be present it as a see which is a summary this is what we just did feature extractor you can extract the column names and the values and there's another thing that's important here called settings remember we have to do a global radio step that's very expensive in some cases you don't want to do that for example if you want to transform features in a streaming or backend light back in where you don't you have infinite input data set and you can't just reduce over it and there are some other cases where you have training set and an evaluation or validations that you want to do your future summarization on the on the training data and then use the same it's the same settings or same summary on the future data sets so let's go back again and futures settings let's call it s it's basically just a JSON encoded with all the information I need to know to transform data like the transformers being used and the parameters like the in this case Mia Max and in the case of one ha encoded or the visible of seeing labels so I can use this future settings in a future transformation let's do this again remember I have the spec specification that can reuse to transform any data set so I can do a extract with settings actually takes two parameter instead of one the first one is data second when setting in this case I'm gonna actually take the original input but append extra data point the data is too poor to write so I can use turbo to have a double and string so a 500 which is greater than the previously seen maximum and stream let's say Z which is also a previously unseen lie because i've we've only seen a 2f before right now I have a extractor feature variables actually we can use the future results which contains not only the transform values but also other extra information that's useful when dealing with cases letters you'll see in a bit when I print it out alright so the first eight result results are exactly the same but the last one actually has some additional information first one is a report of me max transformer have out of out of bound error because the original min and Max observed values are 0 and 100 and now I see at 500 which is out of thumb the second one is a 1 hard encoder also and seeing error I've observed this new label Z that's been observed in original dataset so that's great because if you deploy a specification in life back and you see and unexpected input you want to be able to reject those and the other fancy here is we are extracting from sequence but it's actually another high kinda type in this case just em and the only thing we need to implement for any data type to be able to do this kind of future transformation as three operations a Map Reduce in the cross which is mapping with a summary so it's implemented differently in different distributed systems like inspark implement with the broadcast in Apache beam or she'll you use a sighting put some some way or the other to make in memory lookups of the summary of your data and the last part where we extract the values it's also a trailer that way and decoupled so we can build features in any of the supported data types we have seen building sequence of doubles maps we also use pass vectors and tensors flow records pretty heavily those are basically sparse representation of your future and we can feed them directly into one of these systems yep and the future builder you only need to implement these couple of methods how to initialize given a dimension which is the output size of my future how to add a individual value which has a name and in the double value how to skip in the case of a sparse feature representation and then finally extracting the result and of course it has a map master so you can build feature builder with on top of other feature bureaus and setting is the one the last scene we just saw we can use feature on to analyze one data set extract the settings and apply it on other data sets in this case the skips the reduce step and it is a pure map function we've seen this the future rejection like the case where we have two values that's out of bounds unseen and you can be rejected some extra features like combining specification can have several specific issues merge them into one and then apply on the same data set because it performs once one's parts of Map Reduce mappings that instead of twice one for each specification future causing is another feature that we use basically you can have take two out two transformers and output and just make a Cartesian product of all the future values so it's great and an easy to experiment with different types of feature engineering you probably don't want to do it on a user x track though because we have 100 million users and 30 million tracks so the Cartesian products very big anyway it also has a Java API so because what if I mainly use Scala for data processing and most of our data pipeline things in Scala and our Perkins usually in Java so we can actually build a feature specification publishes artifact if I use it in both data pipelines and in life Java back-end in a Java Java backpack and we actually use the tensorflow j'ni binding and feed transform features into tensorflow directly for prediction I'm gonna talk a little bit about the use case in a bit these are the available transformers and it's growing as you've seen it's actually very easy to implement a transformer it's a MapReduce Maps three steps I increment and there you go some of the more interesting ones like we have instead of just a one hard and hot encoder we also have the hashed version those are used when the input dimension is super big like if you want hot in code on the user ID that could be like hundreds of millions of code space and hashing is just way more efficient then then the brute force set things like that alright some fun facts about feet rent it uses scholarship there's property based testing library and there was 100% code coverage on the first try didn't even try to cover all the colleges show the test run it and it was a hundred percent so it was it stuck ever since but the hardest thing I had to do was Futura has a Python numpy output support and just to cover that one edge case where you have the Python PA code serialization format that has a weird by a padding logic but fun fact it's 100% code coverage but there were a lot of lessons learned when working with scholarship in this case we a simple example I wrote a method that normalized vector by the Sun basically normalize it so that is sum to one what is pass the test take a guess or not the edge cases obviously and those are watch property based testing or scholar checks grateful in this case or not party if you have an empty vector because some were just computer actually yes some will complain it doesn't make sense on an empty vector what about adding a criteria like for any input that's not empty I want to test that the not the sum of the normalized vector is actually 1.0 no there are extreme cases where your double values are close to boundary like the maximum or minimum boundary and it will either overflow underflow from facts you can learn from writing property based testing actually a scholarship also has Auto thinking so for cases like die it might have a very complex input of not lots of numbers they will try to figure out the minimum minimum input case that would reproduce your error and in this case just the two numbers one positive extremely large and when active extreme nudge that's enough to produce the error the failure other fun facts that we learn and this is just some of them like testing the maximum minimum properties or not a number infinite they and they might not always have the Equality behavior that you expect so let's see what else it's a hundred percent cold coverage still has bugs because software it's hot serialization that was the hardest part in any distributed potato processing framework so lot of trying era the usual transient everything extend serializable interior walls concurrency a mutable state we do a lot of optimization at the hood like sharing arrays and vectors to avoid memory allocation and there was a lot cost by using interested bear instead of implicit death so every cost site reuse the same instance instead of creating a new one and they end up sharing state in a concurrent concurrency application it works fine in the data pipeline because most data pipelines does it in a single threaded fashion each map function has its own thread and internal data structure but in a in a Java backing where you have asynchronous operations and a lot of shared states it blows up these are the phone scene so we hit performance yes so we want to reduce the boilerplate the developer data engineer spent on writing pipelines we also want to make a performance so you don't have to like and you know a lot of the the typical Java Java Scala collection type of operation optimizations so SBT jmh that's great with a lot of micro benchmark and then just pick the worst offenders and then start optimizing from there a lot of handcrafted while loops specialized primitive arrays like there's a scholar key work on specialized and we use specialized array so avoid using like the Java bossed primitive like the uppercase integer or long and used just on stack allocation picking the right data structure like the cases we talked about hash versus brute force set for the encoders a lot of cases where search was as array or since neither anyway the last bit which is a use case so we talked about all these from fun facts about future and about testing and how do we use it so one prominent use case is the home page ranking it's similar to networks you go to Spotify app and has shows all these recommendations for you and we want to rank based on a time user or these contextual information so we have what we call shelves like horizontal scrolling paths where you have a name of a bunch of items like playlists and tracks we all want to order them so the user has a higher likelihood to start playing music these are the contextual information we have time of day week of day week day location and attributes or that so these some of them are categorical labels some of them are numerical features when a transform them into some kind of vector X could be binary it could be double continuous vector this is what architecture looks like so on the very left hand side you have this feature ization library which is basically the future spec that we just saw you have a future spec define all the fields I want to access and the transformers and this is shared by both the recording the reordering service and a data pipeline so first we take the future spec apply on the logs generate vectors that's ready to be fed into machine learning algorithm in this case tensorflow and then we train a model generate the tensor flow graph know they're back into the live Java service where we used the map only operation to transform features and that serves live production to users so this one highlights where if each one is used we have a common library shared between pipelines and Java services and both of them used the same code so it's guaranteed to be like deterministic and even the nice things that you want to have in a in ml workflow so this is the code she'll is the Scala idea that we use for data processing on the left hand side right hand side Apollo service qualifies a Java micro service framework code looks pretty much identical so left and right right and left that's it and yeah that's I probably have some time for pressure any questions the question is have I looked at performance of the elevation that's a good question it's probably not the most efficient scene the biggest potential point for improvement is in in a regular like a single eligible semi group there's a some option where you can some like you basically a user's immutable States to add to mutate in place instead of making copies every time you you know some two items it's hard to do that in a parallel where we have like a feature vector the writes feature of many you know many semigroup structures in parallel this is probably the one place where we can optimize but we're not doing it yet the question is if we have a Python from an actually not not exact we don't have a Python binding for this we have a output format that's numpy array so you can generalize your generate your features the binary that the float or double vectors and writing into my file and you can read it in a Python like cycle and process yeah the we don't have a Python binding because it will be hard to write Scala and you know have a compile to Python all the other way around but if you look at the feature transformers they are all very simple so it won't be really hard to to like reemployment a transformers in prison yeah again there are just want to point out like there are other libraries have similar feature sets I think those tensorflow spark in cycle and has some kind of feature transforming capabilities the reason that we are not using them is because they usually tied to a framework like if I do feature transformation in spark I cannot run sparkle in a live production back-end the same for for cycle and I we usually don't want to run Python in the production back end so you want something that works both in a data pipeline and in a by a back end with the same code and the way the the execution logic is decoupled like I've shown that extracting features from a memory like sequence but you can also run it down shields coding I think there's support of all of them she also scouting spark and fling because all you need to do is implement MapReduce and across a lot of business large area networks so that's it a couple more minutes anyone any more questions against that's the other [Applause]