Cognifest NYC 2017: Fallon Chen, Featran77

Recording: Cognifest NYC 2017: Fallon Chen, Featran77

I was gonna have to tell terrible jokes alright so hi everyone I'm felon as Josh mentioned I'm working on his team as a on machine learning infrastructure and I'm here to tell you about patron which is type safe and generic feature transformation library it's mostly going to be a discussion of the motivations for it how we designed and built it and a little bit under the hood of how it works after me Sammy Hanson is going to talk about how his library actually gets used by machine learning in engineers so look for a different talk cool so let's begin I think it's probably good to start with what feature engineering is some of you know and some of you probably aren't that familiar with it basically the definition is it's the process of using domain knowledge to transfer features into a form useful in machine learning algorithms so this definition tends to just raise more questions like what is a feature what is useful mean in machine learning algorithms so let's start with what we mean by features feature is any measurable property or characteristic of a phenomenon being observed so to make it concrete as followed by we serve songs so one of the things that we want to learn a lot about and want to use in our recommendation algorithm is another his song features so a feature could be something like the name of the song maybe or the genre or the number of times it got played so as you can see it finds some code here a case class for song probably referring back to this song throughout the slides and created a dataset called songs it's just a sequence of songs and we'll be referring back to that as well so hopefully that makes concrete what a feature is it could okay so what does it mean to have a feature that's useful for machine learning well generally machine learning algorithms operate on collections of numbers so these are usually doubles or floats for instance in our recommendation systems we tend to use some form of collaborative filtering you create a 2d array of doubles representing users by mapping information about songs or crossing users your set of users and your set of songs and then mapping those two zeros and ones based on whether or not the user liked the song and then from there you apply some matrix factorization and other algorithms to pull out vectors of numbers representing user taste and just representing songs and then from there you can use your user vectors in your machine to find to make recommendations you can check if the users taste is similar to another users and you can do many other things probably watched some explanation I think there's some mixtape engineers in here so you can correct me but yeah so the main point is that but features useful for machine learning generally look like a collection of numbers raaah features that we get day to day generally look like records with multiple fields in them that aren't necessarily something you can easily process so the question is how do we get from wrong features like song to machine learning algorithm friendly features like song vectors and the answer is feature transformation so to make that concrete and a walk through three basic feature transformations and try to pull out what makes them similar so the ones I'm going to talk about are the finer Iser min max scaler and one haunting coding okay to start with finite assert it's a pretty simple concept even a feature and a threshold return one or zero based on whether or not the value and that Reacher crosses the threshold your some code to make it concrete so let's say we're looking at whether or not songs are popular so we're gonna pull up we're gonna check to see and we're going to define popularity as having more than 50 million place so that's your threshold so we take our list of songs and we map them to one if the number of places more than zero more than the threshold and zero otherwise and as you can see that's what you get so that's a minor Iser for simple the next one another basic transformer has called them in max scaler basically you give it a min and a max so you define a range and then you scale your features to fit within that range there are many ways to do it but generally you you have to at least find the minimum and maximum of your feature values so and then combine that with the min and Max in the range that you were providing so in order to implement this one you could actually do it in one pax to get them in in the max you can take your songs pull out the number of place map those two tuples number of place and then on one side of the tuple and then do it reduce and on one side of the table find the minimum and on the other side of the tuple find a mess and that way at the end you should get the minimum and the maximum so and then from there you can take the min max that you found and calculon skill your features down to that range so as you can see in this last line of code we're taking X which is a song and scaling the number of place to arranged between 1 and 10 and they all here fall between 1 and 10 so that's min max scaler also pretty simple just take note of the map in the reduce in the aggregation step okay the last time I talked about it's called a one hot encoding this one is if you have as you have categories that describe features and you have organized them in a sequence you can transform features into a sequence of equal length with one in the position matching as category and zero otherwise so this it's kind of a confusing definition but it's pretty easy to understand if you look at an example so for instance we have this song called bleed by Meshuga its genre is metal and we have a sequence of genres let's say it's hip-hop metal and rocks so your one line encoding of this song would be zero one zero because you have one in the first position zero indexed which is corresponding to metal and this is roughly how you would calculate it you would first find your unique or a category this is known as Sir John Res so first you extract the genre then you wrap them in a set so now you have a map of sets then you reduce the sense and that gives you a set of distinct genres from there you sort them and you have your sorted array of genres you can then do your one-hot encoding by mapping over the songs and over the songs and for each genre in the list of genres is that genre matches then it's one otherwise it's zero so you can see in the output I'm in this comment you have the one haunting codings c-101 so that means it's rock zero one zero that when it's metal it's gone cool so are you seeing a pattern what it's been happening in each step is roughly you extract the feature you're interested in from your records so for instance in one hot you've pulled out the genre from the song then you've done some kind of aggregation so in some kind of global aggregations are so in one hot that's mapping your jonard to a set and then you reduce it using site Union to get your unique genres and then you map that back to a sequence so that's something you can use to do your one hot encoding and then at the end you can transform your song your original set of inputs which are type 8 here using the aggregate of type C so you've calculated and do your encoding so this is roughly the pattern and the binarize err didn't have the aggregation but otherwise you can see this also works for in max scaling and so actually those last two steps the irrigate and the transform are called a transformer so I guess now we've set up what feature engineering is what feature transformation generally looks like and what future transformers are so now you can finally talk about btran which is a play on the words feature transformer but also a really old scientific computing language so the other name for this libraries f77 this was written originally by neville and rich Whitcomb actually mainly by rich and they collaborated to make it into the early extensible and nice library dudes today ok so we talked about what if what transformation looks like and so now we need a nice way to specify it and organize it because what I showed you in the slides was kind of a mess like a bunch of maps and reduces and pretty soon if you're building a feature that combines all of those things you're gonna find that your code is getting pretty unreadable it'll be hard to reproduce it'll be hard to handle other people to use so you want a nice way to specify what it is and if you try and gives you that by providing something called a class called feature spec and stands for features specification the two methods you probably concerned with right now are required and optional they pretty much do the same thing except and so I'm just gonna talk about required first you can see it takes two arguments one is the function that extracts your feature of typing from your record of type T and then the other argument is the transformer you're interested in we'll get to Transformers later but we just covered basically what they're going to be the binaries are max scalar and things like that optional is pretty much the same thing as required except that the extraction maps to an option of your future and defaults to none cool so now you can specify your features how do you actually do that it looks like this you specify your future spec you tell it what type of record you're using then you start talking about so a feature spec will build up one feature so in this in this entire piece of code you are specifying one collection of these transformations so the minor eyes are from before the min/max scalar and the one haunting you can see for the bio riser we're pulling out the number of place defining it as it's popular and setting the threshold is 50 million something for the min max scaler we're called renaming it and I'm play normalized and pulling out the number of plays for it to scale it has a default minimax and that's why it's not specified and one heart cutting coding for John Hren as mentioned both and I you have your specification what do you do next well know you want to extract your features what you do is you call extract on your future spec and you pass it your input set so remember that I had to sequence of songs at the beginning and they could just pass that in without really making any changes and that returns feature extractor before we go further jump into a little bit about the extract function because it uses the type class that uses type class implicit pattern and to give you a lot of collection types for free so here is the trait collection type and basically if your collection is something that's included in few Tran and it extends already it provides these methods then you can pass that collection type as an input into so things that you get out of the box with you trend include Scala collections as well as several distributed collections such as sparks RTD well so enough about the input we can look at what feature extractor actually gives you and these are the things that you're generally interested in and one other thing so the feature names feature values and the feature settings talk about feature names first because it's easy future names are what get passed to the transformer at the beginning so I see here was it's called like it's popular and um play normalized genre and after you've done it you're done extracting your songs you get and you want to know the feature names you get a list back of what they are and they correspond to the order of the future values so I can move on to the part that's a lot more interesting which is the feature values and the way to get those is you call feature values but you can also pass it on type and you'll get back the type that you specified so sequence doubles if you ask for a list sequence of doubles pasen arrays and many others and another look under the hood how is this done another type of class of class it this time extending the trade future builder and this one gives you these sort of generic methods on a one-dimensional collection type you want to initialize it by telling it how big your feature is going to be and then you can do things you can add values to the future you can skip values if you don't you know and you can get your future result back and you can also transform one feature builder to another and if you provide those things then those are valid output types in the trend and so feature and provides these out-of-the-box traversable sparse vector tensorflow example protobuf and many more and you can check out the github if you want to see them okay so that was future values the last one was pretty interesting is future settings so you remember that we covered an aggregator stage in the original example for one hot that was your set of genres and it through this like sometimes you can save those results and to recompute them and Futurama provides a way of doing that called feature settings so once you've extracted your from about your feature extractor you can call feature settings and it will return a list or sequence of string and those strings are actually JSON strings and they contain things like what transformer was and also the values produced by the aggregators so I can mention in the binarize er there wasn't any aggregation so there's nothing here in the mid mac scaler you actually have stored the minimum number of plays in the maximum number of place and for one hunting coding you stored the genre labels are they unique yeah list of labels so that's how use and then from there you can save this as a file and do whatever you want with it because the other thing you can do is you can load it later now that you've you can call the function extract with settings instead of extract and extract your input set of songs and passing your settings and get your feature extractor back you can go on your merry way and keep working with it so those are the three major features of feature extraction the name the values in the future settings there are also a few more I can talk about one of them is built-in it's called future rejection and basically you can some try out some of the transformers will define will catch errors and provide and start over you can get a collection of rejected features for a record and based on that you can choose to keep or reject a record this is useful because sometimes you just want to know something went wrong when you were building your future maybe you had an out of found error or the method didn't work or something so that's feature rejection is that you can also combine future specs you're basically these stains separate you're building separate features at the same time it could be useful if you're iterating over the same data set there's also feature crossing you can cross the values of a pair of transformers you just specified a pair so your binarize are in your mid max maybe and then you specify method for combining up lastly there's also a Java API so if you aren't able to use Scala for some reason you can do this in Java and here's the list of transformers that are available we've got the visor as I mentioned a bucket Iser hashed end hots the one hots heavy-hitters multiple other scalars don't check them out I guess just to summarize so why would we use the trend the main thing is like trying to make reproducibility of future engineering easier I'd argue that you get a lot of clarity and flexibility in how you compose or build features from the API and it gives you the ability to save intermediate stages using feature settings so people can also reuse your work also many common transformations transformers are provided out of the box and it supports many input action types and output feature formats so that's pretty much it if you want to know more this is an open source project check it out at github so just bottom by slash patron thanks for listening and do you have any questions [Applause] it's a that right uh yeah it's a supported output format tensor flu example protoboys I think his question was you mentioned something about tensor flow and yeah I did very quickly it's a supported output or not further features so you can do some feature transformation and then output it as tensor flow example there's an answer your question [Music] you do use tensorflow this isn't whether they listen and the question was what happens if you load your aggregation and there's something wrong with it there's a library do anything to detect that that's that's a good question actually Sammy might know all right it got a spot I don't think it does anything special to China we have writers or so you or yeah [Music]