Bay Area AI: Jim Downing, The Feature Store: the missing API between Data Engineering Science?
[Music] so I'm gonna start off I'm gonna do a bit of background on the feature store what is a featured store motivation for it I am a professor at a university in Stockholm but I'm also CEO of a company called logical clocks so trying to be as scholarly as possible before I put on my company house so I get to talk firstly about what is a feature store why do you need one what's the motivation for the feature store and then we've built a feature store it's part of our platform a data platform for AI called house works and it's the world's first open source feature store and then I don't know how much time I have for a demo but have to hold the microphone at the same time but luckily I have a touchscreen laptop so I'll be able to do something okay here we have our intrepid hero here the data scientist and if you've never worked in data science before and you had a vision of what a data scientist does apart from getting paid obscene amounts of money they might work on great problems they might be able to help us improve the efficiency of our wind farms improve the efficiency of data centers they may help contribute to humanity in a great way and through the application of data-driven science and this is a great thing to do and you get paid great to boot so you're like great I think this is the job for me and you know I worked previously with Python I love - and I love all the frameworks around it I like Jupiter like I torch I like pandas and numpy of course and I'm gonna go off and get this nice new job now the problem with this is that our data scientist comes into this enterprise and no enterprise has a single store of truth there's never one single data Lake where you find all your data you'll have different silos different places you can find your data and the data scientists come in here has been working with Python and is very good statistics is very good at machine learning understands how to reason about data when the data isn't that good format and you throw them in here - this data at Lake and they kind of go what you know what am I supposed to do here you know how do I find features in this sea of data sources I'm not a data engineer I haven't worked with all these platforms before and the people who employed them say well you know you paying you big bucks you know you should be able to just pull all this data together and you know give me some insights build me some models make some good predictions so the issue here is that data scientists are not data engineers we we don't ask business analysts who work with tableau or power bi to go down and dig down and find files and different databases and join the data from casket to hive and so on we give them the data in a format in which they can work a higher level of abstraction they can work with power bi and tableau to generate nice visualizations and understand the data when it's in a good format for them so before I you know we may have some data engineers here in the room and they may not have a full appreciation for what a feature is because it will try and define what a feature is before we talk about what the feature store is and a feature the general level is it's a measurable property of a phenomena that you're observing and you as a data engineer might think well come on it's it's it's the age of the customer it's a column in the database you know or but you might think well okay well I know that machine learning and deep learning they also use images if they're doing image classification so it could be a pixel it could be a row of word if you're doing an LP or if you're doing IOT you know maybe you're doing some sort of anomaly detection with IOT data so it's gonna be sensor values and then you try and say well hang on you know features are things that we use to make predictions with so often it's useful to have aggregates right so we want to know the mean of something that's a good property with which we can make predictions and if you're more sophisticated you'll know that we work with things like embeddings we might take text documents and break them up into word embeddings or sentence embeddings we may even use you know cluster representations or k-means clustering and so on and we might have graph embeddings so we basically expect all of this data which is features you can see here in the the x1 to XM vector that that's going to go in a trainer model we're gonna try and drive the loss to as low as possible using you know auto differentiation and sarcastic rating descent and you might think this is fine for a feature but the data scientist does a little bit more work than that they if they're working with natural language processing MLP they may take some raw text and they may write what we call a feature pipeline they might take that raw text you know make it lowercase remove noise tokenizer and then it'll get some words out of that and then they might decide to group those at my post and then we're gonna look at some different features that we get from this text document I mean they get word to Veck embeddings and a bunch of other features that you can see so these features will be great if in any organization every group didn't reimplemented entire future pipeline so right now without a feature story you may have groups that write this is code and Python code and maybe they published that Python code internally and some other group if they're lucky might find the code or they might not find it and they might rewrite the whole feature pipeline so the idea behind the feature store is that our intrepid data scientists now can work directly against the feature store instead of this sea of data and they that means basically somebody still has to take the data from all the data sources and build these feature pipelines the ability to take the the columns or the aggregates and put them and publish them in the feature store so that they could be consumed by the data scientist and we can see that as the data engineers job ultimately so the work still needs to be done but hopefully it won't be duplicated and add-on incorrectly so what we need to see to get to a feature storage we need to have features as first-class entities in your data platform and that basically means you need to be able to discover features so if somebody else has implemented a feature pipeline you would like to be able to look for it find it reuse it and we'd like to feature maybe to be access controls maybe not everyone in the organization should have access to it features should be versioned if you have a new version we want to make these machine learning it's reproducible if features are not person that's going to be a hard thing to do we want to govern them they're assets that a company wants to know which ones are there and control access to them and we do this for the reasons of reproducibility as as I mentioned already the other thing the feature stories is if the data management layer so if you have aggregates of your embeddings that you've computed you'd like other people to be able to reuse those embeddings and aggregates because you know it takes time to compute them so if you can just reuse the ones that being computed already you you'll go faster you iterate faster as a data scientist you may also have features that need to be computed on-demand you want to be able to do that as well and the main overriding goal of this feature store would be to help solve the data problem so data scientists don't have to solve that work themselves so we've been working on our own feature store called the Hoffs works feature store and we have a bunch of properties that I have listed here on the left of the future store so you know we want features to be reusable between models and teams we wanted jobs to backfill features were able to compute them pre compute them and have them available when you want to generate your training data from them we want to document our features we want to have versions of them we want to standardize access to these features between training and serving and you know what able to discover features and access control of them so the main point here but if the diagram is showing here is that if you've invested in machine learning you've decided as an organization machine learning is core to what we do and you decide well we need a feature store to manage our machine learning assets as you add more and more features your features store the cost of doing machine learning should reduce because features will have already been implemented somebody would have Dyl not burt pipeline already for you to compute it to transform your document into a format that can be used with burst for example which is a used widely in NLP these days so the goal for any organization who wants to have the investment I should be let's reduce the cost machine learning as much as possible in the future store should help you get there so there's a couple of other advantages to the feature store I haven't mentioned I mentioned Burt just a couple of seconds ago Burt is a very popular pre-trained model that Google released for doing NLP because you can actually use that pre-trained model and specialize it by training adding extra layers to specialize the the training of their models you may take emails on the left in marketing and you want to classify your emails you take the pre trained model do you add some extra layers but you still have to take the emails and convert them into features that Bert retain model wants to use and there's gonna be other teams in your organization that see Bert and say this is cool and they're all working it in parallel and you'll have different implementations may be compatible maybe not of the same earth feature pipeline so you know you don't want to do that you want to be able to reuse any effort made by one team the other issue that's a problematic is that if you've different teams then you don't have a feature store and one team is implementing a feature in training and another one in a deployed model that's being served and those features have different implementations maybe they're different pieces of Python code they're not necessarily going to be consistent and that can give you correctness problems when you comes to making predictions so there are a number of feature stores out there in production but it's a very new field so logical clocks our company we've released hops works it's an open source platform for data in AI as I said and it already has a feature story in there there's another open source platform recently released I go Jack a company from Indonesia based on Google Cloud data products it's called feast and then there's a bunch of other feature stores by uber uber we're gonna hear about Airbnb later twittering Comcast also said they happened I know there's a bunch of other companies that haven't announced that they have one so they're being used and practiced by a lot of companies but they haven't hit the mainstream yet so just to kind of summarize this background introduction to what a feature story is what we can basically see here is that the feature store is going to act as the API between our data scientists that are going to consume features from the feature store to generate training data and then we have our data engineers who are gonna write feature pipelines that either with data scientists or or are themselves and publish them in the future store and this is a nice separation of concerns enables teams to organize better and we think enables data science to be more productive so I'm gonna have a couple of words now but if this slide changes for me there we go I bet our particular feature store of hops works at logical parts of the company and a little bit of background on us because I guess not everyone knows who we are we're based in Stockholm Sweden but we built a platform called hops it's a data platform it's a next generation Hadoop platform we got like 16 times the throughput of HDFS with Spotify about a year and a half ago we added GPUs to Hadoop we were the first vendor to do that and we've recently added small files on nvme disks and our metadata layer of the file system so if you're doing machine learning on millions of images you can read them in seconds instead of minutes and then finally we we were the first company to release a open source feature store so the platform itself is it's a data platform for AI and it does things like batch analytics so you can have your data in from s3 read it in you can write pipelines in spark and we have a data warehouse I've and we also support streaming analytics so both Lincoln SPARC and then we have deep learning as a kind of core tenant of the platform I say deep learning we also support machine learning of course but with deep learning we need to have specialized support for GPUs of course and it's all based around the feature story so what we can see is that we were actually a very notebook centric and Python centric platform and typically users will write their jobs in in notebooks and schedule those jobs and run them and then you you you tend to run feature pipelines with air flow so you push those notebooks together to make outlines and then you can run those as a machine my plans so the a couple more things about the platform just to put things in context so we can understand with some of the features later on and our platform has something called project based multi-tenancy so what that means is that if you have users and here we have three users and five different data sets and the data set could be a hive database it could be a set of models that we want to put into production it could be a feature store it could be a calf the topic or it could be a subtree in HDFS what a project is is really just a line you draw around users and data sets so within a given project the data scientists who work on that can be restricted to only processing the data within that particular project they're not able to copy the data out it would copy data in not able to cross link with data outside of the project so we do this with where we implement this with certificates so it's not like you're your mother or Father's Hadoop it's Hadoop with TLS certificates and a distributed metadata layer so it's a very flexible abstraction because we can use it to basically share data share feature stores share models that we publish amongst teams and you don't if you don't need to maybe have boundaries between your groups you can just have a company-wide project where all of the models and all of the features are available one other point about the platform it we we also support distributed deep learning that could be distributed training so it could be collective all reduce on tensorflow using this is reason sparked Rwanda or it could be hyper parameter optimization so the way we're doing this is we're using PI spark to manage distributed jobs so when you run a job on in hops works from a notebook or from from the jobs UI and it's going to be a PI spark application so inside the executors we're going to run tensorflow or probably torch and each executor will get a GPU because we have the whole stack so we're using our own version of yarn to manage GPUs so what's slightly different about this is that instead of using docker for example to have immutable infrastructure we have a every project will have a count environment which will be replicated all the hosts they really they really they're looking hard for me I'm an illegal immigrant I guess it's okay I think I'm from one of the good country so that one of the other ones okay so one thing that we've learned a lot heavily from experience is that if you're gonna do distributed deep learning you know if you're going to have these large infrastructures for machine learning you're not going to do training on one GPU or you don't want to do it hard for an authorization of one GPU and really a distributed file system makes everything easier and possible so everything from sharing your training data sets and your test data sets to the executors so that they can just read them to do their training or evaluation to outputting the results or logs of your training runs to managing the experiments that you run to managing your tensor board log files to publishing your models so when you put all of this in a single distributed file system it becomes very easy to manage it and to to have this things scale and make it usable for data scientists so that's the basic way in which we're managing both training with thoughts gpus and i programming optimization with lots of GPUs and just one word under replicated current environment approach we're using each project will have its own count environment and we do have base environments like for Python 3 and Python 2 and the Python 3 one would have tensorflow and pi torch and map lived and numpy and pandas but you know there's always people who want different libraries and maybe 20% of the time the users will go in to pip and they'll search for a library or Conda because we support both Pip and Conda for searching for libraries and they'll install it which will basically mean that we have to create a count environment at every host in the cluster for that particular project but then the Python libraries for that project will just work within that project so different people can have different versions of libraries without any any interference now many of you will have seen this diagram before it's it's been quite well widely written about it so at the technical dead of machine learning and skully at Google published that I guess about two years ago and the point of this diagram was just to show that the training of models which is this small box in the middle is only a small amount of the work that you need to do is a data scientist so what we've been talking about today is really how do we make the data scientist job easier with a feature store so what we see the feature store is doing is is solving part of the problem so the data validation data collection feature engineering part work we abstract away and we make it available to the data scientist as a feature store and you can see a little imperfection in the artwork there and in this case art does you know reflect life and we have imperfections in our features store you know we're trying to make it better but it's something we're working and the other part of the the diagram is you know distributed training out prom during publishing models workflows we have an API for that and we use air flow and that's the way in which people work with that part of the platform so really everything becomes programmatic these are systems that do different things but we're making them available through AP eyes so that's putting the feature store in context but the I guess the starting point for a feature store is well if I want to work with a feature store what does it involve what are the key concepts in a feature story and I can only talk for our feature story well we try and make available to the programmer the programmer basically sees features and we talked about features already so these features are the measurable properties that could be a column from your a database it could be some aggregate or some embedding but we also have an abstraction called a feature group and the feature group is effectively an a job that will compute a set of related features so typically for us that's a spark job and you know when you compute the feature values you would run you will get all those Peter values within a group to get computed together what we have at the top then our training data sets because that's typically what a data scientist will do the feature store they'll sit down they'll look at the feature store and say okay I now want to train a model using these features that I see in the feature store and they select the features and then they say please create the training data set for me now that may sound like magic and many of you will know and think that well hang on if I pick features from different feature groups and they don't have a common key how does that work right well we have a we have a query planner which tries to find a common key for you and but if you if that comm key is not obvious then you can supply that keys yourself so even though these are the abstractions we provide we I wouldn't say that they're leak free you know there is a certain amount of leakage and that and that's something as a community I think we're going to work around and find it what are better ways of you know keeping these abstractions as clean as possible but you know if you have data related to customers and you've data related to time and you try and join them together you need to specify how you join them so the building blocks of this feature story we we had a discussion before they said with with Toby and some others about well what makes up a feature store and this is really you know our opinion on what what our feature store has but firstly we have a registry you need to be able to find features and then you have an API you need to be able to create features publish them and you need to be able to consume them the features will have metadata the version of them the name the documentation and you'll have jobs to compute your features as well and then we store of features we store training data so we see that as a kind of a key part of the the feature store as well so here's the API for the data scientist if they want to read from the feature store they go to the feature store and you can see that we have a data frame equals feature store that get features so this is a global namespace you basically put in the string of five name of your feature and here we're looking for two features average attendance and average player age and we get back a data frame and with that data frame we can now create a training data set and we support a number of different file formats for train data sets you can have TF records numpy peda storm which is a new park a data set for you know if you're really working at scale it's by uber and CSV and hdf5 so the data scientists will typically use the feature store this way and there they've got training data they can train their model the way that they would normally do with the training clean training data the data engineer who's who's writing feature pipelines will maybe not necessarily write their code in PI spark we also support Scala so you know at one company we're writing our our feature pipelines in Scala but basically what you can see here in this particular example is we're reading up some data frame raw data from a from a file using spark park a file and then we're engineering that particular raw data we're just taking the square of the values inside us we're calling that polynomial feature group and we register that or insert it into the future group so now data scientists can go and use this polynomial under or feature group feature in a to generate training data so technically what we're using to store features is we're using hive the hive many of you may know as an Apache project it's a sequel on Hadoop database and we have metadata we extended hive to include feature metadata we're using the same meta store back-end a database called NDB to store that and I'll just then put our feature store in context of a pipeline that you would build a machine learning so if you're starting it with your raw data or your event data and you're ingesting it in via Kafka you can obviously invent ingest it other ways using scoop and weave arrest API our platform and your pre-processing your data in our case typically will be spark for computing features typically that data will then flow into not just hive databases or our file system which is HDFS compatible pops of s but will store all of this data in our future store and from the features store you can run your tensorflow or pi towards job and in our case where we're embedding it inside our PI spark application to do the model training and when you've trained a model you need to then serve the model and we use kubernetes to serve models that's where you know models that are not used necessarily just for batch but that models that we're gonna serve over the network and but you're not done with that right because this is just a a pipeline it's not we need to share through the whole pipeline and we need to monitor the pipeline so to monitor the pipeline we write our logs to Kafka and then we can have spark streaming jobs to to read those logs I'll talk a little bit more about that in the next slide and we're using air flow to manage the entire platform so the interesting thing I guess about this is that you can write everything in Python this is Python the whole way from from feature engineering to training to to monitoring your applications and then to even running workflows with air flow so I said I mentioned the model serving so the way we support it is that our platform has a REST API so clients can use access control to their models yeah via REST API and then the requests if they pass access control gets sent to kubernetes it could be for example tens of a models serving server that's running on a docker image there and the response you can then load the predictions to Kafka now if you're generating a lot of traffic you might want to log all the predictions just a prett certain percentage of them and but the reason why you want to log your predictions is because to the keyway the gold standard for monitoring the quality of your models are being served in production is to to link those predictions with outcomes so that you'll know what is the performance of our model how well is it predicting and how well that prediction performance that it has at the moment how does that compared when we were trained when we're evaluating the model and if there's no concept drift there we can keep going otherwise we can notify and maybe retrain or do something to kind of help improve the prediction performance of the models so I think I'm done a huge amount time but I'll do a quick demo if I have just a couple of minutes just to show you because this is all very abstract and let's see if we can get the screen to show this do you case okay okay I'm holding the mic at the same time it logged me out to this so this is running on the far side of the world so the Arctic so I'm inside a project here just shipped over out and we have this thing called a feature store here and we have tours for this feature store to help you get started and what we can see here is that you know we have a number of feature groups we have training data sets that get created we have the features and you can search for them there and we have jobs associated with those features for training them I can show a quick notebook then to show you what the API looks like so in this particular example notebook is it's some housing data you can reproduce this yourself by the way it's a part of the tour but what we're doing at the beginning is we can see that we're creating a spark data frame from houses for sale and from that data frame we do a little bit of feature engineering we're computing another fact the spark data frame house is sold we have a look at some of the features for that and then registering that particular or generating the features from that here we have some aggregates who are computing so averages over the houses sold and basically we can then register that with the feature store so that basically involves this line here where we call feature store to create feature group and the API has quite a different number of features you can have your description of the feature you can decide if you want to compute some statistics over which we shown the UI feature correlation in history rounds and so on and then the data scientist who wants to use that feature that's been registered can just call here feature store get features I'll just make this a little bit larger so we can kind of see we can they can call get features which are picking basically features that came from different jobs that are region used to compute them and create a training dataset and this would typically be a whole notebook just this particular cell here and once this notebook is run and you've got a training data set you can then go ahead and you can go ahead and create a model to find a training function and then run your training job and that's basically how you get to use it so let's have a look where we going okay okay so it's a like a very quick overview of our feature store and jump back to the slides so if you want to learn more about the feature surah I give you access to it and a couple of slides time our platform but just to summarize what we've been doing we developed this feature store as part of our hops works platform because we're really trying to make it easier for data scientists to do machine learning and easier people to put machine learning pipelines into production we're not done there's obviously always work left to do we don't have an online feature store yet we're working on it some other companies do have libraries or our dsls for doing feature transformation we're working on automated data problems I didn't go into that much details but being able to show which jobs which data sets were used we're trained were generated by which jobs and which models used which input training data sets to train them it's very important and you know technically we're also looking at we're using hives so we want to use hoodie to do incremental updates on how if when we backfill features so if you want to look more the platform we're doing a few events in the valley in the next you know six or seven weeks I guess one of interest is we've a hands-on lab and pelts on the 23rd of December I think it starts at 5:30 these slides are available on the meetup side so if you just click on that link or go to our website logic class comm you'll find it in the event section and it's limited in place in spaces so get there early you can read more we have a blog article put a long blog article on features are written by Kim and myself and if you want to try out the platform you can follow this link here at number two and there's a couple of instructions we have a manager version of the platform running at hop start site and we'll give you access to it with a bunch of GPUs to use so that's it and I think I'm a bet on time ish [Applause]