Bay Area AI: Nick Handel, Machine Learning Infrastructure at an Early Stage

Bay Area AI: Nick Handel, Machine Learning Infrastructure at an Early Stage

Recording: Bay Area AI: Nick Handel, Machine Learning Infrastructure at an Early Stage

[Music] hi everyone I'm Nick handle I'm the head of data science at branch international so crew from branch here that was the group that raised their hands when when Toby asked does anyone know about a future store so yeah so branch International is a micro lender so we lend anywhere from two dollars and 50 cents to about a thousand dollars in Kenya Tanzania Nigeria Mexico and as of a few weeks ago India so it's a pretty small company we have about nine engineers and four data scientists and so we're you know deploying ml infrared a pretty early stage but credit modeling fraud modeling and you know improving our operational efficiency it's really all core you know to being able to deliver loans that are you know really really quite small and so we really need to figure out how to do that at scale efficiently so basically the the case that I'm gonna make is that the future store is probably the first thing that every company should think about building out as they start to harden some of their machine learning infrastructure systems so I maybe you've heard this I certainly believe this but people say that machine learning is 99% infrastructure I really agree with this I think that you know there's a ton of code just on the back end that's doing stuff that isn't actually you know it's not them it's not the machine learning it's not the modeling it's just all of the work that it takes to get the data there to get the model to train on the right system and then to deliver that model unfortunately the infra is really really difficult and I think this is the general AI landscape but there are you know hundreds of companies with different solutions trying to do a lot of different things and there are a lot of great ideas in there but you know where should you start like how do you start when you first want to deploy a machine learning model you know you have a few engineers you're getting started it probably looks something like this it's you know you take something that really you know maybe it wasn't meant to be used that way and you you you know you take something else and you kind of Jimmy it together and you get that first machine learning model out there and you know there are a lot of great solutions but there are also you know Thoreau there's also a lot of complexity and just getting that first machine learning model out there but you know once you have that going there's a lot that you can learn from some of the big tech companies that are pursuing you know some really interesting stuff so I happened somebody joined the meetup so I think this is a really great paper and there are a lot of really great papers from Google and a lot of other places that we can all learn from this is the same one that I think Jim just mentioned but you know there's tensorflow extended there's a lot of you know interesting stuff happening at Google to make machine learning happen uber has Michelangelo there's a lot of really great stuff there and before I joined branch Toby mentioned I was had Airbnb so I was the product manager for a big head and then before that I was the data scientist so everybody's you know big head system I obviously I have an affinity for that it's a really great system that they've built out but what about the rest of us you know their public solutions are kind of lagging there's not there's not a ton out there you know big cloud providers are providing these kinds of n10 systems and or they aren't providing these n10 systems I think there's you know they're they have the goal of providing them but when you really get down to it when you really look at what what it is it's probably not going to take you the whole way there really aren't very many enterprise solutions that go intend I think that this is changing and there really isn't any widely adopted open-source solutions for a lot of this infrastructure so the option set for the rest of us is to buy by various pieces of this and figure out how to kind of glue it all together and that you know requires engineering it requires money and a lot of these pieces of inference I had to change this from don't have solutions to didn't have solutions so there's there's a lot you know and this I think is really gonna change but feature stores is definitely one of them and then your other option is to just build and if you're a you know a company with nine engineers and for data scientists that is a really really daunting task that's a lot of infrastructure they have to build so it requires a ton of engineering and most likely you're gonna get a lot of stuff wrong because it's really hard to figure out how your company is gonna scale what's gonna change about your team's needs and you're probably gonna have a lot of tech debt so I think that the general idea of you know how machine learning happens is is kind of agreed-upon if you look at all these different systems they all kind of you know have the same pitch its extract some data build some features train models serve models and then monitor those models and that makes a lot of sense but data is really at the center of all of those different pieces so first you know you'd probably have to connect to arrange different data sources maybe you know you want to connect to a Hadoop database you know maybe you want to connect test3 you know maybe you have some other system maybe you want to connect to you know a Postgres follower whatever you want to do there's a lot of different places that you can connect and most likely you want to validate that data because there's probably going to be a bunch of systems in there that change over time if you're a growing you know company using machine learning and yeah so you want to collect features for a variety of different subjects you probably want to run experiments you know with models that use different subjects you probably you know especially if you're a business like branch where our labels for our loans come in with a you know a very long delay and you know as long as the loan and maybe even longer than that you probably want to be able to backfill historical features with some concept of point-in-time correctness and so what I mean there is that when you go and you backfill this feature you probably want it to be exactly correct as of the point that you ran inference on the model and so you definitely do it's you know it's the way that if you're going to go and train your model you're going to have an accurate representation of you know the decision that you would have made at that given moment of time and you know you probably want to collect and transform features for testing new model ideas you know you want to make that really easy you probably want to make it consistent you want to you know make sure that it's consistent between inference training development that's really challenging unless you have some kind of common way of doing this and yeah maybe you want to share model outputs as features you know have some concept of a learned feature a feature that is constructed from a model and then input into another model so and you know kind of you want to also you know potentially cache features at to go back and validate your training with your with your inference calculations of features then you probably want to monitor it all modern or for feature drift concept drift so there's just a ton of stuff you know like you look across this and along the whole way you're you know diff you know finding ways to calculate features that exact you know precise moments of time to monitor it it's it's across the entire platform and that's really really challenging and it's pretty obvious that data is just at the center of all machine learning infrastructure which is incredibly obvious like of course it's at the center of all machine learning infrastructure so the way that we've the approach that we've taken is to kind of start basic get something out there build a future service and it's you know it's basically at the center of all of the machine learning that we're going to do from here and I think as it you know as early stage company hopefully we're avoiding a huge amount of tech debt where we just don't know you know how our features are calculated or you know really just slowing down iteration and then the last bit is just you know finding ways to mature the pieces of your business that are most important to you so going into our future service a little bit it's probably good to start off defining a future service luckily I get to go second so this is far easier for me but the really simple definition that I have and actually maybe just to back up a little bit I think that defining a feature service is really challenging because this is something that you know pretty much didn't exist not too long ago and how companies are implementing it at this point is very different there's no kind of you know common understanding of what this actually means but a very simple definition in my mind is it's a service for computing and managing machine learning data and but you know obviously you have to go a little bit deeper than that so in order of importance I think first it's a framework so what I mean by a framework is its reusable code it's you know the code that actually you know runs your extraction of data your transformation of that data and then the final construction of all of your features it ensures consistency between you know training development and inference and it makes it far easier to develop features you know you build this framework around how you construct features and then a data scientist can come in and say I have a feature idea and rather than you know going and writing a long query and figuring out how to pull this and maybe joining it with a bunch of other data and then you know writing some you know chain of transformations they're just pulling from these already constructed transformations so second it's a computation engine so it's the way that we calculate our features it's the service that actually runs those and it's it's also the ability to go and backfill you know these new features for old inferences which i think is probably you know varying level of importance to different companies but I think a lot of companies it's more important than they then they then they understand now and at the very least I know that a urban view was really important and at branch it's really important and then lastly this is core to the definition in my mind but I think that it's you know basically the ability to just store features to reuse them so just storing the features as you're calculating them in you know in training and then in inference so at a very high level this is roughly what we have like this is an incredibly simple architecture which is probably a good thing for a company with a really small team but we have a future repository that is dynamo TV we have a feature service that is a flask app and that runs some code that I'm about to show you and in inference training and development we make requests for versions features that at some specific timestamp and the future service basically handles the rest so you can see that everything reads from the future service and then the future service is writing to this feature repository and because we're caching all the features there we can then read it back to the future service and send it back to inference or training wherever we want to do machine learning so the life of a feature so at the point of inference this is kind of in some ways this starts at the end so bear with me but at the point of inference we calculate and cache some feature and production then let's say we go and we decide to iterate on the model we read from that cache and use those features for model development then during the next training of the model we also read from the cache and use those features for training we know that those features are exactly calculated at the point in time that inference was run and that's a really great thing and then for future iteration we also read all of our old features in so that we can then explore new features to add so let's say we've explored all of our old features we've added a new feature and we want to train with our new feature again so this time we have all of the old features we're already in the future apposite or e it's stored with a key that is the feature name the inference ID which is probably like a user or something like that and the timestamp that we calculated it and then we will write any new features so the second version of the feature that we just created back to the future of mod story then in the future all we need to do is just pull all the features that are calculated and the really hard part there is this training train with new features and saving them to the cache because that requires you to go and actually backfill features for the particular point of time which is this inference that happened a long time ago so as a business we've really prioritized being able to recreate any of our data at any given point in time and we can then go and validate that by basically running inference and then going back and trying to recreate that exact same feature at some future point in time so as a business this is kind of cool this is a milestone that we just hit recently we know that we can recreate any feature any piece of data at any moment in time for all of our data so that's a really great thing this is one of the benefits of being a young company we can do stuff like this we don't have to you know manage a whole lot of tech debt so how do we actually define our features so this is very close to an actual example of how we would define a feature so to very high level you can think of an extract pulling some data a transformation transforming that data in some way and then the feature wrapping together some combination of extracts and transforms so features are built off of these versioned extracts and transforms so you can see that everything is versioned as a part of our folder structure within our repo and everything is built on these abstract base classes so extract transform and feature are all built off of these base classes that have testing frameworks that mean that as we add these in we're getting testing on top of that so we can make sure that everything is really accurate and also we can make sure that our features aren't changing underneath our feet which is a really really terrible thing if you've ever had that happen and because it's all just Python classes it's really flexible you know you can define constants you can do your own transformations I think actually here there's a transformation for email domain it's some custom transformation that's written into this particular feature which is selecting some attributes about a user and you could also you know loop over some transform if you want to apply it to a bunch of different subsets your data in some way so if anyone's ever used air floats it's kind of a lot like that where you can have a ton of flexibility about how you're defining these pipelines so within so here basically you can see it user attributes extract is an extract that goes and probably runs a query against your users table and pulls a bunch of columns so there are some huge some features this all gets fed into a pipeline or there's some data that gets all fed into a pipeline you can see column mapper probably that one just takes the user attributes selects a set of them and then returns that as a feature and and you can think of this kind of like the feature groups that Jim mentioned we call them feature families but you can see that there's here a custom one-off transform for email domains so this is to basically select the email domain that the user signed up with and then this is a chain of transformation so you can see first it takes you know the time that the user registered that converts it to a local timestamp selects some columns from there so it looks like day of month day from end of month a few other time features and then it adds a prefix to it for registration so this you know calculates some set of features it looks like about you know it might be like 15 depending on the attributes column and so then from your model you can just go and call this feature and pull all of these features for your users at some given point in time and I forgot to mention that there's some a bunch of flexible methods for merging joining concatenated data it's all it's all pretty flexible so the reason why we chose Python and is it's really approachable it's fast enough for our inference needs I think it's really nice actually and I think a lot of other feature stores were doing everything kind of in an aggregate level trying to you know pull a bunch of users at once everything here is based on an individual so you can make a request to the feature store for an individual feature for a user it keeps it really simple it's really easy for data scientists to go and write Python to find these features especially when you know you can basically go in and write some sequel for an extract and use a bunch of pre constructed transformations and everything is versioned which means it's really easy to manage all these features at our stage I think that's probably going to get more complicated down the road the transforms are all consistent they're tested and yeah so we can use you know different versions of features for different models if you already have a model in production that's using an old version of a feature then you can use that if you have a new model that you want to use the most up-to-date version you're not going to break your old model that's in production the transforms are reusable they're organized under filter map and reduce and then I mentioned that we have a testing framework which is great because it means our code works and our production models hopefully don't break so kind of reflecting back on where we are we have common feature transformation code that makes it really consistent between training development and inference we are kind of validating inputs where at least validating types and soon hopefully we'll be validating ranges and a bunch of other kind of you know pieces of data that we want to make sure remain consistent we're enabling training on much larger data sets because we're just caching everything so when you want to go and train on a million you know loans it's really easy to just go hit this cache and pull a million feature sets and that's more about enabling that there's other infrastructure challenges about training on much larger data sets obviously and yeah we can backfill historical features at specific points in time 100% of our data that's pretty exciting we're sharing model outputs we're not quite sharing model outputs as features but that's something that we're kind of evaluating right now and so learned features and we're restoring transform features at the point of inference so that we can then go and validate training we are starting to track metrics I think that there's a lot in here basically like monitoring and and you know validation and tracking or like always the last things you do but they're kind of the most important once you actually get it going so I think that there's a lot that we can do there and then everything is in dynamodb and we want to make it a lot more accessible to more of the company it's obviously accessible to the data science team so my big prediction is that you know there are going to be a bunch of young companies you know that are getting going and thinking about how to deploy ml infra early on and I really do believe that feature stores are going to be the centerpiece of everyone's machine learning infrastructure in three years I think it's going to be at the core and I think that it's you know that's obviously really exciting that we're doing our first meetup ever this is like a historic moment that's great so you know big thanks to Dave who was here Dave Bernthal Dennis Vander say Spencer Barton and ting ting who's right there and then the branch team is here too - yeah support me thank you and so thank you and then I think we can take questions or yeah okay you translate and plain English what this is purpose yeah in a womb context or credible yeah you just read absolutely okay what is the business context for backfilling features so I think that there are a lot of companies who will basically log features in production and then go back and use the log features and that's really great if you know your YouTube and you have a ridiculous number of visitors coming through your website your logging you know you go you create a new feature you log it for two weeks you train your production machine learning model that's great it's really really difficult if you make loans and those loans are shortest loans for 28 days we have loans that go all the way up to a year so if we want to create some feature and understand how it's going to impact our loans that are more than 100 days out who you know if we want to then go and log a feature and and try to understand how it's going to impact our model we have to wait for a hundred days you know well that loan well we collect the label for that that signal and so the business use case is just if you want to do great machine learning and you have lagged labels at the same time if you just want to you know have a much larger data set for your machine learning it can be really helpful to actually be able to go and create and you know create an entire training sample for a much larger data set and I think that a lot of companies have the ability to kind of recreate data you know there's some kind of vlogging of how data is changing but it's not perfect and so we really prioritize making it perfect just because it's so core to the success or business great okay yeah great talk thank you what you're doing is creating new training data yet you want to create the training data rather do you see that as a key feature of future stores or is it just kind of my compare using them it's because you're asking how much companies are getting new data you're getting your data coming in and you know the predictions we want a parallel and then you get you get your outcome maybe percentages on today's and it's not a key from chopping that you see in the feature store or is it just a prop I thought man you were yeah that's interesting so I mean I would say that the future store would not be useful to us if we couldn't do that you know and so maybe maybe that's a feature set that maybe it's a feature that if it's so poor to your business that it actually wouldn't be useful without it then yeah maybe it is I do think it is core I mean at least for us you know I think maybe other companies that's a lot less useful but even in those scenarios it's so beneficial to be able to go and backfill you know data and actually you know test your models in the past I think that there were even teams like the one you know the example at YouTube that I was mentioning there were teams that Airbnb when I was there where you know they did that because they could like you know search it or maybe is a great example it doesn't mean that it doesn't you know help them massively if they didn't have to do that right if you can go and backfill your your data and increase your training set your models are probably going to improve your online service your your your interaction your data collection yeah is that where other companies that sensor network or from share out some actions or other your system right well it's a it's going to be able to but maybe there should be no question I actually run it so the providers they can actually really optimize the infrastructure you can actually run it right what would a end solution look like to you I think an end-to-end solution for me would basically start at what raw data you're storing and and at you have a production machine learning model and it's monitoring everything that's happening all of the raw data that's coming in how its being transformed you know how it's being fed into the model understanding you know how the model is handling that like maybe you know individual inference level understandings of future importance is like really deep monitoring of that yeah I mean I think that cloud providers at some point they're going to provide this and I just don't think that right now you can you know jump into a cloud provider and get a real intense solution without piecing a bunch of stuff together [Music] AWS approaches here's when you can build and what connects us all the services kind of concerned what hmm I mean they're there they're a lot and which ones are great like what yeah I'm curious and like what pieces of that because there are SAS services for kind of every piece of that right so so what you're saying is that they're Assessors pieces but there isn't a player that's really yeah pretty much I mean I think that I like at the very least before you know like a year ago there was no future store and so that seems fought to me a you know eighty I think that 80% of all machine learning is just collecting data transforming it basically you know the first like the majority of what data scientists are engineers today who are doing machine learning say this is mostly what I do and this is pretty painful and so there was no solution for that and I think that solutions for that are going to really mature and solve you know all kinds of use cases in the future but to me that's the most important that's what I'm saying it's so core you know once I saw what it could do to everyone be I really you know feel like that is so core to how companies should do it all but there I mean there are good pieces that do these other you know model serving model monitoring there's a bunch of you know companies to do Ottoman all like those companies all have you know great solutions for very specific pieces it's just in my mind the remaining 20% of what it takes to actually get and then all across the line so yeah yeah any more questions then thank you [Applause]