How to Elm-ify Your ML

scale.bythebay.io: Oscar Boykin, How to Elm-ify Your ML

Recording: scale.bythebay.io: Oscar Boykin, How to Elm-ify Your ML

you yeah thanks so like if you if you actually came for for the Elm like you're gonna be really really sad like Alexi told me though like I needed to sex up my talk title and I'm really gonna talk about functional reactive programming type stuff but like a lot of people have learned about that from Elm so so that's the own part so spoiler sorry we don't get to have a pure script Elm fight like now we can do that later or something so my name is Oscar please follow me on the Twitter they need all the user active users they can get if you've been following the news you can email me at stripe sorry for the burn for my former tweet friends okay so okay so like what's machine learning all about machine learning is really simple it's about like you have like like matrices like this and it's like some beautiful ground truth about reality that you just like you come by somehow and you've got a bunch of columns and some rows and you have something you want to predict in this case I want to predict X so I've got some you know some 0 1 1 0 0 on these features and that's good evidently and then you know it can be bad so that's like that's how we start with our ml on like all we have to do is like I'll give you this and then what I'm gonna do is your job as a machine learning person is to produce some kind of function you know on the right hand side it might get in this example it's a tree many popular algorithms are tree like that we we take this over here and we somehow infer some generalizable function that's gonna work over the same domain of inputs on left hand side so if you like been hearing about machine learning and you think it's all hot like you probably think that this is cool or like and like that arrow is really like the sexy thing like how do we go from the matrix to the function that's the thing you get really super pumped about so and so I'm on a team called the machine learning infrastructure team at stripe so you think I would be like like worried about that but actually I'm not at all worried about problem so because like really we're worried about this problem he's like where does this matrix come from so that's actually turns out the hard part like if I just give you the matrix you can like download tensorflow or scikit-learn or anything and just throw it in there and it's like you know you can go win your cattle competition but it's getting that matrix that is a really hard problem especially at a company that's like operating with a large scale data or a diversity of sorts data sources so really what machine learning is really all about it is like basically akin to this kind of job where it's like you've got a bunch of serialized data laying all around probably and consistently serialized and you've got a like somehow get it all together collect it back and turn it into one of those pretty matrices and then like you know it's pretty easy after that like so that's what this talk is gonna be about so what are the key challenges that we had to face or we're still in fact facing so the main thing is that you know you have all these different data sources sometimes they're like you know God forbid mutable state and some like database some might be log data sources that get published under our Kafka queue some might be on the Kafka queue and put it on disk somewhere so how do we integrate all of those things together to produce that matrix of like different rows that we can train on so that's like a big big challenge so does anybody else does anyone else face that challenge in their place of business or in their homes okay the second one this is like one I could probably a geek out about it's probably already tipping my hand that I get more excited about the infrastructure side of it but it's at stripe a big problem is is is is a fraud problem we're trying to predict fraud like but you know or you know Twitter maybe they wanted it to predict clicks you know so you're trying to predict a click if you're not careful if you're really super if you're you'll do your job right you can include in that matrix information like about the future as it were you know because like you have all this data from the past you could accidentally leak back into it something that was like caused by the clique itself and then an algorithm can learn extremely well whether or not things would be clicked because of this like sneaking in information from the future so you need to make sure never ever to do that in a predictive domain like fraud or a clique prediction and that's like not easy it's like if you just give someone like go go to town with like you know spark or like MapReduce dataflow type programming on batch data on disk you have to be very careful to do these time aware joins so that I've joined this data as it would have been viewed at that moment in time this is a pain because you may never dealt with that pain before I'm not the only one you've dealt with that paper okay so people this is not you know I'm talking about so the next question is how do we make sure the data consistently with training is consistent with scoring so you might have the luxury of doing batch scoring and then it's not such a hard problem because you just run the same tool and batch to get the data to Train and then you can run that tool again and score and like life is easy but a lot of people such as stripe we want to make a decision as to whether or not to process a payment we need to do that like a moment that an API call is made it's really not very useful I mean it could be somewhat useful but it's not nearly as useful if we have to do that later or something we need to be able to do that in real time at the moment so that means the the models were trained like you know you know some period of time ago let's not say how long ago they were done with a batch with batch compute now if we inconsistently compute the data at real time like we're gonna like totally goof that up right we're gonna send the wrong inputs and that's bad news so the last thing is well if you solve this problem poorly step three step four becomes really hard how do we share code between the training phases and the scoring phases like if we have to if we have to write that code twice like we have a good chance of having a bug which increases the probability that we do have inconsistent data for training and scoring so those are the kind of problems we set out to solve on this and not not just be like my team the whole team that I work with been working on this for a while so it turns out like there there are some cool things in the functional programming community that are related to time so usually you might think that functional programming is about immutability that like off the top of your head seems like it's like anti evolution in time right you know cuz time things tend to mutate as time goes on but actually there's some really nice work on this and I don't really actually know all the whole history I've read some api's over the years some of the papers and I think this was kind of like like someone can correct me later if there was actually an earlier work than this but I did find it weird that it was so focused on animation when I dug this paper up but this was a paper by you know some well-known folks in the function from 1997 and they introduced a model where they have events and behaviors okay so and this is it's quite it was quite influential to I mean if you if you didn't know about Elm they had a notion of a signal if you've looked at this in the past in the in the earlier iterations of Elm I think the latest iteration has done away with that but their Elms signal is very much behavior it's very very similar to behavior so these are the main kind of methods re-expressed in like scala like syntax that you have available to these things so like an event is basically just a tagged like value with the timestamp associated with it and of course you can like transform like an event with time that's pretty clear just based on its structure now a behavior is something that you can ask about it at a certain time and get a value a total it's like a total function from time to a okay so that's what behavior is all about um we can pretty obviously lift any value into behavior like if I just want to be a constant look like one here's my behavior I'm gonna behave one from now on like that's a legit because you just ask later what what's what's the value of it it's like 1 so you could do that for anything but the cool thing is and this is where it becomes like more useful is that you can compose these in a somewhat interesting way and it's like deja vu all over again when you go to a freaking Kollek conference and you you have to like listen one more time do like like all the type classes but we're gonna do it again here we go so this is like one formulation of something like it's an applicative right so it's something weaker than a monad Mona would be too much so very often in life we actually want to restrict the power right so behavior is applicative I can combine two together and imagine doing some function with them and making a new one so what does this look like this is actually a very very natural thing signal is you know you know signal behavior feature might be another way to think about this so what does this have to do with machine learning so I'm gonna say like a feature so feature and machine learning is basically just some kind of generally time varying input into a model okay so just what mean by feature we could say input you could say signal whatever we're gonna use to our feature here so as time comes along like a feature has a value but like we for the purposes of the system we're gonna describe today those features can't change like at any moment time they change in reaction to some event happening so event happens here we go here's a one now feature has changed to take on a new value and you know and so on and so forth so that's the kind of story we want to present like I'm you can imagine and I'll draw this again a few more times in this talk that there's this you know curve going on and that's really what our what our feature will logically represent okay so back to our magical input or magical matrices that we kind of like somehow have to come up with if that's kind of our goal like are these we're gonna say that these are features that this is like a series of events that have been attached to a feature so like a feature here might have been you know BCDE and our event was a and so at some point in time a came along with the value of zero and it looked up the future and it got the rest of the row and that is how like that's our story for modelers I mean actually I didn't really draw it like this they'd probably be very sad about it and like they probably hate all this implicit of functor stuff anyway but you know I just kind of you know forced it on them you know but that's like basically the story right so we're using that feature abstraction to build these matrices and once you've got the matrix as I mentioned we've got a like solve problem right then you can throw at it scikit-learn tensorflow etc so what is it like look like so it's kind of vague we're gonna like look like go in a little bit more so here's like the Scala code of kind of what it looks like so we have basically two objects as I mentioned an event and a feature and somebody kind of told me and I kind of guess I agree with them that like event was maybe a bad name for the first thing because it's really representing like a like a series of events so it's like you know but whatever like imagine it is the full history of this event the event of click the click event so it has many inside but you can't do too much with it so the think of them things that pop out of Kafka or whatever at some point in time now this is basically the only thing you have actually you could have represented in terms of a fancy concatenate but you can map it on to you can get the time that event happened and change it and like maybe I need to decorate it and like if you know I can do some transformation at I'm aware I can expand one object into many or zero and then some of the rest of them kind of fall out of that I can merge two streams together if they're the same type so this is kind of a Combinator it's a it's you know it's it's a mono Wade right tomorrow because we can make an empty event stream like no events ever happened that's pretty simple and we can like merge streams together and this is one difference between some the classical literature on FRP and what we chose to do sometimes in FRP you say that each event screen can only have exactly one at most one item at each point in time in practice in a real system like there's no way to guarantee that like you may have many front-end machines that are consuming events and we relax that and we allow many items to appear at a point in time and a feature is relatively simple it's something that it's a well we can change the value and we can glue two of them together now I don't think I jumped up and down about this yet but I probably should have earlier features unlike what we saw in the paper where you have behaviors that have values in reality features are about something so we have features about a tweet we have features about a merchant you're gonna features about a user and so we explicitly expose that in the API that the K here is what is the feature about so notice that we can't really change what it's about once we make it we can change the value of the feature though we can like if you give me a feature well we'll see some examples in a moment so this is the basic API that we have and this is pretty much all our customers need to learn these are the people who are actually building models so in you know we got to get our type glasses on let's let's go so so event is actually a functor it's actually something slightly above a functor and I tried to get people and cats like it get into this but like you know it's like you can only you need people to water man but you know whatever so there's something above a functor but below like i'm on add or a blo applicative where that booth spark or spark scalding lists vectors this event they all share is the idea that you can kind of like they're like you can merge them together you can map them and you can expand one item into zero or more items so those there's a type class there that could be very useful in data engineering because we could write code against it that would apply to many of these things but but so far I haven't sold that yet so if you're interested in that I dig up that cat's PR and say yeah I get it now Oscar we actually probably do need that type class anyway this one is applicative so we can join two of them together so what do these mean in practice so applicative is like a functor with a little bit or and functor it's just something in Scala world that we can like you know map on we can definitely map on this thing we can map the second con we can't change what it's about feature about me is still about me after we change it but we can change what it might like the value it might have we could say like you know one feature about me is like how many like pets I have but I could change that could map that into do I have zero or more pets right that's I don't need to do any fancy computation there I could just say map like if you know greater than zero like true all its false right that's not but what's really cool is the combinators right like so how can we glue together features with events or events with features or events with events and you have like a little bounded set of things you can do to glue these things together and then it gets pretty cool so the cool thing here is that time aware joint problem you just can't even express bad code so that's like the cool thing so there's gonna look up notion I can make a new event from an old event and a feature but I noticed time doesn't show up here so I don't say look it up at like last Tuesday like the system only can express that you can only look up a feature as of like kind of now from the point of view of the event as it happens and there's no like thing that a user has to do or can even express that can get this wrong so that's like I think one of the best features that the system has probably the best feature the other thing is like well where these features come from I mean I don't know if you look to the previous page I hope you were asking that because like you know constant feature is pretty boring if you can only make constant features and glue them together and map them that doesn't sound like you'd be doing too fancy machine learning here around so so the cool things that you can do to create features are a very common one is I want the latest value of an event so an example of that is like a tweet might have you know we might want to feature about a tweet and we could say the last time it has been viewed who is the last user who used it that's a very simple thing to express you have a series of events of like viewing tweets you the tweet ID give the user that viewed them and now you can see who is the last person at any given time to view a particular tweet so that you just expressed that this way that's all you can say about it like you don't you can't do more you can't I mean that's just what it has the next thing is and you know I like I can't stop forcing people to express computations in terms of mono AIDS and I wasn't about to stop and my new position is stripe so the other thing that you can do to create features is to take a series of events that have some value V and just sum them up in fact the original FRP paper has a notion of this as well they have an integrating feature that is kind of nice for like mouse movements and things like this that they were drawing which is pretty cool again we already mentioned product the ability to glue two things together that's another Combinator that you have and I didn't call it out again but you can also combine events with themselves so you have this like two by two matrix events can be combined with events features can be combined with features and events can be confined with uh with features in different ways any questions about this cool so also people love the free right so last year was the year free I think Rob Norris said like you know so we got to get our free on so actually in fact like when we were building the system you know people were impressed they're like wow the basics of your system you were able to get it working for us to use so fast that was pretty cool and the secret is we didn't actually do anything you know the secret is the implementation is that just these are just these free things that I described like so our implementation doesn't do anything it just describes the business logic so you actually just build up a like they're sealed cases there's still traits are actually there may be abstract trade whatever anyways they're sealed people can't subclass these and make their own or like program them in fancy ways they just can't do that um so you it's just a way of telling me what like describing your logic that's all it is so that's all the users can do they just build up this free thing now they can't like talk about anything outside of the world right so they can do we talk about how to I don't think I'm the only way you can get into it is this function here this is the only external like way to get in and all it does is say like I have a source of type a and like I give it an identifier but that doesn't like give me the data that's just like a thing that you can have and like make graphs with and you know traverse and stuff like that you know so that was it so a man is like you're free functor and fan the feature is just a free applicative for the most part and we have these special combinators you know all the things that you see they're just case classes you know how do you implement like a product and feature it's no problem case class feat product feature boom you're done you know it's like one line of code map same thing how do you implement map feature you got a case class with map with a function you're done it's it's easy so once we've got that we can well you know I don't know like we will talk about we're going to do in the end but what this gives us is our feature engineers are like 100% dealing with business logic there's like zero percent systems like concepts in this API so they can define with like with zero dependencies but if they wanted to just define the business logic of creating very very complex features and that's pretty cool now once we've got that we can have various interpreters and compilers to like compute them in any number of ways but we've constrained the system so much that it's very simple for us to do that basically we just have to someone has to provide us the right inputs and then we have like you know a rich enough palette for you to make very very cool features but constrained enough for systems people who like to geek out to actually like schedule this effectively by the way I think this is like one of the biggest deals in building systems of abstraction is like finding like how much can you tightly limit things so that someone can barely express what they need but the system's people can go over and actually efficiently execute it and I think that's like it's a it's a really fun and been a productive productive area for people who like systems and machine so many examples of this so let me just talk a little bit about like what these things do like you know some of you must be in the audience and not really wanting to talk about applicative functors and everything so now I'm going to restate some of these actions in terms of the matrix that you might like if you're thinking about future engineering they might make more sense so what map does is all it does is if you have some column if you have some way of producing a new column or columns from the old column that is mad so if you're thinking about like some system that you might have worked with to produce you know that's what map gives you very very common operation not weird that we have a functor for this thing that's cool I mean a canonical example might be here you have a feature of merchants and total charge count and we it might also have a total charge amount well average is pretty trivial to drive from that right I could just divide the two and I can produce that feature by doing a map and dividing it right great feature dot product glues columns together so you might have one matrix and another matrix and the third and is that's exactly what we're going to build up when we're finally giving a big fat wide matrix to some training system and that's what product gives you so one way is that someone may have a really cool feature that computes the total charges amount in u.s. dollars and someone else may have some fancy hyper log-log powered feature that generates the unique cards that have been used with this merchant and you may say hey actually I think my model will perform even better if I can borrow your feature and and put it all in one big model and so the to the user it's really simple you call dot product you say now give me some training data and it just all pops out and it's like totally magical it's great lookup is where we do some cool stuff so now I don't know if you notice there's a little bit of sleight of hand in the original papers on FRP you could ask things the behavior for what is the value at a given time we actually don't expose this to the user so it's kind of weird it's like you're like it's a little bit like the the Monad trick you can get into the moon app but you can't get out right you know it's like you know there's like like not yeah so the feature is the same way we let you get into features we don't actually ask you like there's no way that you can say like okay but it next Tuesday or last Wednesday what was that feature value that is not a method on this thing it does not exist so you have to understand that conceptually it's providing you something it's providing a model of how to compute but it's not giving you like the ability to reach in there's no unsafe IO here you know so um so you've got the feature it conceptually exists but maybe we're never going to compute it maybe we're just like very lazy about it and we're not we don't need to compute all this but these events can come in at certain times and at those times what we tell you is that we could produce a new event scream that would tell you the events at those times so that's what that look up did look up gave you a vent to feature back to event so we can learn how to schedule that later but that's what's going on so this is what's very very tedious to do by hand this is that time aware joint if you have a big pile of data to look up II you had to go back and say okay let me get all the data up to that point do you what would the value of the feature have been this is a real pain nobody wants to have to do with this and it's really easy to screw up this is like the big value add this is the takeaway of this system so I called this property like temporal consistency you know it's like I don't know what kind of time travel movies you like but I like the ones where they loop back and like everything like still make sense and you don't have to like worry about it and like you know so like we're ready to go for modeling systems like that like that we can't do the bill & ted's type but uh we could do or back to the future not gonna work with this feature system so here's an example and I'm like I'm sorry I'm like a bad bad guy like it's like I like abstractions and then I know a lot of people are like yeah like I put up let me see an example and like this is where I'm gonna fall down and this was taken from one of our unit tests that like I think Eric Austin wrote it cuz he's even better at like like at least he could he has a dog and you know feelings for their animals so like here you go so this is actually from a unit test so we've got events which are barks jumps and howls and we can like make a feature has a given dog barked or not so we make that by we map the barks we can sync the name of the barker I actually didn't include the case classes of those those three objects bark jumping Hal but they have a name associated with them in like some metadata so we're going to just take the name and then the value I'm just gonna throw it away entirely cuz I just want unit like I want to see what happens and then I'm map I take the latest value and I'm gonna get an option if you remember that Combinator the latest Combinator gives you an option of the value and all I care about is it defined or not so that's like a real kind of feature you know cool we can take the average volume of the of the Howl's embarks like we're going to combine them together so here we exercise some and by the way I lied to you earlier when I said that the methods names so I just I cats off I'd the names but actually internally I'll jab right off I'd the names and we use some for combine and zip for a product but like I don't know if whatever so some is actually like a mono you're you're combining with the mono would and I've talked about this in the past I turns out Shocker averaging can be a mono it so we have this thing already we can average the value of these things and combine them together and we get the average value and here's another way of computing that same one which is actually pedagogically clearer but from the numerical stability point of view worse that we could have just computed the average of the the average value of the deaths of the howls and the average value of the Bark's product them together and then combined so this is like two ways you could have done it because it's a mono it doesn't matter which way you do it so and this is a just a dumb unit test it actually lives in our repo and sorry that I don't have a sexier example but that's like the challenge for you like what cool features could you make so how do you actually run these things so I've talked about very abstract I said I didn't actually implement anything which is like you know maybe I understand myself cuz it's all free thing actually what I said was that was really easy to get the free structure and then because I put this talk in I'm like yeah problem will I talk about it in November and we'll have all this like great like operational war stories and I'm like you know as of like last night dark reads were actually live so the actually the back end turned out to be a massive pain in the ass so we have several back ends what the first one that we met made was a little interpreter that can schedule this too like a scholar stream so this I am alighting this code here of converting the event into a stream but conceptually you can imagine I should be able to do that so I can give a lazy stream with timestamps and events in the order in which they happen and then if you give me a feature and a key that you want to look up and a timestamp I should be able to give you the value and you know if you give me all the inputs as streams I should be able to do this and you know it's pretty straightforward how you do it you just kind of like case match on those sealed classes and go through and do the thing so that was the first one we built on which we can write unit test locally our users can like you know make sure they're exercising their business logic with this that actually has some use our second back-end that we built for this was a MapReduce like back-end so because of the structure of these of the combinators that we have we can make it look like basically and when I say MapReduce I mean like the logical notion of MapReduce so it's an another ast an intermediate representation that has like series of mapping operations followed by reproducing operations so that's like the model that spark has or flink has or any kind of data flow system has so we have this intermediate representation like that and we do the same similar kind of thing we actually get a little fancier there it turns out with the structure of these things you can always kind of imagine that there's a free functor hanging on the back of these things and that's kind of nice to like discard that free functor and like like apply that at the very last minute so we did some tricks of like you can convert features to unmapped features this is like always something like the unmapped part along with its mapped part that turned out to be useful for the copy for the building the compiler but that's like their second back in and our third back-end was useful for real time so then we had a third and that we is like a push back end so in the push back end we're planning it on to two objects one thing that we call a multi consumer it can consume from different sources you say source a just happened and here's the value for a or source the house just happened and here is an example how and I can push it in and it's going to return me nothing I love it when I can work nothings into type signatures and it's gone it's like one of my favorite things you know and I also will give you a way to read that a real time you know now so you give me a key and a timestamp and I'll give you the best answer for what the value is and it's the same kind of trick I don't really know what the takeaway of the slide was except for the fact that we do you know traverse is always the answer you know so if you have a question its Traverse so that's pretty much it so you know what we have here is I think a very minimal and principled API for feature engineers in the principled nature of it is not just because we're like dogmatic you know f-p jerks you know it's not just because of that that's like aside in addition to us being dogmatic happy jerks it also means that the backend system has a lot of opportunities to optimize this and plan this in many ways so we have three or four like the examples I gave when I see the MapReduce back in we actually can run it on we run it on scalding and another system called summing bird that's kind of similar but it would be like probably a hundred lines of code to like run it on spark maybe maybe two hundred we just don't do a lot of it it'd be very easy and that's because of how can screen this model is the next thing it's really awesome I think it's like really important is that it 100% separates the business logic from the implementation details like just there's just no way to think about like even expressing things like some forms of expressing things the wrong way you can still write garbage features that aren't very useful or something but like you can't do a pretty wide class of errors that can come up and I mean this is maybe a restatement of it it frees the engineers features users from having to worry about temporal consistency because it's just it's not possible to Express like inconsistent temporally data in the system just weave like that's like our favorite trick in FP right like make errors unexpressable like so we I think we've actually accomplished that for this problem in this case so that's all I've got I think I've got some time for questions if anyone has any [Music] right yeah that's a good question so if you go back and you look at what we the API that we give to people to come to produce features out of events there's a little bit limited like this one's pretty straightforward and like it's it's very useful it comes up but the second one you require some way to combine values as the mana weight but the most common thing that doesn't immediately grab people as being a mono it is windowing operation so you might want to take the maximum value over a window and it's non-trivial to express those some of those kind of calculations as models so let me let me answer it like there's there's several answers to this let me get the last one first your question do we plan to support that I think we may plan to support that I'm still coming off of my experience and biases from engineering systems at Twitter where like data rate was like really like you're constantly drinking from a firehose so we were really struggling with that transactional data turns out to be much smaller and I think a lot of people are dealing with smaller data so you don't you could kind of like relax things a little bit so you can imagine a more general operation here than a mono in that like my it might still give us a lot of the benefits and we might be a we might we might think about doing that the reason why we want to like that we don't want to reach for that right away is the ability to combine like this kind of lambda architecture systems that a lot of people have built of combining batch and online stuff becomes very very easy and tractable if you have a mono it and it becomes very very difficult when you have general windows so that's one answer the next answer is there are many things that are window like that are mono AIDS so it one thing is that you just like it's wasteful but you could keep a the value V could be like a fat memory that remembers the last like week or something and like it means that it expensive to go fetch an update but it can it can be tractable the other thing is an exponential moving average turns out to be efficient but also a mono eight and several exponential moving averages can approximate a window very well for many things so that's kind of the space of answers we give people when they always ask and maybe two I'm glad it was the first question all the future engineers always ask this and we that's what I tell them I don't know how how happy they are I would like to ask to accelerate a bit about compilers into the production code because I assume that's a that's a good question so we actually made a mistake when we were implementing this that like I'm a little bit embarrassed about which maybe I shouldn't even I'm just some idiot like anybody so I can tell you what a dumb guy I am okay it's fine so we were like hey we've already implemented things similar to this we could just like plan this feature framework to be like something that's more constrained on top of a system that I built before called summing bird and like that can be cool but actually it turns out that was false because this gives you look up is actually it gives a consistency model that if that isn't like what people expect it's gonna be inconsistent with the data and systems like likes like storm which is just like tuples racing through systems don't have this notion that they can like kind of put a barrier someplace and make sure a happens before B so we we built the system we built it on top of like something bird and storm and we first of all we didn't like the latency numbers and for the system that we were prototyping it for we want latency on the order of like you know we had a we had a budget of like something like you know it's if you're running turns actually if you if anyone uses stripe you might notice that like you know you're willing to wait a second for a transaction to happen so the transactions windows are pretty long so we were given like it would be fine if it took 300 milliseconds on the outside even but before things came in it turned out like we had trouble meeting that in with storm by itself and it was like inconsistent because of this racing issue of tuples racing through like a data flow without synchronization barriers is like not what people actually want so we then had two minutes part of the reason we were somewhat delayed we decided to build this directly as a Kafka consumer so the the third one the third platform that I mentioned that resists it basically just sits in a loop read some events from Kafka and then like like transactionally like all-or-nothing pushes you through the whole feature now that can be act extremely fast so like four very simple features it might be like simple features meaning maybe you only have three round-trips of like talking to different like like materialized views a part of the future that might be like 10 15 milliseconds and for like really really complicated features that have more like 15 or something like a larger a much much larger graph it's still like less than 50 milliseconds for the cases that we see and that's actually been really really fine for us so I my regret is that we didn't just go directly to like on top of coffers and I wonder now if we can throw away the mono ad restriction and just go with like try to get it to exactly one system only built on Kafka that might actually work for us or at least be a possible back-end so I think ken had a question I wanna make sure you had a chance to ask it I did you have a question right that's a good question yeah I mean worried about it I mean I'm worried about it I mean I don't know but I think the main problem no we're the main problem you're trying to do is combine many sources together in one consistent view but each of those sources they still might be very dirty or ugly and we haven't tackled that part of the problem I really can't show you any more than the code that I got there sorry I probably can't there are there are some really fancy stuff but but like I mean of course like we do fraud we do fraud models right and like it's really important for us probably not to broadcast anything that goes into like what our fraud features so I really can't do that but I'm available please come talk to me thank you for attending my talk thank you to my team Travis Brown is here he's worked very hard on this Eric Osteen worked very hard on this and Sam Richie he worked he worked some [Applause]