SF Scala: Oscar Boykin, Reproducible Machine Learning with Functional Programming
Recording: SF Scala: Oscar Boykin, Reproducible Machine Learning with Functional Programming
[Music] so Thank You Remy Thank You Alexi for hosting this this event and it's so good Thank You Twitter on all my former colleagues whom I love so dearly it's so it's so great to see you all and be back here so unfortunately well I mean you know Alexi wanted me to talk a little bit about basil but I wasn't really gonna talk about basil with all fear who can tell you so much more about it and I would embarrass myself so what I'll do instead is give a brief sales pitch for why basil is arguably one of the most exciting pieces of technology that's happening right now in the open source world that I that I'm and I'm a big big booster of and I really want to see it succeed so we do use basil at stripe and what is basil basil is a build tool so I think the functional programmers in the future will look back on these days and think that the way that we're building code right now is as like you know primitive and grotesque as the days before code review or CI and each of us thinks like Oh back in the days before for code review I would have done code review those people were terrible or unit testing I would have done unit testing but actually people developed software for a long time without doing unit testing and then now we sneer at them and functional programmers sadly can be a little bit of an arrogant Bunch they may even look down on some of these people but what basil is trying to sell you on is the idea that your build should be a pure function and like you know some from the south and I've had half a drink so you know it's like let's uh that's like you know we need to you know get religion about that idea I think that functional programmers should understand they should be like hallelujah on that that build should totally be a pure function if I come with my sources if I come with my dependencies the output of that should be the same thing today as it was tomorrow as the following day it's kind of crazy that we would accept anything otherwise now people will accept otherwise sometimes say oh it's hard to do that or it will take too long to do that or you know they'll have a long list of why it you know it can't be done but it can be done and so therefore in five years when this idea has won and everyone knows it and it's like it's like you know of course you have reproducible builds and of course you do unit testing and your code coverage you want to look back and say that you are one of the people who understood that you know way back before everyone else and you helped usher it in so that's why you should be excited about elfs talk he's gonna tell you probably about caching and about how basil is very programmable but to me I think that that that basil really has the opportunity to be a Linux like project that we stopped like like having these parochial build tools you know SBT here maven there may be some weird package manager for my new fancy language over here and we just say look here's one way that we have do we describe functions from source into the outputs and I can get bisect rebuild that and get the same thing in the past I don't have weird things that your laptop builds at one dawei but ci builds it another etc so it's hugely important and people who don't get it yet they're going to get it and you want to get it early so that's about all I'm going to say about the buzz about basil we're using it a stripe we're eighty-five percent happy with it and I'm use the 15 percent to give give all anytime I have an opportunity to but but that 85 percent is growing and it's going to be 90 percent and 100 percent soon so it's very exciting so that's the basis of work sure my talk so I'm going to talk a little bit very similar talk only slightly different about about feature engineering at stripe so if you are interested in machine learning you are often thinking about you want to make a model of some thing like we want to model fraud and if you download tensorflow or if you download scikit-learn you can treat a model very easily now it's a-you know a lot a lot of programmers are doing and the first thing you're probably going to do is give it some something like this some table of data so here's a bunch of columns and here's a bunch of rows and maybe they have a bunch of bits and you want to produce some label so at if you work on ads you might be wanting to predict whether or not an ad will be clicked if you show it to a particular person in a particular context if you work on fraud you might want to answer the question is this charge a going to be disputed in the future you might want to know if you're moving money around is this a bad actor moving money or is it actually the person who should be moving the money there's lots and lots of questions if you're you know driving a car you might want to be you know predicting a regression of like you know where the road is going to be and you know 100 milliseconds or something I don't know but there's a lot of excitement about what tools to use to predict these what kinds of models to use to predict these and you know or even you know what kind of architectures to to run these models on but we really have a really challenging problem is where does this data where does this matrix come from and that matrix in most organizations is very difficult to produce you may have some database to some place you might want to go hit the internet to some service you might have a geoip table that map's use to IPS but that's changing and dynamic you may have you know you know any number of like data sources you want the weaves together how do you do that and produce this matrix because this matrix is literally cut CSV you get download or you can put it like it's nothing more magical than that so how do we turn those databases and events streams and everything into into these tables that's what I'm going to be talking about and how how can we solve that with functional programming so you know this is just a model so we often use XG boost random forests you know we're you know I'm just a simple country boy from North Carolina so I don't do these fancy models that maybe Google but you know where we're from you know something simple like this would be fine as long as we have this matrix so let's uh let's figure out how do we get this matrix so here's Larry David expressing confusion so by the way please interrupt me at any time with questions I think it would be more fun if you want to steer the conversation in any particular direction or if you want to drill down and say I'm not really interesting tell me really how something is actually working and I'll try to tell you more other than like telling you what features we actually use to prevent fraud for reasons which I hope are obvious so this is what feature engineering is all about like you know a bunch of product engineers are all over the place you know and everything up and then your job as like a data scientist or machine learning engineer is to collect all this stuff over here and turn it back into that table so that's your job it you know it sounds exciting apparently all the kids want to do it but this is mostly what it's all about okay so what are our key challenges that we want to do so functional programming is often about like the exciting thing about functional programs the reason we don't care about it other than they kind of get a pseudo and quasi religious aspects to it are the fact that you can with an architecture rule out certain classes of errors so immutability allows you to rule out certain classes of you know race condition type errors where you know maybe two threads are mutating something at the same time can't happen because they're not immutable things like this you were just rule certain classes of errors out completely so that's one thing about functional programming that's cool so if this is a functional programming system for machine learning what what is it going to try to do how can it like kind of like rule out a class of error so the contribution of this work if we were you know if I still was in academic when we were writing papers we're that we are able to like rule out a class of errors that you might call label leakage and a label leakage error is really bad a machine learning suppose you are trying someone told me that exists is the example of this in Salesforce at Salesforce they have user enter data and I might be going to enter my data so so maybe I'm trying to predict if I'm going to be able to close a sales lead it seems like an important thing to do for a lot of salespeople if I know that I closed them then I get lazy and stop filling in the rest of the data and I just said yeah close them right so machine learning what does it do it's very clever it says oh I got it if a lot of the columns are empty you're gonna close them you know this is an actual problem it is label leakage because that label got applied to data like in a kind of sin that information back in time in fact missing data when it the firt Row was first created is not predictive that you're gonna close them but missing data like you know that was filled out with future knowledge that is predictive obviously that's label leakage the label kind of leaked back in time to what you're trying to predict this can also happen in trading system algorithmic trading systems you train it on data you know if it went up or down if you somehow have any function that can somehow loop back around and encode that variable you're trying to predict and that get into that matrix in an earlier row then it tells you a lie it can learn very effectively that you know you know the label that you're trying to do that's what we're trying with that's what this talk is going to do I'm gonna try to convince you that we're gonna rule out a whole class of label leakage errors and that makes it much easier to a feature engineer and produce models that work without getting some surprises that they work in training but not in evaluation okay so to a lesser degree we do a couple of other things which are cool but they're like I don't know I think it's less cool than this one thing is that we're able to describe these features in a way that's consistent between training and scoring scoring happens in real time and the events come in and in milliseconds we need to rebuild the features as they exist and then score them so that's going to necessarily be a different system that produces that big matrix for producing the model in the first place if the if the data scientist has to write one batch of code over here to produce the training data but another batch of code to in production this is an opportunity for another error right because they might mismatch they might actually like have like systematic bias not like not like random bias and you can really have a bad day we rule that out so these are kind of the last thing is like this is kind of cool it's related to this is that we're going to produce a historical value at any point that we would have seen at the moment that we were scoring the data so it's impossible to express some like ability to look in the future we take time out of the picture so there's no time in there the system owns this notion of time you can't like as a user talked about like you know one feature is like you know what the Dow will be like you know on the you know April 1st there's no notion of time you can't like look up in time there's you just can't express it so that rules it out so we'll see how do we do this ok those are the kinds of challenges were trying to solve so it turns out in the functional programming community they've talked about this notion of functional reactive programming and it dates back I think this is the first page paper to 1997 I think two big names in the academic literature on functional react on functional programming but the idea is that you have two concepts an event and a future and in a behavior I just gave the whole game way but anyways you have an event and a behavior and the behavior is this kind of time varying thing that can change in reaction to events so that's the reaction part so it's functional because we have we you know it's there's no mutation we have referential transparency we have types everything's nice but you know we can lift a value into behavior behavior is kind of a weird name so but we we we do something very similar we so we learned from this well we learned from this we learned from Elm which is a nice programming language that they talked about functional reactive programming for user interfaces we also learned from there's kind of this reactive programming kind of Marketing pitch that was was done but all these kind of ideas are sharing a lot of similar kinds of concepts so we have some of the the standard functional programming concepts here we can take an event and we can transform it so it's a functor if you're into function programming it's the map and Map Reduce so we can map our events the reduce in Map Reduce is we can reduce our events and turn them into behaviors I don't actually have the really fancy function that does that here the other concept that we have in that are really nice we talk about monads if you're interested in functional programming lot you probably heard this or at least seen jokes every if you're not a program you at least know you're supposed to maybe laugh when you hear someone say monad or there's something related to burritos somehow or whatever but there's something weaker than omona I've got an applicative and that's when you have like two of the same kind of containers and you can glue them together and make a single container like if you give me two lists I can put them together with a function that combines each element wise and make a new list or I could do that with you know so many things I could do that with an optional value I could do it with a map I could do it containers are very common examples but you can do these with parsers there's lots of interesting example command-line parsers recursive descent parsers there's all sorts of different things that can fit into this model so this abstraction becomes very very nice so what does this have to do with machine learning so rather than behaviors we're interested in a feature so what is a feature so if you're not working in the machine learning community we use this term to refer to like a you know like a facial feature or something but it's any input into a model so that is what a feature is it's just a dingus or whatever you want to call it but like that's just the term we use so a feature really is you can think of it as a real number but it's some time varying thing like you know it might be you know how many times is the user clicked on like this person's tweets it might be how many charges has merchants run-through stripe for Google it might be how many times is this person searched for this query how many times when they when you search with this query do you click a page that also contains this word things like this any kinds of like like predicates or scalars or very simple examples of what can be features but features critically change in time and that's very challenging part of them like how many clicks Lady Gaga has had on her tweets today or ever that's a time varying thing it's changing all the time and if we're going to use it to predict something we had to know what it would have been when the when the thing when the when it was we were trying to predict it right we don't just want to know what it is now so we have this time axis and the feature has a value and then an event happens and you know it's like I used to be a physicist like you know it's a little bit like there's like you know here's what the particle comes in bounces off the feature to flex it up to some new value so there can only be like changes between the features are only caused by events what is an event event is I mean if you want to be if you're if you're an engineer an event is some record there's on a Kafka queue it's as simple as that like that is what an event is that you know we don't have to look further than that so that's our vision of what a feature is and so if we have something like this it sounds like we maybe we have so we've got some traction we've got a relationship between events and those are something I know those are things that come on Kafka queues I know where those are I know the team that can operate Kafka we're gonna call and I don't know something with functions and I can turn it into these variables that I can feed the tensor flow so it almost feels like I'm getting close to the full loop of what I'm trying to do in my job in order to get a paycheck and continue to exist on Maui right any questions yes no they do they do they do change over time they change over time yeah when and the event happens yes sorry the question was I said features well I think he's gonna say more but but he said you said features don't change over time but I think he wants to clarify but I said that they do change but whoa yes so this is kind of like on dots or you know some kind of assumption that we're going to make that like we are going to you can imagine any kind of feature that like just some somehow magically intrinsically changes but I believe in a physical world so if it changes it must have been caused by something and there are only causes for changes are events this is kind of like a design decision yep so a window function can be like yeah so that's a that's an interesting picture keep that in your mind so the question was what about a window function which is some kind of like like convolution or integral of this thing is like a natural way to describe it so think about what we'll come back to that later well let's put a pin in that get a concept like what kind of things do we do by the way what I'm doing yes oh there's a great question is it like are you trying are you planning to apply a TTL A+ or temporal logic and in fact we have been playing with with that only a little bit but we're still at the Hobby stage so I have nothing to tell you but but I do find it very interesting I think it can be interesting mostly to prove the correctness of the systems that implement this I think but TLA plus if you're not familiar with it it's getting some increased attention has been used it's a model checking language it's a way that you can kind of write computer aided proofs of certain system properties that you might want so that could be very useful for us to probably to prove the correctness of an implementation that would would run the system okay so so what we want to do is we want to take our you know you know this our well let's see so the key thing here that you might find limiting and by the way my whole career as an infrastructure person is and and again back to basil if you can find these limiting abstractions around which like once you've made this limitation you can go forward and you know leverage quite a lot in basil it will be that like things can be pure functions so we can cache them and we can get a lot of nice things about them in this model it's that all features change as a result of an event and events themselves are immutable values we get to like derive a lot of systems benefits downstream of that so you know one thing that we're going to do is that we're gonna allow an event to look up a feature value that's something that's very natural I need to be able to say if like the event just happened that the user wants to run a credit card charge okay before I run that credit card charge I want to go and say how many charges has this user had today with this credit card number that number should be the value when I go to train my model it had better be the value that we would have seen at that moment it shouldn't be the value in the future or when I'm training that's irrelevant it doesn't matter when I train it matter so I need to be able to away to describe when this event came in which is user charge lookup use the user has some value associated with it which it might be a user ID how many charges have they made so we're gonna allow that user event to read a row out of any of these features but we're only gonna allow it to happen at the time that the event happened not in the past not in the future like now so there's like this implicit now that hangs along with everything they're always this now is relative to some immutable event not to a clock on your wrist or your laptop's clock or anything like that right so that's a concept that we're gonna use so what does the code look like it actually is very very simple so like you know the idea is you want to make these abstractions that like limit you enough that we can get a handle on but are powerful enough to use and so far this appears to be one for our team so there's two if I hope that people this is the SF Scala meetup so if you don't like Scala like sorry Kirk whatever you know just screw you you know you got you got a got a look at Scala on the side sorry off you probably doesn't like Scala very much but we have to events we have to a two types we have the event type and we have the feature type the event is just like it's like a list but it's like a list of things that have times associated with them right so that's one way you can make it instead of list of a it could be a sorry event of a it could have been list of a and feature is a little bit like map but it also has a time associated with it it's a map that each of the values has a time stamp in some sense and so rather than in each key you you can't look up a key on this on this map you come with a key and a time and I'll tell you the value that would have been less than or equal to that time stamp not in the future of so like the greatest least upper bound I guess of that of that time stamp or greatest lower bound I don't which one anyways one of these so there's a couple of things we can do we can transform these much like we can transform Maps sorry lists so everybody uses Scala or they even maybe they're you know you know impoverished and can only afford Java but even poor Java programmers know about like the stream use the API and they can do maps now and they can have lambdas and everything so you can map an event from if I have an event of you know that has my that I'm about to make a charge and it has my last name on it I could map that event and did you something that had like the first letter of my last name that would be easy to do right we can just discard that information right so that's kind of what what event can do or I could throw away some events I might be watching all the events that come into stripe but that event is not interesting to me because it's a test of it actually stripe has an API and you can say test mode equals true and those aren't real events so I want to be able to filter and throw those out so filter is very useful if I have two events streams of the same type I can concatenate them together much like a list if you give me a list of it and another list events I can concatenate them together no problem so events have the same kind of behavior this is why people geeked out about functional programming because once you think in abstract terms this behaves very similar to many other things you've seen before which are very there's like a set of like type classes you can associate with them they're you know they're functors they're you know functor filters if you use the cat's api there's lots of other kinds of things you can associate with them it's nice but the feature is more restrictive so we're not going to let you to do too much as and notice time never appears here I can't actually ask a feature like what is the value at a particular time this is how I prevent label leakage it is not even in the API there is no way to express this okay so I can change the feature value but not the feature key or if you give me two features feature of V and W I could glue them together damn it I had a type it should have that should have been VW but now I pulled a fast one here I realize I haven't discovered what this case is and a feature is always about something like how many charges has Oscar made how tall is Oscar how much money did you know lyft you know charged yesterday they're always about something and much like the the nice thing about Map Reduce before before the Google approach of it I think I I don't know I believe it was the Google's contribution to say we don't just have a map function and reduce function we can also parallel eyes with keys so we could imperil l reduce many keys at the same time so we can get parallelism on the map side we can partition by keys and get parallelism on the reduce side as well and so in the same way you're going to be able to architectural II get a lot of the same benefits so the key here is the thing you're about it might be a user ID it might be a merchant ID for us it might be a credit card it might be a geolocation if you're trying to predict where an X like you know event is going to be but your K is what your feature is about okay and that you can't change I can't map a feature and make it about something else I can't map the key part to make a feature about me about all fit doesn't make any sense you're not allowed to make that mistake the feature about me is just about me like there's no operation on this object that allows you to change it in that way so again this is another class of error that can be removed from like like a nonsensical concept is not allowed right so back to the functional programming this thing is a functor it's not a monad it's not an applicative this thing is the duplicative it's not a monad it's we give that a monad so if you're interested in those kinds of things so we have a very constrained set of things but in the previous slide we could only turn events into events or features into features so here's where the magic comes we can take an event and a feature and get up and attach the feature value to an event so this is how we do these lookups imagine like the event was racing through your Kafka queue and it went over to your Redis store and read some data and attached it back to the Kafka queue and went back through it might have done that or it might have been a MapReduce job that was looking at it all in batch and all the data was lined up but it found the correct value in time not the value in the future not the value in the past we're not gonna even allow the user to try to do that join you won't have the chance to do label leakage we're going to compile it in a way that that's there's only the right way to do it okay so that's one this is a very powerful primitive and then the other thing is that we can take an event and we can say we could just say what is the latest value of this event that is a feature so as this event is changing in time I could say what is it now what is it now what is it now so that's just a primitive that we can do we can lift an event into a feature so what is the value that the feature has so if you have an event with a key and a value the event the value has to be an option of P because it might not have happened yet what if I filter all the events out nothing's come remember a feature has a value at every moment in time from minus infinity to positive infinity so the latest value has to be optional or if you're a dirty Java programmer it can be no ok so anyway the other thing here back to the question of windowing and talk to me after the talk we like like one of my hobbies in life is forcing everybody to write their computations as mono AIDS so we actually only allow you to some to do that integral so an integral is a sum and a sum is like a mono aid so you can do this windowing but you have to express it as a mono and we have some nice libraries that allow people to do that so some mono aids are more efficient than others to express this way but there's a lot of nice reasons why we want that associativity so remember again associativity amano it is just something that's associative and it also has an empty you associated with it and the associativity we want because we want to leverage some more tricks and MapReduce and systems programming so again like the great trick of systems programming is finding little bottlenecks that you could push everybody through that they can get their work done but if you know if they're going through that bottleneck you can also do some great systems optimizations that's the main trick of abstraction and like systems design in my view so the other Combinator that we have we can combine two features together so we have feature with feature event to feature an event and feature to event notice we don't have feature to event which is very interesting and our users like often get mad at me but just because I'm perverse and I have no great reason to disallow it I like keep not allowing it no the reason why we don't allow feature to event is it would require us to fully embrace that everything is an event in the universe so if I say what would that mean how do we go from a feature to event well it could mean give me the event of every time a feature changes and tell me what the value is great we could allow that and so maybe I don't have the courage of my convictions that would require us going 100% all in on every feature forever in always is event driven and in the back of my mind I have anxiety that will have some partner that will come to us and say I have this mutable database and I'll periodically give you a table of values but this is the feature and I will not tell you the events that generated it and since the users don't seem to really need that function I can usually just say no we're not going to add it even though we could actually add it but maybe maybe well maybe we'll just become total zealots of the church of the event and we'll add it in the future okay so it's really actually the implementation of this system is actually very easy well actually like so all we do is we we make a bunch of like fake things like this is the trick of functional programming it's like you know like when you call the function you're just you make an object that says okay I called the function can you return that so it's like like how do i implement map it's like I I returned a subclass of event it's like I'm a mapped event here you go like user doesn't know everything is great and so once we've done all these things what we've got is basically then we have a compiler so that portion of the library is very easy we have a compiler that can look at basically a syntax tree that describes what the user wants and then when the user goes to like run their program on Kafka or on spark or on Hadoop then we just go and interpret that that list of like this this syntax tree into like what it should be on that platform and it's very simple similar to compiler design I'm actually like totally lying so basically what I'm saying is like the actually the implementation is a massive pain in the ass that is extremely hard but like the front level implementation when you like look first at what it does it does nothing except this this free design of like returning the logic of what you want but this is nice because our users can know nothing about the systems this is 100% business logic there's not like some dot reducers somewhere or some like you know put on Kafka or deserialize bytes or whatever it's just about business logic and then it can totally be an ml infra systems team on which I and run story my colleague who's sitting in the audience are on this team we can worry about these kind of systems questions so in principle it's a great it's a great deal in practice it turns out it makes a ton of work for our team and I'm sorry to regret it so anyways so because we're you know I don't want to spend too much time but I I could describe how map works but I want to actually give you some nude data that we didn't have before when I've talked about this in the past which I hope might be interesting to people so I guess the main thing I wanted to jump up and down about is this avoiding label leakage that this lookup function is very magical I know many I worked with colleagues at Twitter I'm not even sure we could describe this problem very clearly if we were trying to predict who would follow whom whom the whom to follow product as you might be familiar with or you know tweet recommendations I'm not sure we even really thought very much about this problem of temporal label leakage but it's incredibly important that means you screw it up in like Twitter I mean obviously they're not paying too much for attention to that probably but um but like you know in a straight there straight product you know there's actual money on the table so we got it we got it we got to really worry about it so we have this pretty magical primitive of look up and as the feature is changing in time I want my user to be able to say okay read the value in the system is going to make sure it goes and reads the correct value not the value it's going to be in five minutes or the value it would have been a few minutes ago it's gonna just give you exactly the right value and you have no way to SEC read the feature at time you know you know 1830 you DS UTC we don't that's not available to you so that's like you know the thing that I think it was very very interesting and you know ten years from now when I'm receiving my turning award I'm sure they'll be citing this this is this pretty other thing so here are some examples and of why I won't be winning the Turing award because I have terrible examples but this is kind of what the code winds up looking like for very very simple features so you start with some events in this case it's a dumb unit test that we have of dogs like they might bark and they howl and they jump at different times and then we can calculate a feature for each of the dog's name how how much what's the average value of how loud it has been the the volume of that of that bark or we got a feature of has a bought a dog barked right notice time doesn't appear there go ahead and vote oh I know it is a month late I'm so into mono AIDS I've got I got mono AIDS all day long for you buy it so like see me later but anyways it turns out it is but um I mean but average value is computing the average in fact computing all moments is associative but it doesn't require tricky encoding you have to also this this type carries with it the total count and also a stable about so by itself the average is of course that is correct that you can't just like combine averages but if you carry with the average the total number of events you've seen you can write the function that combines them associatively so it's nice so that's that but but do we actually run it so we do run it and it is a real thing and so here's like how like this compiler that I described so if you give me this feature I I can convert it in this kind of like interpretive loop back into how we would if you gave me a key and the timestamp I could you know loop through and compute the value and I bet if you see this these kinds of examples these kinds of functional interpreters they're relatively common if you're familiar with a functional programming language fs2 you can find some examples of this in there this is a very common thing this is not compilers work it's very close to a compiler so you ultimately wind up writing Ichabod our users of course don't do this they don't even they they freak out when they even see this like I want to add this feature can I just look at the code and add that yeah no problem take a look it's gonna it's gonna be a disaster you're gonna hate me so so but our users see the nice API but this system that we build is basically a compiler for them we have like a MapReduce back end that can plant it to this intermediate representation that is like MapReduce so we could run it on say you know storm or you know scalding or spark is this very similar computational model and we also even have a third back-end which is this kind of like push based real-time that we actually use this is the one that we we run in real time that it is going to go and talk in real time to a bunch of stores as fast as it can and update those stores and so the coffee reads from Kafka reads one event and does this massive fan out you know scatter gathers thing and goes so we have very you have several different implementations and will likely have more I imagine that will the ones that we have will probably some of them will throw away some of the mole keep and will make more and everything's great so the scatter gallery API is actually pretty cool we're actually using it in production right now even at this moment people are buying things on try and we're scoring it and it takes about 60 milliseconds for our most complex features so if you're into ops at all you might be the first thing is like all 60 milliseconds of slow it is slow so what the hell are they doing and if you look at some of these graphs some of them have 1,400 events and feature nodes in there the depth of these operations can be 60 and 80 different like you know read this then map it to that then go to this store then do a little bit more and now read this then go over there and then we got 60 milliseconds later 60 milliseconds is a long time as far as the computer is concerned but from a standpoint of making a transaction it's actually not a very large portion of our budget and if we can approve the the fraud protections for our merchants 60 milliseconds is of Oregon so we're very very very happy to be that but I was quite blown away with this this is an important number for anybody any aspiring systems engineers out there when we started working on this there's the system that allowed one phase one MapReduce operation and people were like I don't know if we really need more than that and I was like we will need more than that people want more than that I promise you what we do it we will do it we haven't been using it that long so some user gives us this thing it takes a very long time to do a backfill I'm sad to say this backfill took about a week to do for all the history of stripe and we were like why is this am ia terrible programmer am i bad person all of the above so we looked at like well maybe you know it's the kids who are wrong right that's the that's that's the meme and so so we look at we actually wait a second how damn big is this thing let's just take a look and it's 1,400 and I'm like how did you even how did you even write the salad that I mean like what wears out the code and of course once you give them a composable API what do people do they write loops and they compose and like oh for all these and for all those and I'm gonna Jam all these features together and it's amazing so people will take your API and push it way as far as you'll let them push it and the performance of our models is improving though and that's the bottom line so that's what we want so how does it fit together I think this is the most interesting slide this is a new slide this is before actually again because Eric us Haim and I one of my colleagues whom with whom I've worked most closely on this project and I as he likes to say we're terrible people we didn't make this plot and people were wondering how does it work how does it fit together I'm kind of confused so it took a manager actually to get frustrated enough it's like I'm gonna make a damn diagram what the hell is going on here and figure it out but this is what the whole picture looks like so physical events we call them they come in these are things that are bites on a disk they come into the picture and then we have our logical events those were the things we saw earlier these could be mapped sources or filtered sources or whatever transformed in some way these are these are logical events and then we can turn an event into a feature but also features can look up other events so they compose in a very interesting way these are kind of at the logical level these physical events get written to s3 because we use AWS now the semblance compiler this is the compiler that takes that abstract representation and converts it over into some kind of physical job that runs this thing so we have a couple of different ways to run it we can run it offline and MapReduce or we can run it online with this kind of push based scatter gather system that system is called peddler just because I just this so peddler does this neat trick because everything is I'm on way over there it can look at our offline store that has aggregated the data up to a certain point and then it can also look at the real-time differences and because it's associative it can combine those together so we do this real-time lambda architecture trick for free users again don't know a damn thing about it it's great we then push those over onto another cue that the events have been fully materialized into like what their future values are and then our diorama it's a I guess it's like a it's a little model right you know a diorama that's what I think it means anyway that's our model scoring system it scores the models we either have predicted that it's like game yeah it's it's a good good result and we push it out we have this thing here decibel it to measure the volume of things get it stripes into puns maybe not always good puns anyways this decibel thing can read the values back out for other system that needs to read the the feature values at any moment time so it knows how to reach in and look at our Redis and sequins which sounds kind of like sequence file like that Hadoop might produce so anyways lots of puns sorry so anyways that's how it all fits together so people ask the question like hey why don't you open source this and it's kind of like this is kind of like you know alpha is gonna get up and he's terrible at open source and one of the reasons he's terrible at open source is because google has all this great technology that fits together pretty well and he probably looks like I'd like to open source this but it kind of like he pulls like the banana but he gets a banana in the junk you know being held by a gorilla in the jungle right so that's the kind of jerk you don't really want that you just want the banana right so I would like people ask how can we open sourced is like what part do we pull out like any one part of this is not that interesting you know so we're still struggling with that so it's just kind of complicated we're just trying to figure it out maybe we can break it down somehow anyway stripes hiring you can come work with me it's gonna be so much fun we're gonna have like such great times together it will it will be the highlight of your career really and we're building a lot of great stuff you know why would you want to work on data at stripe and like so many people are just moving these bits around doesn't really matter but when we move it around and we it up like people lose a lot of money and that's exciting and high-pressure so like you know if you're into that kind of excitement come come join us we do a lot of cool stuff so if you've looked at data like what does what is data at stripe we have this awesome thing where you can do sequel in the dashboard we actually have this company called run kit that you can write code in JavaScript in the dashboard we execute it and how badass would it be for our merchants who know nothing about all this to write code in the dashboard in JavaScript and then we lift it back into our feature reading system and do it you could come build that with me it would be so great but right now we just have sequel in the dashboard and they can query their data that's a hard problem are you going to do that we also doing all kinds of fraud protection that's a radar product it's about to get so amazing and awesome today we had an amazing new launch so that's why it's so great so anyways that's what my talk was all about I'm gonna hand it over to all if you have any questions before I do that I'd be happy to take one or two [Applause] so yeah calling some folks I'm gonna go in the back I think I saw him first yeah it's a good question so the question was for four features that depend on Delta's like what was the value four minutes ago or five minutes ago you should be able to look back in time not forward in time how do you do that so we do have that and I kind of glossed over it a little bit we have like the event the ability to take an event and get its most recent event or its most recent kay events as like that's one transformation of an event so one of it can turn into like a list of K events which are the most recent k and that allows you to compute different deltas so we have a couple of things like that to make it easier for people they're not super commonly used it turns out but we wanted to support them because we thought people might might want them our main problem is classification so the question was are the majority of Rob's classification problems you know fraud or not classification problem bad actor or not classification problem but we also interested in regression problems for our users our users many of them are very small businesses they may want to forecast how many disputes am I going to have like this month if it's too many it's going to be bad news how what was my sales volume like Italy to be this month next month whatever so those become regression problems we don't really have a lot of the classical but signal's problems like voice image sound we don't have those now we can't have them in the future you can imagine but we don't know also in the back I is there an easy example of observable oh yeah it's a good question so observable like there's this observable abstraction I wish I had a maybe I'll well anyways I don't have a whiteboard but if you think of like a single value and then a list is like one axis of things right so you can like you can imagine that's one kind of transformation you've done you can take an A or you could have a list of a but another orthogonal axis is a to future of a you you're gonna have a value later and an observable is kind of like both you have more than one value and you are going to get it in the you know in the future asynchronously and that's kind of this abstraction people like to call observable in fact observable is pretty much identical to our event but we don't want to make it a monad which you can do for observable because that will make it really difficult to get good like systems performance so we want to try to limit people on that as long as possible but it may be you know we may have eventually promote event all the way up to observable possibly I'm happy to take any questions later I'm looking forward to hearing from all so thank you very much you're done you