New Metrics Engine to Help Drive UBER

scala.bythebay.io: Sasha Ovsankin, New Metrics Engine to Help Drive UBER

Recording: scala.bythebay.io: Sasha Ovsankin, New Metrics Engine to Help Drive UBER

I you thank you yeah so what's third day of the conference people are tired and I really appreciate you coming and you know learning about what would I have to talk about so and part of that is obviously a credit uber so let's talk about that at uber we our mission is to provide transportation as reliable as running water everywhere for everyone so that's uber and this is me my name is Sasha honking I'm dating a latex engineer at uber here's a contact info QR code if you want to scan I'm working at a team that does experimentation so we are Oh yep okay hope I'm not getting too much trouble so we uh we worked a few overs innovation and make people make it easy for people to release features new features in the application and make the whole cycle more controllable more robust and get some data out of that it's part of my team is part of a bigger team which is gold aboard data platform which fuels the whole data thing Boober starting from you know getting the data to processing the data including machine learning deep learning what have you so it's kind of interesting that experimentation platform at Ober was interested in getting getting metrics and getting getting high quality metrics for to do experimentation it's got to make sense I worked at LinkedIn before and we faced the same situation that metrics platform was initiated by experimentation team so I'm as you can guess I'm passionate about metrics about figuring out business performance and it will be interesting to know what you have show hands who've you are dealing with metrics on a day-to-day basis either using them or producing them okay all of people so and then another question is how many of you think that the whole magic process need to be organized and centralized within your company so we have quite a bit of agreement here and how many of you have that or think or working on the solution to do that awesome okay so so where we are we're kind of in seeing here and this talk is about building company-wide metric platform and how it is really possible and even practical and everybody should do it so let's talk about that this agenda why to do it the technology that we did for our solution and the process that that runs with that we'll talk about that more and hopefully some conclusion so metrics what do we need from metrics in the in the company so the alignment is the first important thing so who is familiar with this picture on the right some people are so this is this picture is from a fable from classic classic author of a Russian author of 18 and 19th century banker law that's probably his most famous fable about this three nice oh sorry oops yeah 3 nice animals that undertook to take the carriage to some place and everyone was kind of have a different idea of what direction it should go so you know you all know geometry the the fable ends that the the carriage is at the same place where to start so we don't want to happen that in our company and that's why we want a all you know people who work on in a data-driven manner to optimize metrics we want to optimize the same set of measures because if we don't then that's what happens another thing we want from the MAGIX is to be reliable to do to be able to trust the data so how do you how to achieve the reliabilities is a big big question and part of that is like imaging I do my little magic which i think is good and you do your your little magic that you think is good and then they're kind of similar but you do your testing and you do you're fixing and I do my testing and I do my fixing so we kind of spending twice as more time and not necessarily getting the same result so if you both you and I working within the limits of the same platform that there's there's much more chance that the decode metric it will be kind of higher quality so the reliability will be would be buried or in other words given enough eyeball so the battle bugs a shell right so that's a that's reliable now when you have a platform you usually have a dedicated team or some but some people who work on that and do that so having having dedicated people as always there's somebody you can talk to if something doesn't work they can help you fix it so you end up with the with the result that is easy to trust people easy easy easy for users for people in the company to trust metrics if there is a this ownership behind so this is what what we want from our net metrics now a little bit of about our situation so uber runs in over 550 CC cities in over 70 countries and each one of the city's is a little bit of little business by itself so they have a significant level of Independence so they take their own decisions they they can affect their own pricing do their own promotions and things like that all these compounded by the growth you see you can see that in first half of the 2016 we did we did as many rights as with it all the time before that so with this with in this situation it's really hard to kind of have a have a clear idea of a sale what's happening so this is our situation and then the question is how how do you in general how you do you know well how do you make decisions like that who who knows who is this person okay so this is Peter Drucker who's credited with the phrase if you cannot measure you cannot improve so in our case the question is how do we measure and the answer is very simple okay so let's build a platform that will be used throughout the company by all the people and hopefully that will provide this communication Telegraph this way between different people in the company now i wanna i want to point out that the platform is not only the code technology the france it is also people who who are rounded who are using this technology and the process that that makes it all happen and i would say that getting the right proper process I'm getting the right process is as important or maybe more important they get getting the technology right so but we start start with the technology this is just a very high level overview of our system I i would i would go into details of each one of those but i just wanted to give you the kind of overall idea what it is so basically we have two sides we have people who are writing metric definitions and they are organized in a little organization called magic council we have people who are using the metrics bye-bye different bi tools for running different bi tools now the definitions are written in dsl and we'll talk about that storing the registry and this registry is used by multiple engines so the same definition is used by different engines that they use their own data data stores when users send queries for data so they basically query query data and this engines produce data for them so so one thing to note about it is that it is less of like pipeline thinking where you know we are just spill data somewhere and then I process it and I get data out it's more more like we think about it more like a query engine that you I send requests and then I get the data back it fits our model better because of high variability and what different people may want from from the system alright so matrix platform is I said we have multiple teams in the company and they have their own high level of Independence so we we want them to use our platform we want everybody to use our platform but we cannot force them to use our clock so the way we can make them use use the platform is to make it so ridiculously easy and beneficial to use the platform that they will simply have no other choice so that's that's what we want from our platform to be easy impartial to be well integrated into the whole company infrastructure data infrastructure and we want process but one process lightweight so that people want once spent a lot of time and be bothered by this process right okay so let's just dive into metric so this is a one one of the example the metric is how many hours drivers spend logged in and active in the system like doing doing real trips alright and assume so this is hypothetical hypothetical sequel definition let's assume that we come up with this with this definition and it works produce metrics all fine now somebody else says okay great so this this metric works this is by the way this is like the the decor of this metric which is essentially the formula that calculates that so so somebody comes and says okay I also need in a certain i need this magic in the same in a certain range of dates okay so they add date condition to this sick well copy the sequel and add date condition to that and then somebody else says okay but I need it is in the city of San Francisco so they do a joint and add another condition and then somebody else says okay I'm experimentation to you I need need to join it with the experiment and filter on my experiment right so they they have another copy of the same single and then yet another person concen and does something else like the model drug what what is the type of driver and people keep keep you know breaking it down and keep copying these sequels and running them so what we end up with the whole thing is complicated so this sequence may grow to literally hundreds of lines of sequel it is it is hard to manage because now there are multiple copies of this and you know if the definition changes you need to go and change all these copies and then also sometimes data scheme has changed it changes for example so input schema changes it breaks the sequel so now it breaks like 15 different sequels and need to identify all these users and figure it out so that's that's a actual answer to the question of ok so white just not to use sickle just wright american psycho everything will be fine so not so not so much ok yes so this is probably the most complicated slide of this but we actually need to get through this because we need to understand house house this whole process goes so again we have way of one of the versions of this in our hypothetical sequel so we have a formula we have a joint that need to happen and we have filters and and and we have group by CD of the drivers so that's that's that's one of the examples so here's here are the things that happening here so there are multiple input multiple parts of the expression and multiple inputs and each one of these inputs should be should be considered as its own kind of little pipeline so then you apply certain transformation to this to this input so for example you take minutes active and / 62 to get two hours active if you want right so the the interesting part is that even though the definition is here the part of it like like it is a simple nice formula but part of it happens at this step before aggregation and the other part happens after the aggregation so so when we start looking into how all broken down it's a basically the whole the whole formula is is spread over over several several parts in the process so looking at this we we actually came up came up with with ideas which is one of the core ideas of this whole system and I consider it kind of a very important is that will be much better off if we split the this definition into two parts one part one part is the formula and the other part is actually conditions filters and group buys that that we apply to this so so doing this is going to to reduce a lot of complexity out of the system because we just take this definition store it in the registry like I like a show and have it reviewed by multiple people have it in a in a very good shape and then people who need different ways of slicing and dicing this day they could do so but when they do so they don't they don't say change definition if they change only the query nation stays the same so that's a that's one of the basic premises of this this platform is that we we can split metric metrics into into definition formula and the query to to get the data now here's an interesting question what language would I use to do all this you know we tuber used many languages so dunno sequel JavaScript go no does anybody have any ideas what Scott Scott what yeah let's try that yeah so so totally so actually a skull makes it easy to make dsl's and and this is a one of the reasons we use scholar for this so basically this definition is a valid or most of it is is a valid valid Skala expression and then what we do is we were able to reduce this metric definition to a really simple straightforward way of describing described in the operations and not only that but then we also if we use a scalar it is composable so we can take this thing call it some name and then use it throughout multiple metal matrix so so we can compose it so that's that's a overall very very important feature so it's a composable and it's simple and it's simple to point that people who you know are not engineers and not even data scientists business people are able to look at this definition and say yeah okay it makes sense or maybe it doesn't make sense but at least it provides the media it is simple it was reduced in its complexity to a point where you can reason all people in the company can reason about that so what do we do with this afterwards we like every like in many cases in scale we just break it down into parse tree so this is an example of parse tree each one of this nodes is some some some element of a class I'm object some instance of a class and we just build parse tree of it and then we'll walk this tree what this tree to produce results to produce some execution plan but also there are multiple ways we can we can use it for so for example one experimentation team we need to do statistical analysis of metrics and figure out whether the change in the metrics done by an experiment is statistically significant and this allows us having this parse tree in a way that we can process allows us to look at the definition and say what type of definition is the different different definitions different for moles there there are proportional metrics and there are racial metrics that are continuous metrics and each one of them uses different statistical method to do to compute the student statistical significance so having having even parse tree allows us to to do it automatically we will not get there but we hope to be and to be able to to to copulate statistical significance automatically so then the other part is you remember is a query so query is also language the internal dsl scale in built into Stella and you can see that it is a little bit like sequel right similar to if you look at like data frames of spark they're also considered they also look like they also are a DSL so in our case the dsl is that so it is sequel like but but it's not it's not totally little a lexical for example there are no from here and there are no joints here so the thing here is that we are able to buy having definition and there as a parse tree we're able to actually analyze analyze the data scheme and figure out for each one of those metrics were what are what are the original tables or they did data sources that is coming from and when and in in many key and also we are able to figure out the joints that we need to do on the way in order to do that so basically the simplicity of that is also it dramatically simplifies the writing queries for people who need that because because they don't now they don't need to keep you know the whole dating scheme in their mind and say okay so they I go this this way I'll go this way they just say hey I need I need this thing and give it to me and I also need to break it down and if it makes sense and a free we can we would provide it and and don't don't ask any any further questions so that's a depth that I think is easy in powerful yes here yeah so so the question was how do we how did we split a magic into formula and query so we just that's how we define our or the dsl our platform we so this is the sequel is for demonstration right so it's a it's not not something that really works what works is these things so we're just saying okay so here's your dsl to define the formula and here's your dsl to run queries against this formula and we provide environment for you where you can run this query so and get get the data back yeah it does answer a question well okay you're welcome to come up after the top [Music] yeah sure so so the put your absolutely so so the question is what to do when you need multiple multiple metrics in the same query write multiple formalist combined with one query so so so this is this is the formula and you can build as complex formula as you as you wish so if you need to add several things you just take these things and do plus of them right a plus B plus C okay okay well yeah let's let's discuss it later so so this is this is the the formula query and yeah so as far as integrating into into into the system i mentioned this a little bit but i'll talk about it more so we have a company-wide scheme repository of all the data that is in a lake and we take this repository and then we extract table schemas from that and from this we build kind of scallop last week generates color classes that that implement this state that reflect the stables and we'll also extract relationships in a semi semi automatic manner some some cases we can do it automatically in some other cases we just write it up and this is basically this goes into a big yamo file that that represents the schema let's say yeah what's going engine configuration and then when query comes in there is engine core that looks at this configuration file walks the schema and builds what's the graph of the schema based on what what things we need from this and what what breakdowns what group buys it and so on and then builds execution plan for that which is basically tree in memory and then this execution plan is handed over to the engine for execution so for example in in case of spark engine the execution plan is become simply a data frame and then the this data frame is sent for for execution actually yeah I have this too yeah so on this lived on this slide so basically this is this is the query and then there is an engine that we we give the query say hey convert it to data to a data frame to spark data frame and then we have a data frame in our hand and then we just store it sort story the result somehow or return it to the user okay so that's the technology all easy powerful integrated but what about process so again to remind you there are two types of people in this part so there are people who are writing metric definitions and there are people who are using this this engine to to get the data so the so this is about so people who are using using metric definition this is this is where we you know care most about the process so we we have to give them give them tools so that their process will be lightweight they will it won't take too much of their time so like straight for in a straightforward way we started by handling metrics and in one big spreadsheet with all the stuff and and the dog came up with a certain form of documentation how to how to document the metrics and the discuss a but then over time we just came up with whether you I let's see how it works oops okay let me try again came up with the UI that helps too oops yeah okay yeah alright so so this UI it has a documentation it has a search facility where I can look for metrics Howard's horizon-- line I have no magic I can create one mile on my own and fill up all day all the attributes of it and create it fill the reviewers yeah and and most importantly I need yeah this is this is a like some some subdivision of the library into magic back so that's the name of magic back and also also they I need to fill fill the formula of course so why am I so excited about this trivial you no form form interface which you know junior developer will do in a day the reason is that it all helps to integrate into the whole like development process so basically we create a pull request or sort of a pull request in our internal development system and then people can we can leverage the whole development process reviews and check it in so so we were able to leverage the knowledge of the people this this process of they it also also kind of becomes lightweight for that because it's a okay I know how to check in the goal that how to hold the review so by integrating this is it makes things easier right now I need to move okay so sum it up is a powerful integrated lightweight process these are some of the groups in in the company that either use our metrics over our platform or over 20 years and these are future directions more doctrine of course improvement on the dsl more engines and we would plan to add secure engines like hi presto vertica to it so it's CZ to query real time is a is going to come next year Santa at some time and we're also interested in the question of whether we should open source a so if you are you have interesting that will have opinion on that please let us know these are some ways to contact us so overall building company-wide magic platform is ball we consider it practical we're working on it and you should too this is the team that are working on this platform right now and there's a lot of interesting things we we need to do to it hiring right now get in touch any questions yes a real time matrix so we were just a conceptual phase onda we haven't done a lot of development on real-time metrics we have multiple platforms in the company that we plan to use for that so we use a party Sansa and our team created a sequel interface on top of a party Samsa so that you can query streams streams in real time and sick with sequel so one apart I want to know the possibilities we're looking at is once we have a sequel generation for that use that for for for real time as well and there are there other there's a fling and things to consider [Applause] you