scala.bythebay.io: Matthew Tovbin, Doubt Truth to be a Liar
Recording: scala.bythebay.io: Matthew Tovbin, Doubt Truth to be a Liar
welcome everyone I would like to thank every one of you for coming to my talk because it's the last one today and it's Sunday and we all want to go home so thank you for coming my name is matthew toven I work at Salesforce and we're building a machine learning platform named Einstein and my talk will be a lot about machine learning today and what about Scala so it's kind of an intersection of both worlds and I think it will be interesting for any of you I think every presentation should have something beautiful in it so mine has this beautiful quote and I took also one line out of it put of the title the out truth to be a liar so no matter what you do it well actually it has nothing to do with my presentation I just like like Shakespeare and I read it in Russian so when I when I started in English it backs it oh this is even better so so what I'm going to talk about is I'm gonna give you a short introduction to machine you're in however you oh you know no it will be great to repeat it however it doesn't you'll get some some new stuff to learn then I'm going to talk about real-life machine learning and the challenges that we have the next topic would be how we build an application using spark ml and I'm a leap and doesn't matter they having this renamed issue right we don't know what's exactly it is it's cold right now then we're gonna dive into the type feature engineering that you might have heard from my colleague earlier today Shabana Barr who gave you a short introduction in how we do it at Salesforce using SPARQL leap and our own in-house transformation library we call Optimus Prime then we're going to talk about the behind the scenes about some cool cool and neat stuff we did to make it all happen and I think you will find it beautiful as I do as a software engineer and then I'm going to show some summary and what what is the perspective of perspectives of this project going forward so what is machine learning machine learning is gonna be computer to learn when we build an algorithm when we build a program we want it to be able to run private an ability to learn from a data or an experience right experience just the data so what a field in more data and get some different behavior of that of the algorithm that we have and what are the test machine learning solves well we have a lot of tasks around the world we already know that are solved by machine learning and we are introducing more and more fields that we apply machine learning to such as diagnosis prediction data mining and so on so how does it usually look for from a user perspective to build a machine learning model so we have a bunch of data that we that will get some real data maybe some sensor data or I don't know user clicks or something that we want to be we want to observe one then we develop some features out of it so when we say feature engineering machinery in terms it means we want to define certain properties of of the data that are important for us those properties should be independent and properly distributed otherwise we cannot even trust the predictions we do once we do the future engineers and actually this is one of the more most complicated processes because it might include aggregations and joins of data computing distributions it might take a lot of compute power and a lot of human resources just to pick which properties of the data to model on so we want to make this process to be as reliable as possible as fast as possible and as convenient as possible for engineers then we feed it into the model training model training is the part of the machine learning system where we apply algorithms on the only features that we produced for example logistic regression or random forest or or you name it word to back we usually want to prepare the data in the way algorithm wants it so if it's a logistic regression we want to gather give it a numerical vector so wanna transform all our features that we injured into some numerix and some doubles or once the model training completes would get a bunch of models those models are being used for predictions this is actually what we want at the end of the day we want to pick the best model and that's where the next step comes out this is the model valuation on the model of valuation we compare multiple models one to each other based on some golden standard or maybe some predefined thresholds or some something dynamic so this is all in theory this is what we read in the books when we talk read about machinery in what in the real life we see it's actually a bit more complicated it's rather building a pipeline not a model so we're talking about multiple data sources where we feed which all of them has or some of them might have different data data data formats data I would say different Cadence's of data some of the data might be streamed some of the data might be the flat files some might be in the snapshots of the databases we do want to make some sense of this data that's why we need an extra step we need an ETL step we all know what ETL is I'm not gonna elaborate on that ETL may be as complex as it may be very complex we might have multiple aggregations most everyone have some filtering going to join some data clean it up and so on then we do feature engineering as we as I mentioned model training and then we produce models again do an evaluation well evaluation actually can discard some models when it produces it ok this model is not good we need to retrain the model because I know there was a bug or its sanity checker says this modelers provides bad predictions or something else or there are not enough features we just suddenly get bad results in the model predictions because there is not enough features so we need to add some more features or redo some distributions once the relation is satisfied with the model quality the model goes to production we deploy it to some web server or I don't know in the backend system this binary becomes productionize and I will start scoring new new and items with it so this is a scoring part well actually for scoring we also need to get the feature engineer at all step as well so every new record that we want to score you mean that we want to provide a prediction for needs to needs to be come until the same format this model does so that's why that's why we actually have that once the scoring is completed we have the happy user Wow yay we got the scores back well actually the score results can also become new features and then this loop can go on so if I predicted certain say a likelihood for the lead to convert this also can become a property of my new common models they can base their feature engine and on based on that fields so that's why it kind of it's kind of all circular here so real ephemeral I'm just gonna want to mention sound some problems it's this list is by no means as exhaustive so ETL it was really tough we all know that in really key PD I read papers on this read books this is we're doing it for for for a while already feature engineering is even tougher because it's on top of the on top of the data that we combined and aggregate and everything now we want to make it into the form that our mathematical algorithms can can run on and then next one actually liked it a lot so our prototype in our Python blah blah runs great but when we deploy it anything can happen you mistyped it you mischief did something bad with a code it doesn't scale it's like come on things just don't don't work because your sample ran on 2,000 lines of data copy paste and quad basically doesn't work as well right we don't want to copy paste code into the get into the gist and then share between teams and I hope look I have this new feature I just did you model you want to try that yeah I'm gonna try that and then it's kind of we don't want to lose it we actually want to track all those things so we want it to be part of the code base model training it fells exactly two hours have to go to sleep happens all the time like actually all the time you run something it fails you come next day what was I doing yesterday maybe I'm gonna repeat it and then just no they just go by we just go by and nothing so that's bad we don't want to we don't want to have that there no different problems right that is not there that is there different format out of memory insufficient space our jobs failing because we didn't we didn't do something and then of course transitioners so who guys never seen theorization errors like everyone since oh yes you you start up Sparky gets realization error oh you forgot to what ever cry of thing so now we have models and scores but can we trust them that's a that's a that's a huge question like who can we trust the machine I don't trust the machine don't tell anyone so specifically for Salesforce we have more stuff but I just named kind of the the Moody fun part the multi-tenancy problem for Salesforce we as a platform build build models for all the customers so we have square fanatics we have thousands more customers on our platform it's not like we're part of a company that we have our own data science team and we build five models to do a nano turn prediction and we're done so no we we are customizing it for all the customers and then we have environments so we need to mind maintain our production maintains live staging QA and death for all the customers and then we have data sources so this is just internal ones I mean like marketing cloud service cloud but they can be some external ones so we want to inject Twitter streams for our users how do you do that our company is doing that actually or multiple data entities such as leads opportunities or anything on top of that with built-in multi more applications so we want to do leap score and want to sort our leads based on the likelihood to convert within the next three days or three weeks that's really interesting how do we do that difficult because all leaves a different based on the customer the domains are different things actually different how how do we make them work the same that's the that's the biggest challenge for us how do we scale the machine learning capabilities that our team has across the company we don't want to replicate our team so and then we have we have other verticals or and all dimensions I would say it's such a security so one so for example have our data encrypted and only decrypt it when we model it we want to be able to scale no matter how many records you have just fighting machines automate the processes of course right transparency for the users and cost efficiency and so on so that's why we built this for some time this is like a the best slide for me I love it because it has so many logos I like what you see this is it's a it's from fallout whoever is in theme will recognize so AI for everyone that's that's our kind of moto we want to do artificial intelligence for everyone on the planet but we start in humble we do it only for Salesforce so machine learning for customers engineers and data science however you name it so we're gonna we're just gonna ship it and then see what happens with it we aim to not have any T L so in TL easy pipeline so you just have the data it's there you don't have to worry about this and no PhD read the tutorial and move on so we started acquiring great pieces into our stack so we first of all we said we need some type language so pick Scylla we're gonna do Scala and then we started looking into the great great feeds for our machinery and so we picked spark and we'll run an Amazon and we use this us to - we also have prediction i oh we are lucky to have them as part of our team and we are pushing their project further so ever using it it's great it's going to be there for for a long time it's now it's a part of our party incubator is going to be top level project soon and we have our own in-house Optimus Prime this is the one I'm gonna be talking for the rest of my talk I'm gonna give you a lot of cool stuff about it and I hope we're going to open sort of open sources at some point so it's a it's a transformation framework that's why it's called Optimus Prime and everybody got it already so it's declarative it's collaborative because we can share actually code we can share libraries it's reusable because it's all it's built like this we just we just built it to be reusable built in pieces and it's of course typed we're trying to have types all across the stack we also have services micro services nano services scepter services I don't know smaller services for different pieces or a platform that we do for scheduling and monitoring and authentication authorization a lot of all of pieces that we have I'm going to concentrate mainly on machinery next aspect right now and on Optimus Prime so let's start from showing how we do a machine or an application with spark ml lib or ml I'm not close on the name now because it's I think it's ml so say we want to predict survival and Titanic this is a very famous data set who ever did any Kegel training and who didn't should do that that's like a new Hector ramp for data scientists so what are the fields that actually influenced the survival so I'm basing my the sample on this github example from from BAM which did a great job so building spark application starts from bunch of imports and spin up sparks in the local mode everyone see this awesome I made it like bold enough I heard there are problems with projectors so the spin-up spark and then you please it something from the context further on right so we can reuse this context throw down the line and then whoa we start by defining schema a bit scary not not that bad we I just defined this is a helper function read data and it defines a schema that says okay these are the fields I want to get from the CSV file and their types are integer or double a string or whatever type I want and then I'm saying if it's not able or not so I set everything to malleable just in case because I don't know what's in there very easy to misspell fields well let's hope I didn't so I kind of double check with my CSV file multiple times great and then you can set any type there and then what happens it's just gonna explode in the runtime I don't care once care about this later so we we create a data frame by creating by reading the data and then I actually doing some real names on the field so they will be lower cased and there is no better way I mean maybe there is a better way that I'm just doing one by one and I'm only picking actually the the fields I want to be in my model I don't care about the rest I'm taking the survived field I'm taking the general field age passenger class siblings poses this is the number of siblings and supposes that the person has purchas parent and children and then embarked which station the person barked at the end of this thing I'm running I'm reading the data from the Titanic CSV file I'm getting my data in data frame so whoever you familiar with sparked data frame is untyped matrix of values or basically maybe just a set of rows and each row is just map it's typed but it's typed at runtime so now I want to do some feature engineering write some a little bit I want to modify my data a little bit so it will it will help me to do so I am a dysfunctional it feature so I want to create a new column family size so what is the family size sum of siblings causes plus the parents children and myself and I want to and I'm modifying the data frame this is how we modify our data frame and spark you do it with column you pass the new column name as a string they pass you UDF user-defined function and then you pass cold you also name them by names I can I can do whatever I want here I cannot write columns I can so basically you think about it right I have two functions now the next function and I want I can overwrite the same column my previous function did nobody will do anything about this like because I can I just gonna do that and it happens when I have multiple teams working the same data set so and then this function for example does feeling fills the missing data so whatever data is missing if ages now I want to replace it with some average so I first need to compute the average and then I'm replacing the average with this field same and then I'm doing something for the station so if the station is missing I'm gonna play some of the default Southampton station okay done with this I'm actually modifying the data okay modify the data field missing but first thing to the edge features and I don't need to remember to cache it and then persist it maybe because it's Park whoever you work with spark knows that it's it actually needs to use I mean you want to use the cache because everything is lazy so you want to cache stuff but you need to cache wisely not to over cache because it's gonna start writing to disk and if you don't want it to be right the disk into the use different storage blah blah it's a long story and it's look it's interviews with your code so when I caught now data scientist needs to remember to do those things it's it's it's basically not not happening usually what happens I have the data sense and maybe some other team productionize in your code later and then I'm splitting the data into my test seven and and and a train set I'm taking 75% of my data to become a training set and the 25% my test set that's that's the data I'm gonna use to test my model okay let's let's go on now I'm I'm creating the features I'm I'm actually doing I'm taking the features and making them numeric now because I want my I want to run on top of this I wanna run just one algorithm logistic regression the simplest one but very effective so I'm concatenated on the fields and I'm creating this vector assembler which is just a transformer right it takes a bunch of columns and produces one new column of type vector and that contains all the values from previous calls converted okay now I'm doing the logistic regression estimator which is just a reduced function right it will it will run on this feature vectors and we'll run the logistic regression optimizer and and get some results finally I'm stitching this until in to the pipeline so pipeline actually needs to get the stages in certain order so it needs to prepare my fields first and then at the end run the logistic regression so this is that's why order here is very important so I need to remember that too and then I'm running the Train so great I'm doing a cross validation on the model with hyper parent doing and and the this line like train the model and compute scores this is actually the line that does the does the model train effective model training after this line I can do whatever I want with a model now I can save it and I can score new data and so on so this is great because we had we've seen transformers and estimators and some things actually nice because you we haven't mentioned anything about right and about the old style coordinates such as MapReduce there is no there is no complexity of MapReduce in it and it's quite sequential set of steps so we want to maintain the same same ideas as we go so the good part in a simple abstraction we have transformers same map St Mary's is like reduce and then we have pipelines to stitch it all together pipe ones are actually reducers to so they actually estimators and then we have Sur ization of pipelines and transformers that allows us to save models this is all implemented so it's good we don't want to reinvent the wheel here PMML is Islam is baking still tea you don't want to do it yourself I guess I if you brave you can I guess good implementations for estimators so spark ml has a lot of cool stuff implemented already and they they care about quality so they have a good test coverage and I mean I'm happy to use some some code and don't don't reinvent the wheel the wheel it's a known one framework so whatever you cold it's the prototype or an explorer your data it's the same code that's being deployed it's not like okay we're gonna do octave or R and then go write it in in big hive whatever you use in your stack it's it's here it's the same Scala code that we all love and use it has multi-language support so happening used by them and it has a very healthy ecosystem so this is the little graph of the issues for the last 30 days closed and open so you see it's small gap there is literally guys are working who is key from data breaks anyone from day lyrics anyway thanks for doing a great job what else so the not so good parts okay the not so good parts this list is not exhaustive too and like sorry data breaks guys but there's no type chicken like I miss it like I build spark code and like each time I each time I deployed I pray like it's so transform an estimated scholar also way to open transforming code gets data data set anti Padova gets data set which is a type data frame but the interface right now it's a data set underscore I missed it but it's a data set of any type and it produces data frame so I can do whatever I want my transformer can erase all the columns because I can and then everything down line will fail and it happens so and data frames are everywhere they just they just everywhere all around the code when you start looking into it so it's easy to miss spell call names copy places horrible no integration really with ml vector type which most of the estimators use and it misses a lot of RTD functionalities already is the first type that's Park introduced this is like a resilient distributed data set and it's great it has a lot of it's very its mature and it has a lot of functionality that we like but it's just not there for data frames when you say for simply aggregated data frame you you get into this aggregated data frame that has maybe like ten functions on it we can't continue with it so you need to constantly go back from data frame to RTD they different already if you can because our release time there is a lack of support for common operations so we need that like joy and aggregate feel things things we need and schema our algorithm definitions are all interleaved we've seen that in the example so can we do better so we can definitely do better so I'm gonna show you an example of how we do that with Optimus Prime so Optimus Prime is a transformation fabric framework that we actively use for the last two years already and it's currently in the v2 States Ojeda v-0 v1 I would see - it's an internal kind of V but it's it's getting happy it's gonna get it's gonna get even stronger so won't have declarative syntax we want to have want to be able to argue on syntax better one have usability of i/o operations and one separate features transformations from data operations we want to have it separately even just for unit us you just want to define your transformation steps with your features and the engineer and everything and then and then only then apply it to spark maybe does happen to be spark maybe it's gonna be flick like who knows ideally you can also like distribute replace your computation engine underneath but it's very going for so show me the code show me the source you must yes I must I will here so here's the same example with Optimus Prime I'm doing the same same solution for a Titanic but with Optimus Prime now so for us it starts from defined readers so it would define some reader it does have to be in the same context but you will define a reader that in this case it's a simple CSV reader that reads the CSV file we have a bunch of pretty primitive readers that we did that we use like aggregated readers and conditional ones so this one is very simple it produced some other scheme in this case then we want to define a response feature so okay I'm going to define my response feature it's going to be some binary feature of survived and I'm and I'm passing my extract method it's gonna be applied at some point and I'm doing dot as response my response feature great doing the rest for the rest of the features same thing have you seen any strings here no because there are no strings the only strings that we have are just the values for example equals survive so obviously this logic will be a little bit more complex but we actually deduct the the feature names from the vowels so the cool stuff behind the scenes so it's basically what you see is what you get if I define a variable age this is the column name I cannot define the same variable down the road in the same block it's not gonna compile because we get the uniqueness of the vols how cool is that so if I'm writing my code and mixing multiple features into the context and they they will collide if anything overlaps or they won't and you're happy it means that you could so great we get at the compile time no data frame called collisions no no nothing if you want to rename call it like my age great so will it be a new column so moving on now I want to create family sites that's what I need to write that's it like because this is what it is I don't want to define UDF in nothing okay I want to feel me some great I'm just gonna create this operator feel missing with me great I'll do that and 1/4 Southampton I'll just do embark dot feel missing with some value I'm done with my features and then I'm gonna make it a feature vector backing into the sequence although all the features I want do as vector I'm done am i am i numeric vector in there so actually nothing of that still happens like there is no spark in the picture right and even here nothing we created the small selector that allows us to pass model parameters into it for example logistic regression random forest and so on and the classification and then we stitch into the pipeline the pipeline will rearrange and arrange the stages underneath and only now we spin up spark and only now we're doing pipeline dot fit and we created the model which would amuse to to do something with the test data with a test reader and then we again we can save it because we actually wrap around spark so we're using the same cool features of theorization just to summarize everything is typed reader is typed transformation estimators model certain more time is is typed it's declarative I think it's instant to it if it's like it's in our IDE you do variable dot and then it autocompletes based on type feature names in front front vowels misspells features are always unique in a code block yay we have a lot of common data operations that we did and there is no dear friends I think this is the familiar this is the best line I can I don't want to see data frames features and transformations are separate yes they - so what's happening behind the scenes so we started by ok what's missing spark spark misses features and it features misses types and features so I said we're going to define some features with some primitive types so previous times maybe numeric right kadavo or or text which is an option string and I'm categorical map string to double okay sounds good let's do this and then we're going to do it transformers same transformers but we're going to implement the moon types now so those actually typed I forgot to add the te pui you will see it down the road so features can be transformed with transformers okay so features should have a method transform will or the transformer should have a method transform feature any of those and then transformers produce new features on when it to transform they create new features that no their parent ok great estimators otherwise tomaters do estimators can be fitted into become in transformer same as spark does so when I'm feeding a model it becomes a transformer that you can transform a data data frame to with scores and then we have pipelines patterns that kinda a bit aside because they are they also estimators and they can also be fitted but they also need data readers to operate and that's where the the actual IO happens so pipelines will initiate data reader read the data in some format and then they materialize features on those on this data and then they become they become a transformer again so you can you can do scores so this is like the picture to keep in mind so the only thing we we mainly edit is the feature and the the readers and then we typed everything else so they can all cooperate work together so here are our primitive types that we did as I mentioned numeric is double so just shortcut scanner because they designed as they speak in in the left words I speak on the right words and we want some common language to talk so I'm kind of fine once the types key binary great and then we define the feed the type feature so here's the type feature this is the trade that we did it took us quite a while to came up with I mean I can't I appreciate it's very compact so it has an implicit with type tag that we used to overcome the type type erasure and it's it's heavily used for with our macros for all those things the name of the feature is the name of the column in the underlying data set or data for data frame so when we actually materialize those features in the pipeline this is the name that it becomes now we have some default value that computed based on type the origin stage means I'm a feature where did I come from what was the transformer that created me or an estimator it can be any stage so some pipeline stage materialize me and then I know my parents tool it's actually like a portent maybe I depends on okay and these are the features that I required for me to be materialized this this helps us to build those graphs of transformations and then this is the function I was speaking before to transform with them takes a stage and creates a new feature with a new type so we have a type you here that is produced you so it's always typed we pass in the type everywhere so some magic with macros say I want I want to write this vowel siblings poses equals feature builder some just feel just a helper to build features and then I have to pass the name so I don't want to pass the name I wanted to infer from vowels can we do that yes absolutely so this is an example of how we did that what half me to get okay sorry this is so our macro takes take us a context and then it tries to infer what's the defining well name if you want to know where I took that function it's actually dig into the SBT codebase so they did amazing stuff on that so because when did Google Scholar macros you get nothing so you need to go actually go to github and just search there and then maybe get some cool examples and then they have a lot of macros so this is one of those so this for example takes the vowel name and creates an expression of feature builder dot numeric OPA block or apply and passes on the type and actually just rewrites the second line with the first one it just passes the filename as the strings underneath everything is a string but it's hidden it's hidden like who cares it's it's it works yeah it's great feature transformations with implicit classes so we use implicit classes heavily to so we won't have this annoying syntax right siblings poses dot transform with blah new binary nullable and numeric transform with a plus function and the second feature and then the transform with another transformer with another function and then one this is very very annoying so we want to do shortcuts so we do in paste classes and then we create a bunch of for PC classes for all our types like okay we want to do reach now the numeric and then this will have 250 operators that we support plus divided ampere so like boolean operators you'll miss them with me and you've seen a few examples of those so we can expand him on that whatever they decide is fine needs we just add it so it's very easy all they need to do they need to implement a transformer and then we just stitch it into this thing so very simple oh no we also use macro here correct because we want to miss this because that marker is actually huge to get the volume for this I thought like okay I have only 40 minutes actually I'm almost out of time I think okay where my features and transformers so this is like an example how it works underneath and just a visualization of how it works so say I have this expression and then this is the underlying data frame so for each column we specify a new feature right and then we apply the transformer which produces a new feature which we don't see in this expression right this is like an intermediate one and then we do another binary transform with plus one which produces a new feature that becomes a new column it does not materialize again immediately but what we have from this expression is that we actually build this little transformation dag this is the most important thing when we when we get into the end of the block this is the theme that we hold in our hands this is what what users want when they engineer their features they want to do some interactions between features of normalizations and then they get this this deck from this deck we get the the stages and those go into the pipeline this is gonna see we're gonna see it next so what are pipeline stages pipeline stages are the same pipeline stages the spark has its on the right on the right self pipeline stage this is part part on stage but we had the two types here input features output features and then we do set input gate output so we can stitch them together again and I hope you see the pattern already and then we implemented the transform schema this is a completely final override on the top top class we were able to do this so I think it's I'm not again not showing that code because it's like 100 lines but my own maybe less but we're doing all those schema changes for underlined data frame when we when we apply this at this stage which nobody has to remember to do it anymore so basically for my binary transform if I want to do plus its input feature type will be a couple of feature in American America for example and output will be a feature in America going back to our example and then we'll have to go what we'll have to we did have to go and created those types for all the possible combinations okay this one extends opieop at one stage base and it has one output feature but has multiple so it's something X X - 1 and then we did ok this is the one to one this is the one to one mostly used take one column create another column but what if you want to have two columns so this is the for the binary 1 2 input in two input types one output type how many combinations we have as much as you wish but it's super annoying whoever knows and you obviously do scholar products they only did 222 so we kind of ended up on 5 but we want to have a lot but if you have a lot you're probably doing something wrong so can I so we we still we still we still got sure want to do that and it's already annoying it's like we have like 5 each time something changes like all those classes oh my god I have to change a lot of things so maybe maybe the can be a macro to generate macros to generate code for us I don't know how that code will look like I'm really scared so here's an example of one-to-one transfer this is the most the most used one take a column get a column so a trade of a transformer it now extends you see we're mixing in our stage one and then the untyped spark ml transformer but we have new two implicit here with the type tags that allows us to do a lot of magic at run time and then the user provides one transform function and now this function is typed so when the user operates with it it just says okay here's the function and I know the type of it it's numeric two numeric and do whatever you want with it and we're creating the UDF underneath but the stain is sedum you see so nobody kind of needs to know about it okay and so on and again combinations here it's the manner example I don't think I have time for this the questions after yeah sorry I have a little bit more slides so estimator example this is the spark estimator but again with types so we mixing in our stage with types now estimate has the speed function to basically become a unary model which is just just a transformer another transformer I want to I want to take an estimator it writes my model I want to feed in a bunch of data and so I mentioned data set and type data set and I want to produce a transformer so we rappin it all inside I'm again I kind of stripped a lot of code here because we're also handing the persistence we check in okay what's the persistence of this RDD should we catch it should be on cache it and we do it all around this function so basically I stripped the code here and strip the code here I wasn't sure we can handle it with time constraints but at the end it produces this model and this is this is what it is only remodels a class that has extends a model and extended transformer it so again it's typed you just run it as a regular transformer so and this is the mechanics okay so we have the data that's what happens when we run our pipeline when we run OPP python fit we have the we read the data say it's already we generate features in they become a data frame this data frame then okay state aside for a second and then we take this transformation back from all the features that we want to use for our model we do topological sorting on this because want to have the correct order application of those and we know that nothing will overlap because we know the names are unique so we're kind of good okay even if we just add in columns nothing nothing happens inside the data frame nobody deleting columns because we own everything users only operate on functions and those functions have no access to data frames so I think I think this is like worth appreciating and it's fine so spark pipeline this is what we use underneath so our our pipeline use a spark pipeline because when I use the same sensation same coolness things that spark has so we just do a new spark pipeline we set our stages after we sort them and then we fit in the data frame that we had and at the end we get the model so it actually works beautifully and this model is inside our transformer that we produce so going forward with MS prime so what's still missing we still working on atom common operations so aggregator is a bit tricky joins are really tricky with my custom keys how can they make identify keys for example can we how to identify keys that would be great we do have some aggregations that we do right now with algebra noids that we love but maybe we can do a little bit better than this we want to wrap around existing spark code so we actually I miss I mean I didn't put it on my talk but it's also like a separate topic how we wrap around that we want to have code gen for Combinator's we're going to do again concern for markers kind of scary for me we want to do automatic feature engineering right why are we even doing this what if I just throw in the data and then it should just figure itself out like okay go figure out sniff around my data and say which columns worth mentioning without taken into the model right my example with Titanic I hand the picture and fields I can't do it for me why do we need human intelligence to look into the fields why can't even like take all the fields try all the combinations and produce the best model you say it simply completely yes but who cares I mean I wanted to work like this so this is something this is something we want to build further down the road we want to do sanity checking so just okay eliminate the features I'm not good and so on so key takeaways for you guys real life machine learning is really hard I don't know if you still didn't get it but I'm gonna repeat it it's really hard so unless you're you well if you just want to do some predictions you can go find with spark and maybe even Python and psych it and you're good but if you want to do it at scale as we do you're gonna have to invest a lot of time and a lot of thought into that and kind of this is the reality we live in spark Emily was great needs type safety I mentioned multiple times we want to have intuitive syntax for our code when we develop it we don't want users to remember special DSL what this function code I want to do dot completion I want to do dot and then I have one in my IntelliJ or it clips I wanna see the list of functions that I'm allowed to do in this context so scar has all relevant facilities I mean just not to use it educate your teams dig into Scala there are a lot of good books available and you you are the the gurus to to kind of to do that for your companies and then model arity usability is the key always remember every mom build the code so it will be reusable not just in our copy pasting code you kind of look into the core what are the concepts are missing so we saw forcible featured typed is missing so we went and implemented that and I'm furthers implications followed so try to kind of look into the root of the problem for further exploration look on our website Einstein calm yes we actually own this domain well no way we rented it for seven years prediction I all on Apache follow us with all the issues you have we have a lot of great stuff there to happen in Asia has a lot of templates or like we already implement it and we again once Optimus Prime is open sourced wide start we're going to start integrating and the templates into the Optimus Prime syntax so don't buy sugar bar today please recorded you can watch it later and then there is a talk on Optimus Prime internals version v-0 kind of v-0 mainly yes by Leah McGuire at Q Khan Q Khan maybe not but this is the link so you're more than welcome and then if you're really curious like really curious to know more find me or my colleagues or email us or whatever we're we're hiring okay thank you