SF Scala @Spotify: Neville Li, Macros in Data Pipelines

SF Scala @Spotify: Neville Li, Macros in Data Pipelines

Recording: SF Scala @Spotify: Neville Li, Macros in Data Pipelines

let's make this pit bull all right who's heard of macros scala macros it's an experiment of mental feature that's being introduced into point 10 I think it's going through constant changes but so macros anyone here has written any C or C++ of course everyone has so there's def ifdef those are like the primitive form of marbles you it's basically a glorified string search and replace and of course some of you might have heard of closure or used closure another programming popular programming language on JVM and also functional and macro of course is a key component enclosure programming and closure programs are closed editor strip goodness structures so you manipulate this expand them so basically taking some form of code generating some other form of code the world's most popular macro engine is probably PHP taking some form of spaghetti code and puking out HTML so everyone has some xcode exposure to macros at some point so I'm talking about a very good use case that is helped us a lot so this is the slides I used for the end northeast Scott a symposium talk back in February like right before the snowmageddon so I guess most people there was suffering from brain freeze and not remembering much so it's okay I'm gonna just repeat myself here so yeah I woke who is the rock music recommendation team has qualified and in there for almost four years actually only started using scholars since 2000 searching and start with scouting and expanded to spark also working on stone see the pattern there like exclusively free motor starters s what are symphony back in the days anyway so we use cutting a lot and there's a very popular data combo processing combo so parque open source library storage format for HDFS from Twitter cloud collaboration between Twitter and Cloudera so basically instead of saving us storing your data is like a row major like row after all and to read data like you say in case where data is not your roads are not of equal size then you can just jump to certain offset and randomly seek particular record you have to read or skin through all the bytes to to to just read a subset of the data which is pretty inefficient parque de idea came from the database word relation database work basically story columns values in a single column together in what what they call the row groups so say 50 values of column one then 50 values of column to the nice thing about that is you can skip column groups pretty efficiently you can estimate how far ahead in the file you want to seek and that's a huge performance boost especially a hadoop where you scanning through terabytes of data but commentates like our key log format which a lot that locks use a plate playback music which has maybe 50 or 80 columns and most cases we read three or four columns and that in most cases is a ten times speedup so parque is a very nice to for doing those kind of big data processing there's also a bro which is a schema or data serialization format so you have a schema file that compares to java or some other programming language steady class code with headers in Gators and setters you also have type safety from you have data types like strings integers lungs boolean also notable fields and also erase and maps in a row it's a very popular it's beauty into a lot of the data processing engines like five Scott Ian spark Impala those n spark of course so we use these together Park a library supports reading and writing data is every representation inside of Java or scholar code and then there's the processing and framework which can be scouting or spark in your case a scholar processing free mug so we use these three toes together but there are some problems mmm just look at the schema first so this is a typical pocket schema you have a java ish and like and to maintain River reversed two main aim namespace and then you have fields in the types this is what it looks like in a pipeline code so you can say read from this account class that's compiled from schema and you have gathers which are typesafe can name and amount so one of them astray in the others probably integer you can do the reduce since like that pretty nice for parquet there are a number of features that helps with performance so Colin projection basically selecting the cons you want and it will be able to seek in a file and only read bites I want and skip the column so you don't and that's a performance boost the other one is called the future predicate so because it starts rose groups of radios together and each control group has many maxima and minima index kind of like index in a relational database so you can say i want user who listen to attract for more than two minutes and if say a 50k block of log files has no such thing and in the mix minimize lower than that maximize lower end up then parkway will be able to skip the entire group which further speed up scenes this is super helpful in ad-hoc queries like searching for particular subset of users or a subset of data so yeah it used to combine can often offer like 10 times speed up the problems alright so this is a native code as we have seen before in pipeline you get is data a roll class with schedules that's type safe and also you get Auto completion from the IDE whereas parking use strings to identify columns and also have support nesta columns so it is like you know user dot name or users or email things like that so you can have nestle colon accept the same string so you have to type it all you have to know the scheme ahead of time yeah that's her up from and also it's unsafe the compiler is unable to check for typos there's no autocomplete so you end up typing and enjoy your fingers also a arrow has this convention that changes field names into this cameo case so it can be confusing Sami uses my accidentally used camera case in the field schema and end up you know all sorts of typos and confusion so that's another problem potential problem so to do this correctly you need to have the schema on hand while radical that's pretty bad and also it's hard to migrate existing code so we can just run them as this pretty good is even worse so type top predicate that's like a scholar collection type filter you have you know simple boolean logic and can chain them parque predicated has this uses this so-called visitor pattern so you construct a free yourself using this kind of prefix notation so you have the function and with two arguments first one is the column name of course you need to know both the type and the name ahead of time column two is the value again you need to know the type of the value and in case of Java primitives because scholar again trying to fix Java and failed so you end up with this kind of type system full bar and you have to do the conclusion to make it work so that's pretty horrible it's like closure of awards so yeah this is where Marco you know shines because it solves this exact problem by eliminating all these human errors so just go back and here we have a typical expression you have some anonymous argument underscore which represents a mug avro object and then you have gadgets which are just you know selecting a field and calling a method and then you have a predicate some operator and value internal discompose do something like that so X dollar one some internal representation of the anonymous variable or value not get measured names and then in parentheses for calling it and they actually touch greater than it's also a method or select in this case in the internal compiler bingo so it looks like that and also Scala takes care of the the boxing of primitive types so that's why or unboxing because gate amount returns java.lang.integer which is a pulsed object in Scala compiler convert it back to a primitive int this is what it looks like in a rock hold I mean you don't have to look this but as someone who deal with macro it's a good for for debugging so basically selecting is the dot name part and apply is calling some method or function and of course a constant chain that's a literal and then you have list of arguments in this is a single argument 10 so everything boils down to something like this apply select apply Sharia so on and so forth and then constants and lists of arguments so it looks complex but in fact it's pretty easy to understand once you get the gist of course to deal with code like this we can always use recursion and Peter mentioned two key features in scholar so the microcode could kind of look like this in our projection case we are saying I want to project on I've row type T with whatever selector like t get target value get a month so the user supply a bunch of such Gators which expressing the form of lambda in the macro case they are passed to the compiler during compilation it's the abstract syntax tree that's hence you have the macro implementation and you have a seedot expression which is the expression of the coder user just typed in the in the top scholar file and then side user type underscored okay your name or get a month that's passed into the macro implementation applying power there is see the expression and then you do peito matching and recursion on the tree and you generate new trees predicate is pretty simple similar except the input type is some function some lambda that takes the arrow type T and returns a boolean so you can do all sorts of boolean predicate or chain them with logical operators like + or not etc the return type in this case is a future predicate which is a class in the park a library so this is what the code looks like just a lambda that returns a boolean alright so there are a bunch of things I have to do so that too in order to make the code looks nice and intuitive of course I mean this this user code is parts to the compiler is a EST so I mean unless you expect what use is gonna input and deal with them otherwise there's New York you can you can you can transform a tree that you don't expect so there are some limitations for example their cases ordered the party API you have this operator and first argument is the Colin named second argument is the value one compared to but user my flippin so a greater than 10 and 10 less than a that's equivalent in normal Scala code but not in parque API so I have to recognize these cases and flip them of course primitive and boxed values that's another Java problem both types can be no while scholar types will not be now the unified scala int will not be now but a java integer camino so these are cases i have to deal with again numerical type of cohesion like you know a dog get amount where kate amount returns integer you can compare it with long type which usually works if you type error right standard java or scholar call the compiler will supply some implicit conversion but not in this case i have to deal with it manually but it works again at the last seen as boolean so the park AAP are you compare bullying with you know you still have to do the Eco caller name boolean value but kept bullying is a valid expression in Scala or Java is a evaluate evaluates to a boolean type so you have to deal with that it and expand it to you know true or not there are few things that doesn't work but that's just the limitation of you know using parquet with a bro there's a park a ticket so this for so right now it doesn't have any predicates array so you can say I want a filter our records with this array or repeated filled with size of certain value or just eat inspect each individual items in all right so that doesn't work also you can filter your fantastic record like the record every the entire record is no no no so that eco operator only works on on a leaf node level so that's another limitation and again arrow is not really designed to work with pockets of shoehorn in this case to maintain the backwards compatibility with legacy code base so once you do parque projections say I have 50 fields and I said that two of them the rest of the four da fields the method will still exist on a parquet on every object there was still be get something that I don't want but it will return now and when try try to serialize the charger like writing it back to this code to a chef or drawing not serializable exception for everywhere the serialization format is funny like it funny that it's like one of the most hard to serialize java type but yeah anyway so this code is open source we use it as qualify in most of our pipelines with parquet so you can look it up on github and that's it yes yeah the question is if I would recommend Park a way several i will recommend parque and depending on a framework of choice if you use a spark or hive or maybe presto i think its built-in support and also in paula has built in pocket support in those frameworks you don't have to use a rope because they they treat parque fields is like part of a sequel query or you know data frame api you treat them is treated field names as strings and you don't have to deal with arrow and they work really nicely if you wanted to use them in a type safe manner like doing the work the one I'm showing it's not a bad scene but there are no good alternative this photo buff and stripped but there are no parking to protobuf us maybe this drift compatibility layer but not protobuf so you have to kind of you know experiment yourself so that the ideal solution in my opinions something like sparks eco or data frame API of course it's not it's not type safe by its more more angel and also the execution planner will be able to do some type checking before you know actually perform the query so you're saying that when you take the parkade filters or whatever like that you get them into the abstract syntax tree if you mash it with pattern matching stuff like that have you looked at using positive owning yes so there are two steps go back a little bit that's exactly what I'm doing I specialist look in the code so apply to predicate which is the map Marco implementation and you can see a lot of pattern matching a case apply of something extract the list of arguments in slider so the first step is of of course to understand the abstract syntax tree and extract you know names of methods and types from the tree and the next step is to generate a code and that's where we're all these you know chief you or whatever any any q those are quasi codes so it's a lot easier to to generate code with cody constant constructors syntax trees yourself like apply many parentheses it's like writing closure or game box I have not been it I don't have anything against culture it's pretty nice knowledge but in this case you probably don't want to do that oh I forgot to mention there are they feel like even nicely use cases for macro and a lot of you are trying that I'm actually playing around with it and hopefully it was have something to show like a lot of the seco like engines you know I think slick actually does that so if you perform Seco of course a return something I could roll that's untyped you have to access fields with numbers or strings and written objects that's kind of you know painful to use and I think the spark guys and also some Google people bakery and also some sleep people on a dbms stay they are working on those kind of macro so you can basically at compile-time generate a return type based on your psycho query because you already have the schema and that's a interesting use case I think there was a talk at the PDS think row Norris he's doing something like that so using macro with a DBMS so you