SBTB 2014, Matei Zaharia: Next-Gen. Languages meet Next-Gen. Big Data: Leveraging Scala in Spark
I'm going to talk about basically you know doing next-generation big data using a next-generation language which is Scala and the way we're using it in spark so this stock will be you know it will not be sort of the usual introduction to spark kind of talk maybe you've seen such a talk before and I know there was also one on spark streaming yesterday it it will have a little bit about what spark is you know in case you haven't seen too much about it but then most of it will be kind of interesting things we're doing in it with Scala and this will include things that we're doing in the user facing API to let you have an API that's you know easy to program with and efficient at the same time and it will include some things that we're doing under the hood inside the project to quickly build out you know basically a new data processing platform for these clusters so that's what that's what I'll talk about so what is SPARC so you know overall it's it's supposed to be a fast and general-purpose cluster computing system that's compatible with Hadoop and the Hadoop ecosystem and it's supposed to provide a more powerful computation engine and it does that in two ways first of all it improves efficiency over MapReduce by giving you the ability to do computing in memory and also giving you the ability to do general graphs of computations not just map and reduce and second it improves usability through these functional api's essentially in scala java and python and one of the fun things about it is you can also use it interactively from the Python shell so on the first part you can get you know with the in memory if you use the in-memory feature you can learn as much as a hundred times faster than MapReduce based systems even on disk with the more general computation graphs you can actually see a significant speed up and in the second part you often write you know many times less code then than traditional MapReduce programming spark started out as a research project at UC Berkeley I was until recently it started in 2009 and we open sourced it and it's been going as a community since so last time I counted there were over 50 companies now contributing codes to it you know some of the ones are up there data books by the way is is a start up you know started almost exactly a year ago by the original spark team that continues to build spark and and also offers it as a hosted service so that's what we're doing and you know in in the past year actually spark has also become the most active open source project in the Big Data space more active than projects like Hadoop and storm and you know Hadoop file system and stuff like that and increasingly spark is not just a computing engine but a general purpose platform so on top of the core engine we've built this higher-level libraries for things like sequel to deal with structured data spark sequel is the new sequel engine that replaces you had we had a previous project called shark we've built a system for streaming which you saw yesterday graph processing and machine learning and the unique thing about these things in SPARC is that these are not separate computing systems that you have to you know go out and install that happen you know to talk to each other the actually just libraries built on top of the same engine so that means it's very both very easy and very efficient to compose them into an application and you know once you install SPARC you can actually on any of these things together because the underlying engine is quite general so this is just a graph you know to show some of the activity in the project so as I mentioned and in the past year especially we you know we had very fast growth in the community and we were by many metrics were the most active open source project in the Big Data space as I can see so this is commits per month or commits here for SPARC is is an entire patch it's not just a git commit and this is lines of code changed again in the past six months and you know we you know we have a lot of stuff going on throughout the project and then this graph here is showing the goes and contributor is over time and you know you can see even you know even though the projects become very active it's still going really quickly and if we have in the past few months we've had over 50 people contribute to the project each month and what's exciting to me about this is the activity and all the hadoop projects is continuing to go really fast it's not like people have stopped working on hive on mapping do so and stuff like that but with SPARC I think we're able to reach you know a wider community and in many ways were able to reach for instance data scientists were able to reach people who code in Python not just on the JVM and because of that I think we see we see a lot of growth and in the community okay so a little bit on the the programming model just to set the stage you know for the other stuff I want to talk about later so in spark the main idea you work with is these distributed collections called resilient distributed data sets or rdd's the resilient part is just because we automatically provide fault tolerance for them so you know we end up with this somewhat longer name but basically these are just immutable collections of objects in the scala api they're just scholar objects that are divided across your cluster and as a programmer you know these collections are statically typed so each RDD has a type of element in it and that actually affects what you can do with it so just as a one slide kind of you know what does the API look like basically you have you create this thing called the spark context and then you create rdd's format from input sources so for example we might say okay we have a text file and we want to read it as a bunch of lines and this is an RDD of strings and you know if this is a file in HDFS we might be reading it in parallel across the cluster then you do transformations and basically all our transformations are you know very closely matching the the Scala collections library so if you know how to use that you know how to use a lot of sparks so and you can just pass in a Scala closure in day or just a function in your program and we automatically like grab it and its dependencies and ship it to the cluster nodes and one one interesting thing with this is these derived data sets that you get from transformations are lazily evaluated so we actually wait until we see everything you do and then we can come up with an efficient plan to execute all of them and finally you can do these output operations called actions such as saving this thing as a file and that kicks off a computation and figures out how to deal with the lazily evaluated stuff so it's you know basically if you've worked with collections and especially with with the the lazy collections or views in Scala you've probably seen a bunch of this stuff on top of just the basic collection stuff as I said you have these rich libraries that you can actually integrate together to do different types of processing so much as already these you know can contain any type of Scala object our libraries are set up to just operate on on these rdd's and you can easily combine many types of processing that you know in traditional like an impervious big data systems required you to install a separate system so for example if you want to do sequel for instance to access data and Apache hive for sequel on Hadoop you have this thing called hive context and you can run a sequel query on it and we push that all the way you know to the cluster to where the data is sitting and you get back an RDD of whole objects and once you have these objects you can just start a map on them you know using the core API maybe you want to turn them and to say vectors to do some machine learning so you know have a function to turn these into numerical features from say these tweets that we have here and then you want to train a machine learning model so you know that's just a library function and finally you might want to take a stream a stream is similar to an RDD it's a type of collection except new data comes in over time and you might want to actually use that model so here we want to filter out tweets that that match you know that fall into cluster 0 according to this k-means crossing and maybe print them out so you know if this looks very simple that's that's good that's that's really the goal of it but the point of this is this kind of thing and in the past required you to you know install two or three separate systems one that can do sequel one that can do machine learning one that can do streaming and then figure out ways to hook them together and here they're just one one program you put together and you know the benefit for that apart from the programmability benefit is you can also optimize execution across these types of computations so if you use separate engines the more traditional ways of doing stuff you might do something like MapReduce for the extract transform and load part for just pulling in data you might use something different like say giraffe or for for for the training of a machine learning model and then you might use something you know a third engine like say Impala or storm to use the model to query data and using spark and and basically with these you know you have to read and write the data from a file system in between each step and you actually spend a lot of time just moving data around between systems not actually doing any work using spark you can hook all these steps right after each other not have the data leave memory for example between them and actually finish the pipeline faster and you can use this stuff from the interactive shell or you can just write the data back out and have a batch job and you know from my point of view the the the reason you know this stuff is is interesting is and and the reason we want to have this concise API is you know the basically to really make this you know Big Data useful to a lot of organizations you need to not just make it easy for you know for programmers to work with big data which is a lot of you know when you heat about MapReduce it so you know that now any programmer can build a distributed system easily but you need to make it easy even for people who are not programmers who are for instance data scientists or you know business analysts or people like that and these kind of very high level api's such as sequel and you know the machine learning library help you know help those those kinds of domain experts actually use these datasets so that's that's what we're trying to achieve so why did we pick Scala for SPARC at the beginning I'll just talk about this you know quickly because people you know people often ask so when we started the project this was in 2009 Scala was definitely a very much an up-and-coming language but it was not as well-established and as popular as now but the good thing is you know we were a research group so we just wanted to do cool stuff and in a sense if you know if people didn't end up using the system but but they learn from it and other systems were built in a separate way that was still a good result so but we chose scholar for three reasons so first one was just usability we wanted to have a very concise API Microsoft's link had come out pretty recently and we wanted something like that that lets you mix you know very high level operators with snippets of your programming language and very quickly write applications and we wanted the interactive shell and you know Scala was a language that had both of those the second really important aspect though was efficiency so you know for that first part you could have imagined using just hoobie or Python or something like that but when you work with big data efficiency actually really matter is a factor of 100 you know overhead for doing something may mean the difference between you know your query taking an hour or your query taking you know a couple of seconds so so that actually really matters and static typing is still essentially you know the best way to get code with predictable performance so it helps immensely to you know to control the performance to control the memory usage of code and with something like Scala you can get that and still have the very high level API in many cases the typing is fairly invisible to the user and as a whole you know doing big data stuff on the JVM has been interesting but we find at least at the JVM can can can give you a good balance between usability features and performance so by usability or utility features I mean things like oh I want to add codes to my system at one time that's that's kind of important for users like the interactive shell that I talked about where the user is typing the code you know after your program starts so and that's that's pretty hard to do in you know in in something like C or C++ to just compile it and add in some new code that easily at the same time though the way it executes is documented enough and enough hooks that you can have high performance code as well and as an example in in spark in the machine learning library we increasingly use J and I J and I to access fast numerical libraries and actually we found you know when we tune the algorithms in there they can be very competitive with with even C++ and MPI systems for these things depending you know what the algorithm is but for anything you know CPU bound we can we can reach the same performance so that's you know I think it's a it's a nice balance between these two and finally the third reason we went for Scala was the Java compatibility and that's because a lot of the existing Big Data ecosystem was in Java and using Scala we were able to just reuse big parts of that so one of the biggest things that we used is the Hadoop input and output API called input format and for the longest time the only IO code in SPARC was about five hundred lines of code that lets you access any Hadoop input format and with this we could talk to HDFS we could talk to HBase Cassandra you know MongoDB all this like zoom of of scalable data stories we could talk to all the data formats they support you know as people added data formats to Hadoop we were able to support them and so we could just focus on on building a compute engine and not having to reinvent all these things and this was I think one of the the key decisions that that led the project scale and and continued to be useful okay so that's that's kind of you know why we use Scala initially and I think everyone say ok you can do functional programming makes a lot of sense for a big data you know you want to pass a function to on your on your code so what I really want to cover though in this stock is sort of fun part is using Scala beyond these are V things beyond like oh you know we're passing functions around and you know maybe we have types and I'm going to go through four different places where we're using Scala in different ways some of them are user facing API some of the internal things but they show you a little bit of what we've been able to do with it and let's let us move quickly and and go to the project quickly so first one is probably the simplest one but it is important for users it's to have friendly types for things if you look at these big data processing systems out there actually even some of the Scala based ones are fairly loose about types so they you know basically you you you kind of have to keep track of the the schema of each row or each record of data as you pass it through and make sure you hunt the right operations on each column of it so they you know if you're not careful you might try to average a column that's something like strings or you know you might try to access a column that doesn't exist and you're not going to discover that until one time when when your program will you know will throw an exception so in SPARC we used essentially Scala implicit to make operations available on only the right types and this is just it's a really standard way of doing things it's used you know in many places throughout the Scala library but it is it is pretty nice so you know if you're new to Scala this is kind of the things that it lets you do so for instance in SPARC if if you have an ID D of key value pairs so here I make an ID D of these pairs of ins 1 1 and 1 2 you can own an operation on it called reduce by key to aggregate stuff for each key so here just adds up the 1 and the 2 you know together C so that's really nice but if you have an RDD that contains something else like just strings you can't actually run that operation and you get this at compile time you know you realize they're not key value pairs so really simple but this is the kind of thing that you know if you're hiding say a complicated like a big script or something like that you wouldn't see until you try executing it and this uses this color a feature called implicit so I think you know maybe people who worked at Scala for a while have seen it basically we have a class called RDD that has the methods on any RDD of any type so things like map and filter we don't care what what the type T inside it is we can just do it and then we have another class called priority D functions that has only functions you want available on rdd's of key value pairs and this is kind of a wrapper object around an RDD call itself that knows that the types are key value pairs and it has things like reduce by key and then you create this implicit conversion function which says if I have an ID D of key and value I can convert it into a parody D functions object so what that does is when Scala sees the line of code you know Rd d dot reduced by key it looks an RDD and it says wait a minute there's no method called reduced by key in here that's kind of bad but then it says okay but there is an implicit conversion from our D D to this thing called PR D D functions and that conversion happens to work because this is an ID d of key and value it wouldn't work if this was an ID d of just double or string or something like that so basically the compiler he writes the code as this and you can sort of extend your ID D class with new operations based on the type that's in there so you know it's basic stuff that that happens in you know another Scala libraries as well but it's nice that you know you don't have to do anything special for this we use this in lots of places we use that for the key value pairs I showed we use it for numeric types of things like things that might have a mean or some or things like that we use that for types with with an ordering to find some things that are comparable you can sort them by the key and then one of the nice places where we improved the API over hadoop mapreduce is we can also we use it for things that you can convert to hadoop slidable type which is just sort of a standard type for making you know for records that you can save into many Hadoop file formats such as sequence file so if you have you know types for which we know how to convert them all types that extend writable you just use your standard scholar types and we automatically turn it into these Hadoop types like indictable and text that you'd have to build by hand in Java okay so this is the the type part okay the second second example I want to talk about is a newer thing and this is the the integration between Scala code and sequel so actually a lot of the other things I'll show here light to this so I'm gonna step back a bit and say why you know why we're looking to do this so I showed before in the in the intro to spark I showed you can use sequel as an input source which is really nice from a lot of systems that talk sequel such as hive we also want to let you use sequel to do more complicated quays and aggregations on the data you have in rdd's and the reason for that is that it lets us optimize the execution of those better so when you hide an application using these functional operators I showed before like the map and the filter and so on it's it's nice you've you've written it down but it's it's very opaque to the runtime system because you just gave it a function and unless we go in and then try to inspect your function at compile time or at one time and essentially write you know a scala you know a compiler for a subset of scala it's very hard for us to understand how to optimize the death code so sequel is a really nice way to write you know many types of queries not everything but but many things that is you know now enough to allow automatic optimizations and it's very well studied how to optimize it and you know again because we want to target for instance data scientists or people who come from this you know less of a hard core like oh let's just write a couple of map functions background we also expect that a lot of users will prefer to write sequel to to writing you know just just general coded in many cases so one of the cool things we've done is using Scala reflection you can use sequel not just anon external data sources like hive but also on your our duties of Scala objects and you can get a lot of these benefits of an automatic optimization on them so here's how it works so in Scala you know you define your data types for example maybe have a class named person and then you construct an RDD out of them and here I'm just doing standard sort of spark stuff so I'm taking a text file maybe I lit things by commas because there's a you know a name comma age and then I create a person from it so that's pretty easy but then I can take this the set of objects and say oh just a stable and call it people and what will happen is at compile time Scala gives us a description of this class person so we know that it has two fields called name and age and so when we register a table we automatically have a schema for it where we know you know the name is a string and the age is an integer and then once I've registered my my people as a table I can actually just armed sequel on it the same way as I would on an external data source so for example you know this this thing maybe you know this is a simple query but maybe you know writing this way efficiently on the person objects would have required several steps of Scala or something like that and here I can just write it as a sequel statement and in the current version API the result of this is an RDD of these whole objects so it's kind of like you're you know like your JDBC whole object is just a general like data container and I can access each field in it by indexing it you know so fields zero so this you know so far it's good I've been able to run sequel but but the types I get back are not super friendly one of the really cool things that's coming in Scala you know in the future it's in development by a number of folks at EPFL is is called decorative types so using records in Scala you can actually also get type safe output from sequel so instead of writing you know T parentheses 0 and then presumably casting it as a string because I know that it's a string I could actually just write T dot name over here and the way this works is there's some magic involving macros that turns this into T 0 as instance of string actually so what's so this is a pretty cool thing so so basically the way this works is there is you know in in when this program is compiled there's a macro that looks at this string and passes it through a sequel parser and understands that okay name and and people you know will be you know will be a certain will be fields in this thing and will be of a certain type and then it actually lets me check this you know what this method name means which field idea it will be and get it back and if I do other things instead like a field that doesn't exist I will also get a compiler which will know because we look at sequel and we say hey you're not selecting anything called called foo so this is a pretty cool thing this is still kind of a researchy thing that's that's being developed at EPFL but hopefully something that we can use but even the first part with reflection is nice for moving data between the two worlds and when you when you do this stuff in sequel we automatically do a number of optimizations for you so these these are already in spark so first on the storage side we use an efficient column-oriented storage format which is often many times smaller than just storing stuff as java objects which is you know what we normally have to do if we didn't know anything about your datatype and it also lets us do stuff like compress your data and operate directly on the compressed data and stuff like that and then on the processing side we can understand all the Quai you're doing you know maybe you're selecting many things at once and turn it into the most efficient execution that we can find so do a lot of the stuff that otherwise you'd have to to figure out and and tune by hand so this is you know this is one way that we're letting people do more you know more sophisticated ways more efficiently on top of our duties so these are just a couple of the things that are you know user facing that you know are part of the API next thing I want to talk about is a couple of things we're doing inside that are pretty cool these are more you know kind of more involved for for people developing these systems they've let us build these kind of features very quickly and the first one I want to talk about is okay how do we actually optimize this sequel so you know I said that it's writing Quays and sequel is is nice because it lets up come up it lets us come up with an efficient execution plan but now as the SPARC developer is we have to actually implement that and it's actually not not that easy to build you know an efficient kind of sequel execution engine so we and we built SPARC sequel execution engine and in particular the optimizer in a you know using a very concise and extensible framework that we developed called catalyst that builds it in a functional way and what's really nice about this is a lot of the stuff that normally you know in a database query optimizer would take many months or years to develop you can express very quickly here and you can iterate quickly and you know and and improve the kinds of optimizations that the system can do and if you you know I'll show exactly how this stuff works but if you know a little bit of the history of functional languages and especially features such as pattern matching a lot of these features were actually developed to make it easier to build compilers so it shouldn't be a surprise that building a you know essentially a sequel compiler becomes easier using a functional language so let me explain a little what happens in quei optimization so sequel unlike Scala well unlike certainly unlike something like Java is is a declarative language which means that you express what you want to compute you know like the sum of these values group by this but not exactly how to how to compute it and that means that the runtime system needs to pick a good plan a good execution strategy to actually run it so just as a really simple example I'm going to have a extremely you know a minimal kind of sequel query so this is gonna be okay I have first I select the ID and name from a table called people and then I select the name from this result we're idea is equal to one this is kind of a weird way but it's just meant to be kind of a toy example that still shows something we can optimize and you can imagine this would happen if this was the result of some subway or some function like that or even the result of like weeding you know of you or something like that in sequel so you know basically we pick these two guys and then we pick only the name where ID is equal to one so this is the the you know in in a database the first thing you do when you see this is you create a logical plan here you just chained together the operators you want to do and and don't worry about reordering them or optimizing them in any way you just want something that has the same logic as that statement gives you the same result and the most naive way to executed then would be to just execute each operator right away so what this means here is okay we have a table called people let's do the first select this is called the projection operator so let's just project out these two columns ID and name let's filter it on ID equals one and then let's pull out the name so that's that one you know one thing that you might do to run this way now if you wanted to optimize this you could probably do more interesting things so for example if the database had an index in it an ID you could just say hey let's look in the index for things where ID is equal to one and find a pointer to those records and return the name and that's a much better physical plan but unfortunately dealing with all the possible logical plans and writing some code to come up with the physical plan for it is pretty hard so you can't just hard code all the possible things like oh you have there's a project and then a filter and then another project we should do an index lookup in in this case so one of the common ways to improve this is to write simple rules which are small changes you can make to the query plan one at a time that if you keep applying all these rules end up with with an efficient plan and basically if you just write a collection of rules then and keep applying them to the plan until no more rules can be used you end up with a you know with something efficient so here's an example of how we might get this this execution using rules so we might say okay here's our original plan you know we have the project and the filter and let's just apply one rule so one common rule is called filter pushdown so basically the idea of that is we notice that we pull out let me see we pull out idea name here and then we filter on just ID so instead of actually loading all of these immediately from you know from the underlying system and keeping all the IDS and names in memory and then throwing out you know many of them when we check out the ID we could instead move the filter down here so we could say hey as we reach people let's filter out only the ones with ID equals one and then we do these other things to project out the name and in this case we can swap these operators because the the objects used in the in the filter are all you know all things that were projecting basically so so it's okay to move the filter down down there okay so this is one you know one optimization next we might do okay there might be a second dual which says hey if we have a project and another project we pulled out some subset of the fields idea and name and then we pulled out another subset of that it's enough to just do that one so we don't need to run two of these at the same time and finally there might be a thing that says hey if we have the table called people and we look at only the ID field we can turn that into an index lookup instead and you know for this kind of pattern we can just return the neck so this would be fields that let us get this there's a result so there's been so the question is how do you actually build a system to do this and there's been a bunch of work on making it easier to write query optimizer is you know the old-fashioned way is you actually write a bunch of Java code or C++ code that you know implements each rule and looks at up at objects and tries to transform them and so some of the previous systems that did this are things like volcano and Cascades that actually built a separate programming language for expressing these rules and then evaluating them and you know that's that's you know it's it's it makes it easier to write those but it means the the database developers have to learn this new language and if the language isn't powerful enough to express all the things you want to use in your rules then you know you might not be able to specify things so in catalyst which is the execution engine on on SPARC we do all this stuff instead using Scala and we do it using pattern matching and functional programming in Scala and we get to use the full scholar language and specifying the rules and also in in transforming these you know these execution plans so the way it works is this tree node library that we have in there so the operators and the logical plans I showed and in the physical plans are these form these trees that you can hook together and on these trees we have a standard sort of collection functionality so you can just run functions on them to inspect the tree we have this transform function which lets us apply a rule and modify the whole tree and we also have a lot of the nice features these tree nodes are just scholar case classes so you can automatically print them out nicely you can hook them together you can serialize them you know send them over the network all that stuff you get because they just come from Scala so let me just show how the three transformations work so basically you express a hole in the catalyst optimizer you give it a partial function from three nodes to form a tree to another tree essentially and we look at it and we say for each operator in the tree we say okay if the partial function applies to it which means that it has a pattern in there for matching and on this you know on whatever we have in that operator let's run it and we place it with the result if it doesn't apply we we just don't change it and if we have a bigger tree we apply this recursively at each operator in it - you know - until we we we run out of places where we can use this so just as an example of these you know of one of these let's look at the filter push down think we did before in that one we said okay let's look at filters that sit on top of project operators and see if we can push them down so we don't have to project you know the data before and basically we to do that we want to check that you can evaluate the filter on the table that's underneath here so without without the result of the project so without well you know if we create new fields and here or something like that so we want to push it down underneath that and if so we want to switch out the operators and move the project on top of it okay so this is this is kind of the whole and so this is the the code in you know in in catalyst and in sparks equal to do that and I'll walk through this code a little bit but you know it's actually not a ton of code and it's basically one of these partial functions so what we do is we have here query plan is our initial tree and transform is one of these that we called and the body of transform is a is a scarlet partial function so this is like a function except it has it may only match certain types of inputs so it has these case statements and here and on other types of inputs is not defined so what we're doing and this one is just some pattern matching so we say okay we look if we want the we look for it you know that's a filter we're just gonna call it F and this is actually like the condition of it so we don't care what it is but we care that it's child is a project and the child has some kind of grandchild the grandchild is like the table scan that we had before so this distress means that there is a filter above a project and then we do an if statement so we we say okay if the this filter only references fields that are already and the grandchild so not new fields that I introduce somehow by the project then it's okay to push it down we don't need to do the project before and this is how we create the new object so basically we take the project we move at first and then we make the filter be its child this thing if you haven't seen it it's just you know a way to copy a case class and change just a few of the fields so so then we just switch the order so it's not you know basically it's a very concise way of writing this rule and in in spark sequel we can just have an array of these rules and just keep running each of them and applying them again the the stuff looks really small too to give a sense though of how things work you know a lot of the holes that we hold in Sparks equal using you know this this sort of an amount of code took sort of hundreds of lines of code to do before and say the hive query optimizer or systems like optic or things like that which are which are the open source query optimizer so using this we've been able to build out the set of optimizations in spark sequel very quickly and we've also had community members just come in and implement you know specific optimizations without having to know a ton about the rest of the system so it's a really nice way to compose them ok and then final thing I want to talk about is also from spark sequel it's code generation and this is another thing that traditionally has been very time consuming to to build an in a database system and many sequel systems actually ended up not building it but that we've been able to do in a very powerful way using Scala so if you look at sequel on Hadoop systems in in particular many of them just evaluate expressions that they see in your equation so if you do select a plus B from table they actually build a tree that represents or a and then a plus and then a B and they interpret that tree on each record that you eat kind of like a a programming language interpreter but evaluating stuff this way can be quite expensive especially on the JVM you get these operators that are you know there are subclasses of some common interface so you get these virtual function calls and the inliner can't always you know inline those in an efficient way because there can be many implementations you get branches so you look like oh is this applies is this an if statement is this a multiply and branches are much more expensive than just executing you know code if you have primitive types so like integers you kind of have to have everything into an object if your system also supports non primitives so that adds a bunch of overhead and these things consume you know a lot of memory in addition to creating objects and as datasets move more and more to being in memory or an SSDs or things like that the CPU overhead matters in the systems like hive the CPU overhead didn't matter initially because it was completely disk i/o bound so it didn't matter that you spent time executing these things so just as an example you know to look at this interpreting a plus B kind of example you might have you'd build you know a class that represents the AB instruction and then a class that represents okay read field a and each field B and then you on each record you'd if you'd call evaluate on this guy and this one would say okay let's do a tart eval gives it gives us back an integer except it's wrapped as an object let's do B as that eval gives us back an integer now we pulled the integers out of those objects and we add them and now we have to hey turn it up so you could easily write kind of a bunch of classes to do this but it wouldn't be very efficient so there's you know a standard approach to do this in in in in database systems which is code generation you look at these expressions and you compile them to native code or you know if you're on the JVM you can compile them to Java bytecode which quickly after becomes native code and then you can evaluate them quickly but the problem is implementing code gern you know from scratch is is usually a many sort of person many years engineering effort if you want to do it for a big subset of sequel and we kind of needed to do it from scratch because we deal with a lot of interesting data types and functions that are not in standard sequel systems so as I showed we wanted to let you just take in Scala objects as the input you know that you want sequel on we actually also make it very easy to register functions from Scala into sequel and call them under and we have these weird datatypes like structures and maps and stuff to deal with things like JSON data so we couldn't just grab an existing you know sequel system and grab the cogeneration from there so there's a really cool feature based on Scala Macker that actually makes it possible to do cogeneration easily and that's called quasi codes and I should say by the way I think I forgot to say it earlier but for both this and catalyst the lead developer is actually Michael Ambrose who works at database but it's he he built both these things you know actually in just the past six months he he figured out you know both these techniques to make to make this stuff on well I forgot to say that before so any so so this so as I said so cogeneration is what you want to do and there's a really cool feature in Scala called quasi quotes that that lets you do this and this is a way to essentially stitch together bits of Scala code after one time and then compile them and it's enabled by Marc by macros so here's how you do cogeneration you know for for an expression if we wanted to support just addition and say integers say integer fields so you you get you know you get just an expression this is like my class that maybe add and you know field access are are subclasses of and then here I'm just going to do pattern matching and on PharmD from this thing you can return back what's called the tree this is a Scala compiler to e basically it's it's a it's an expression in Scala and you do that by just writing the string with a queue in front so some scholar code that can also grab fields from outside so here for instance when I have an attribute and ordinal is maybe its name yeah I can do you know get int as that guy and when I see an ad I can I can write code that okay I generate code for the left operand generate code for the right and we turned them back and what's really cool about this is these three objects get composed together at the one time and then pass to the Scala compiler and whatever expression they had I got a scholar class that has the right code stitch together for it so instead of messing around with like okay I'm going to build my own compiler and deal with with subsets of sequel and stuff like that I can just reuse the Scala compiler in a very nice composable way to get this and then the end result is you get code like this that's actually really efficient that has the right types you know just adds them in line no virtual functions and everything is in line and we were able to build the first version of this and just you know a couple of weeks and and deal with many of the expressions and this is a gun a place where we've had lots of external contributors who you know maybe know very little about how the rest of spark sequel work but are able to put in optimizations for the pieces that they care about and in terms of performance it actually sees significant performance gains as well so shark is our previous system that used interpretation and spark sequel with code Jan you know and in many ways is actually quite a few times faster than this one and this again would have been very hard without without using a high level language okay so that's kind of what I wanted to talk about hopefully is given you a sense of some of the cool things you can do both in user facing api's and internally and just a few slides on what's next so we think there's lots of opportunity to do more with Scala in the Scala API some examples of this are scarlet team is spending a bunch of time on serialization on making it efficient to serialize objects and send them over the network or making it type safe for instance to have functions you can safely send to another machine and this will be great to incorporate in spark Scala async is a cool way to launch asynchronous jobs that we'd like to support to let you earn multiple jobs at the same time and for the math stuff compiling numeric expressions to you know fast C code is also an interesting thing that we'd like to do and Scala 211 is not yet supported in spark but it's getting really close so hopefully it's it's an active development and hopefully it will be out in the next release with all the the cool features of that if you want to learn more about spark there's a lot of info on our website and this is a thing you can play with on just your laptop you don't need a cluster of machines to do stuff so if you just want to play around with it it's it's it's fun to do and we at data brakes have made virtually all our training resources available for free after the fact the only thing you don't get as an instructor in the home with you but you can see videos training exercises on the SPARC summit website which is our latest conference so hope that's given you a sense of some of the things we're doing in SPARC we want to give you a next-generation engine for big data and we've been able to do a lot of that in interesting ways with this next-generation language and hopefully we'll see more of them in the future so if you want to see some cool uses of Scala you could definitely check this out either as a user or as a developer and the final thing is that if you liked any of this stuff if you want to work on this kind of stuff we are hiring so definitely reach out to us about that so thanks