Scale By The Bay 2018: Bryan Clapper, Introduction to Apache Spark with Frameless
Recording: Scale By The Bay 2018: Bryan Clapper, Introduction to Apache Spark with Frameless
all right my name is Brian clapper I work for data bricks if you happen to be at Scala IO a couple of weeks ago this is pretty much the same talk I gave there in fact they gave me 50 minutes I forty minutes here and I haven't cut any Content so we're gonna see if I can do a hundred percent of the content and 80 percent of the time so that's a put your seatbelts on let's see what happens this despite the fact that we're talking about frameless in here to some degree there's no category theory in here so if that's what you're expecting I apologize frameless does use a lot of type level programming but mostly this is about using the api's not you know how are they programmed if you want to talk more about how their program there's a young man sitting right here who can help you raise your hand long so basically it's if you don't know anything about Apache spark hopefully you'll walk away with an understanding of spark when you at the end of this talk at least an initial one what the intent here is to compare and contrast the standard spark data frames and data sets api's with the type level frameless api now this is exactly what one of the pages in the get book that frameless publishes does but I'm gonna do it in a slightly different way in fact I'm gonna do it live coding mostly in a day to Brick's notebook so we're not gonna be looking a lot of slides and these notebooks are actually out there for you to load up and use if you want I'll walk through that really quickly so again as I said it's it's oriented more towards learning how to use these things than how are they actually implemented though I will talk about some tuning measures and things like that under the covers alright and hopefully at the end as I said you'll own you'll walk away with some understanding of spark if you don't already know it and you'll have some idea of how frameless compares to the native api's and so this is the last slide this is not slide oriented there's a github link here but if you forget that really really long URL that's fine just go out to my github repo BMC is really easy to remember and I've pinned it okay so it's actually pinned out there and you can go find that repo pretty quickly and the notebooks are out there the readme instruct you as to how to load them up and of course if you have questions you can just send me you know an email and say hey what you wrote makes no sense can you tell me how to do this so all the code is available for you to look at if you want including and the data is available you can download your own copy of it so let's with that said we pop over here first notebook that I've got is one that I'm not going to go through but this this one is in the the set of notebooks and this is how you go get the data basically I have a scholar server out there that reads from the Twitter firehose which is appropriate given where we are that wasn't planned and then dumps it into reads a certain amount of that data down and then dumps it into a park a file for analysis so you can create your own copy of that file by just running the stuff that's in here and then there's another file that I pulled down only for doing a join against as a demo that's Twitter data from like February so we can do a quick historical analysis that's and it's mostly just an artifact that can join against so you run through this notebook it has explanations as to how to get the the data down and where you can pull it from so if you want to run this yourself that's what that's for where we're gonna be spending our time is in here so I walked to the back I can read this with my try Foucault's so you guys should not have any complaints but if you can't read it raise your hand I'll make it bigger so let's just dive right in that's my three minute intro so I should be able to fit this in the remaining thirty seven minutes so the first thing we're gonna look at is the data frame API okay so you may have heard of the RTD API this is the older API in SPARC that looks almost exactly like the scholar collections API and that's not what you should start with these days I mean there are arguments for using the RTD API but you should always start at the higher level with the data frame or spark sequel API is because you're gonna get better performance right and we're gonna talk about that with a little bit but one of the things you don't get is the kind of type safety you want which is sort of the focus of this talk so what's the data frame API and I've already primed this a little bit I've got some shared deaths here this just runs another notebook to set some stuff up and I've already loaded this guy up so there's my file full of tweets I ran this this morning it has about as you can see here it has about 50,000 and change tweets and again this is sampling right it's the Twitter API that they're not giving me everything but it's relatively up-to-date so here are some of the things that we're seeing so the the data frame API is a DSL so it implements a query language it looks a lot like sequel you know at the programmatic level in fact you can actually substitute sequel here using the spark sequel API and what it would work exactly the same way and it's about as typesafe as sequel as well so here what I'm gonna do is I'm gonna select you can see up here that the hashtags column is actually an array right so what I'm gonna do down here is explode this so that this this particular row has four hashtags in it what I'm gonna get when I exploded is four rows with all the data duplicated but a different hashtag in each row okay so it's like a pivot so that's relatively easy I'm cashing it just to make the performance a little bit better we're gonna select an explosion of that and we're going to call it tag and then we're gonna group by the lowercase version of the tag okay looks like sequel right we're gonna count the number of tags how many how many how many of each tag do we have and then we're gonna sort in descending order so I can see what the most popular tags are and this display function is a day to brick specific function with normal spark you would just use dot show but you get a sort of us a an ASCII display here I get something a little easier to read and I have no idea what half of these things mean I don't does anybody know what daddy's girl on fire I have no frickin clue but that's those popular hashtag at the moment apparently so you know the query language is is reasonably powerful and as I said it's quite reminiscent of sequel which is deliberate so let's take a look at some some hashtags we're guaranteed to be there no matter what is going on given the current political climate pretty sure I'm gonna find some tweets that match these hashtags right now note that I in order to cover upper and lowercase in here I I have to actually spell it differently it's kinda it's actually remarkably difficult to do the kind of transformation I want to do within this array contains function so this is kind of ugly right we can get around this a little bit later I talked about datasets right so here we're gonna we're gonna select these tags and we're going to we're going to actually filter out so that I'm only getting tweets with these four tags and then I'm only gonna select these four or five columns out of the data frame out of all the columns that we get so how many did we get well there are 27 tweets from from the 50,000 or so that use one of these tags and if I display them you can see what we've got right oh look queuing on Joe's up right away isn't that special okay so I'm not gonna actually look into this it'll probably just piss me off so let's take a look at the schema for the data frame okay so this is the original data frame up here what's really happening is this is kind of lazy so tags is really just another data frame right so so applying these transformations the filter operation returns a data frame which I'm not storing anywhere right it's transient and then the Select operation returns another data frame these are all lazy really what's happening under the covers is each with these transformations is adding to a query under the covers so you're building up this query AST you can even throw sequel in here in the middle and we just add to this query AST and then eventually when I run what's known as an action and display is an action right display says you got to go out and actually read the data now and then pull it back so I can look at it that's an action that actually kicks the job off and distributes it across the cluster causes the calculation to occur and then pulls the data back right so really briefly I'm not going to go too much into spark architecture but really briefly there are think of there being two components here there's what's known as the driver that's the thing that's running my code here and then when you bring up your spark job the driver gets allocated within the cluster executors these are other JVMs one typically one per node in your cluster as a typical arrangement and those executors that come up are dedicated to that driver so your spark program actually runs in a combination of JVM those the driver which you can think of as the orchestrator and then these executors that are out there and what the driver does is when you kick an action off it says okay time to read the data we're going to do this thing let's break that data up into pieces okay call partitions and I'm gonna sign some subset of the partitions to each of those executors and then they're gonna process those partitions in parallel so that's the first level of parallelism you've got three or four or eight or however many individual nodes out there with executors and then within each executor you also have a thread pool so that's your second level of parallelism right and what will happen is that an executor says okay I've got these thirty partitions I've got eight threads let's say in my thread pool that means I can try to process eight of these partitions at a time and then it will it will iterate through those partitions a thread will be allocated a partition it will process that partition start to finish until it hits something that causes it to stop SPARC calls that a stage a stage might be okay we're completely done with this it's time to write it out right or a stage could be okay I've reached a point where I now have to share this data across the network a good example of that as a sort or a group by right so now we've got to move this data around the network so that thread will run up to that point and then stop and say okay I'm ready for the next partition so that's the thumbnail sketch of the parallelism in SPARC and so that's what's happening in here so let's sum so this is the original data frame and notice that this this is a schema I mean it kind of looks like a sequel schema and we can see that the first column in fact is an array we can see that this one's actually struct you could imagine reading this in as JSON which in fact it is originally and this thing contains sub elements so I could say place dot name and pull out the you know some of the location information here's the problem with data frames here's why Scala people hate them they're not type safe so the timestamp column is a sequel timestamp you'll notice the schema here says it's a time scan stamp this corresponds to Java that's equal that time stamp okay so this makes no sense right really you can't really multiply a Java sequel timestamp by a thousand this will fail but it fails at compile time run time not compile time right now if you're running inside a repple like this you don't really care that it's failing at run time but if you're actually compiling your code and preparing it to deploy you would really prefer that this be caught by the compiler or by circle CI or Travis or what you're running and that wouldn't happen you'd have to actually try to run it so this is actually you know you can see its run time because you get this lovely obnoxious stacktrace right it's actually worse than that sometimes you don't even get an error what does it mean to compare the user column which is a string to ten I don't know but it works so why does this work this is ridiculous okay well it doesn't bail okay but right and this is even worse what does it mean to divide it by a hundred I don't know but what do you get out of it you get No okay so this kind of sucks right like I can't I have no protection here against doing something ridiculous and this is clearly ridiculous so and then there's also this like what if I select a column that doesn't exist that's a runtime error all right I don't get that at compile time I have no protection the compiler can't help me so of course Scala people hate this so and then it's worse than that right because even if you get everything right now collect as an action so I'm going to take the tags remember this whittled it down to five columns I'm gonna pull this stuff back now what I pull back let's take a look at this so this is on each collect is pulling back rows so eat I've got an array effectively of row objects but the problem is what is a row object it's a heterogeneous collection okay the first type is in second type is so what do these things look like at to the compiler well let's assign it to a variable and you will see it's any so now if you want to do something useful with it after you pull the data back you have to do you either have to do stuff like this this should look remarkably like JDBC the API we all love to hate right so you have to cast using these built-in things or you have to literally do it as instance of to pull this back and make make something meaningful you know back in the back in the driver and of course if you get it wrong you get a runtime error not a compile time error because it's cast all right so one way you can get around this is to use something called the dataset API what's the data set API well so let's take our schema and recast it as some case classes that's a reasonable you know a reasonable layout for it okay this is these are type safe versions of the schema so what I can do is take my data frame and cast it now I'll get a runtime error here if I've chosen case classes that don't match up so what do I mean by match up what it's what's going to happen is there's a certain amount of reflection going on here and a spark is going to look at the columns and match them up by name to the to the values in the case class the fields in the case class and it's also going to verify that the types work properly and this is exactly a caste if I screw this up and I'll show you that here's one where it's bad okay so these are wrong wrong types this is the wrong type okay and if I try to do this caste notice I'm getting again a runtime error but once I get past this runtime error I've got this this data set so I'm gonna quickly throw this in here let's take a look at the type note that the type now is data set of tweet data it's no longer data set of Rho so now this thing has a type that the compiler can actually help me with nevertheless I can still use the traditional query API against this if I want okay I can just I can treat this the same way I use a operate on a data frame and in fact in Scala a data frame is nothing more than a data set of type Rho it's a type def it's just a data set with not a very useful type so I haven't changed anything here I would still get a runtime error here if I gave it the wrong column okay so I'm still not getting any compiler help with the query API but where I can start to get compiler help is by doing stuff like this here's a traditional filter right this looks like the kind of thing you would use with the collections API and so now I can say look I'd like to I'd like to look at only the rows that have hash tags so hash tags non empty boom this works perfectly right and now if I change this to the wrong thing I'm gonna get a compiler error of course because it knows the type should probably kill those lights if we can and if I display this this is where we solve that problem from before right where now I can actually do it to lower here so I don't have to have five different array contains I can just do do the one thing here so great we have our types back right isn't this great now the compiler can protect us you know now if I try to do a comparison of timestamp you know I get I get compiler errors if I try to multiply it by a thousand I get compiler errors and when I collected the data back I actually get back data in something that is useful to me it's an array of tweet data no more casting I did my cast up above but now I've got my types back this is awesome right so why don't just stop here why do we have to talk about frameless we can stop here right this is great we're done we're not done pret datasets have problems okay there a couple of problems with datasets problem number one is tungsten data format and you want to hear here of that a couple of people all right so when you're operating at the data frame level the sparks equal level the data that's read in when it reads the data out of these partitions into memory it reads it into this compact spark optimized serialized format called tungsten now contrast this with the old RDD api the old RDD api was programmed entirely with lambdas right and so the way the data was read and it was it was read into literally an RDD models an array of some type so if you read a text file in your first RTD is an RDD an array of strings right a distributed array of strings each of those strings is a JVM object right because you need to have an object so you can pass it to a lambda right you can't pass something weird to a lambda you got to pass an object in this means that that entire partition every column in that partition is has object overhead and is also subject to garbage collection tungsten doesn't do that it's very very compact it's actually optimized for common spark sequel operations like sorts and order in a lot of cases the the query when when the query is actually compiled down to the underlying internal rdd's we don't get to use they can operate directly out of tungsten so they don't have to have to actually deserialize and then Ricci relies this data when they do have to Risa relies and deserialize it it's orders of magnitude faster than say cryo or or the standard Java serialization and it's not subject to garbage collection and it's much more compact so you save memory you're not paying GC penalty and it's quite efficient right what he suppose happens up here when I add a filter operation to a data set my filters are getting a JVM object right I'm not operating out of tungsten so something is gonna have to manifest an object for me what spark actually does under the covers is it says oh you've got this object going into this filter operation so it actually writes Java code specific to how to decode tungsten into my object and then at the other end of my pipeline of land operations it Riaan codes it writing on the fly code right the the one of the guys in michael Armbrust one of the guys in the engineering team described it like this what if you gave a first-year computer science student a the task of taking this cereal I seen this case class what would that first-year computer science student do will probably come back with something that's very very specific to that case class you can't generalize this but because it's specific to that particular data type you can now optimize it for that data type right and their realization was hey maybe we can implement that first-year computer science student in spark so they do they write java code under the covers and they use geni know to compile it on the fly but there's still a penalty right you've got to actually run that code for every row that you're piping into that filter operation and then you got to Rhian code it on the other side so you you're paying a performance penalty now it might be okay if you're running a job late at night you're not around to watch it and this is gonna make life easier for you but you need to know there's a performance penalty associated with this you've got to deserialize the data of tungsten into objects and then ricci relies it back into tungsten you've done so your cut type safety here comes at a cost another cost is is is right here so let's take a look at this operation we've got a filter operation here I'm filtering on a user now that's a data frame then I'm going to cast it to my type my tweet data type I'm gonna do another filter using the data frame API right then I'm gonna do if this red here I'm going to do a filter in a map operation using the lambda and then I'm gonna go back to a data frame and write it out to a CSV file now the interesting thing here is that spark those first two filter operation can actually can be combined into one operation and spark can actually do that because it's a DSL I'm giving it all the information it needs it knows I'm operating on the text and user columns it knows I'm doing a comparison either alike or an equivalence comparison it knows the literal or the string that I'm comparing it against right that's all part of the query and it can actually optimize that and it can say let's combine those filters or if I happen to be querying something like a and our DBMS it can actually shove those filter operations down into the our DBMS so the data doesn't even manifest hey ID the our DBMS is capable of doing a where Clause let me shut it down in there but as soon as I hit this filter it doesn't know what's going on in there that's code that's byte code by the time spark sees it it can't decompile that code and figure out oh you're operating on the hash tags column and you're calling the equivalent of lower and I could rearrange this it can't do that it can't do any of those optimizations and I can actually demonstrate this right so this is um so here's an example of one we're gonna actually load up this remember I told you I have this historical data out there and what we're gonna do is we're gonna read up that historical data we're gonna select just the screen name alright and I'm gonna union that screen name with the screen name from my existing data frame and I'm going to write it all out right in fact we could get rid of the Union here and let's just write this all out okay and what we're gonna do is we're basically getting all the users who were tweeting back in February and then we're gonna see are any of those users still tweeting these Magga hashtag like eight months later nine months later right so while that's running so here's my join right so we're gonna read up the the users from the historical stuff and we're gonna do this join now if you look at this join it's kind of inefficient if you were to execute it as it's written because it's doing a join its joining the most users thing against the tags data frame which is this window down Magga tags data frame that I created earlier but the and then it's doing a filter to filter out I'm looking for that particular user that's prob about the r8 user now it's very different data but you get the idea here right so here's the join and if you look at this that filters in the wrong place wouldn't you rather I'm filtering one half of a joint it makes more sense to filter that before I do the joining right if it filtered after I do the join I've wasted the joined a huge amount of data together only to get rid of half of it so I should really do the filter ahead of time but that's okay SPARC does that for me and if you analyze this you can actually see that it moved the filter right so let's see if we can find it there there's the array contains that's the first filter from the tag number one there but it actually moves the filter object there the filter on here it is there's the filter here it happens before the join right so the join is actually done at Step five it's moved this filter to step three automatically nevermind what order I put it in but if I did that filter with a lambda it couldn't move it because it wouldn't know what it's doing and that's what you can see down here is that if I do the same thing down here but I do the filter based on using a lambda you'll notice that the filter is the very last thing that gets applied it's up top now that's the function apply so this is a downside of using lambdas SPARC can't optimize anything for you now you go back to being very careful again so data sets gives us back our type safety but we lose a fair amount of stuff we lose performance because of the translation the D serialization in the reseal is a ssin into our lambdas and we lose the ability for the catalyst optimizer that's the name of the optimizer that optimizes your query it doesn't know what you're doing in the lambda so it can't move anything around so it can't protect you from yourself so this really kind of means data frames are sort of better for you in that regard stick with the query API and the type safety again comes at a cost so this gives us that what we now have to step into and I've got 15 minutes to go through this let's see if we can do this frameless and typed data set okay so this is really interesting because what's going on here is that frameless gives us back our type safety and there are some costs there's a little clunky usability here and there but we get back compile time type safety but what frameless actually does is it gives us our protection and yet it still generates optimal data frame code under the covers as a you know through type trickery right so let's pull in the bazillion imports that we need and it's really easy to create one of these I already have pardon me I already have this data set so I just used the frameless typed data set create function to create myself a type two data set and it failed so why did this fail after a while you begin to be able to read these things but it's telling me there's something in tweet data I don't know how to encode so it uses these encoders to try to figure out to try to infer the right you know they're the right types to use from your case class so it's looking at the case class and trying to figure out how do I get the data in and out of this these native types here and there's something missing in the tweet data here and how do I know which one it is well if we go all the way back here to tweet data this is the only problem with these notebooks is trying to find stuff in them okay let's go back it's up here somewhere isn't it there we go all right so if you look at all these things how do I know which one of these things is failing well I'm gonna take an educated guess that the likelihood is it's the time stamp right because everything else is a primitive or string and it's likely that frameless has already provided encoders for Long's and and strings so let's assume it's time stamp that that we're looking at so look for typed data set again so I take that guess and you can just use the docs you basically just have to create this thing that frameless calls an injection right it's trivial to create this thing I'm gonna create an apply method that will go from a timestamp to a long a primitive type it already knows how to handle and an invert method that knows how to go backwards okay so that's pretty straightforward and there's actually a fairly lighter weight version that you can use in most cases this is just lambda driven okay so pick which one you find more readable it's basically the bottom line so we'll stick with one of them here and as soon as I've provided this implicit this injection now it will work ok so lesson number one is if you get that error you don't have to go back to look at your type but try to figure out which you know which of the types in there aren't likely to have an injector automatically for them but they're easy to create so you just create them until the error goes away basically okay so now we go down here and the syntax changes a little bit different here I'm using filter symbols and I have two subscript the typed data set now by the way this syntax is very very similar to what you can already do with the data frame API I use that dollars interpolator but you can also just use the subscripting notation with the data frame API so it models the data frame API here and as you would expect it's a transformation and it's it's lazy I get back another another frameless typed data set with a weird-looking schema okay so let's call an action so in action I'm gonna use show here instead of display this is more SPARC like okay and normally if I were to use a data frame and I call show that's gonna trigger a job huh that's lazy okay so why is that lazy well because this is this is type level right we actually have to do run or run unsafe or whatever the equivalent is in this API so what really gets returned by actions here is something known as a frameless job and I'll talk briefly about those in a little bit but the interesting thing about these frameless jobs is that like regular transformations they're composable so it's really kind of interesting you can actually create these jobs and compose them later and I'll talk about that in a little bit so it actually turns out that even though this doesn't model what SPARC does where an action actually triggers the job we actually have to call run it's kind of cool because all of these actions give you back these jobs which you can then meld together like a lot of the way you might melt futures together or put tries together so it's kind of interesting so you notice here when I add the run I get this this output okay and let's check the type safety so I'm gonna misspelled timestamp here in this symbol and now I get a compiler error I know I'm sorry I get a runtime error because I'm not actually using a type data set so this is I just want to show that you can use the symbol notation with data sets and data frames too so here again we're getting a runtime error because this is the standard data frame API on a data set if I do the exact same thing here it's slightly differently now I get a compiler error so now I got my types back right I've got my type safety back there are various column operations almost all the operations that you will find in the data set or data frame API you can find in the typed data set API so here we're gonna say I want to select the ID column and multiply it by two so we're creating a derived column you can do this with data frame API and with sequel right and I'm gonna give it a new name ID doubled and then I want to pull out the the user screen name so this is the standard derived column syntax for a data frame okay and we can do the same thing down here what we're gonna do here is we're gonna say select TS ID multiply it times two and then select the user screen name one here we're going to do the ACK same thing and now the only weird thing is these odd column names so why are we getting these odd column names like up here we have reasonable column names down here we get these underscore one and underscore two why well because the select what is the type that this select is returning turns out it's a tuple tuple of type long string right so there's no column there are no field names of the tuple so it's got to use the underscore one and underscore two so how do we fix that there are a couple of ways to fix that we can cast where everything lines up so if I do a cast let me clear all this stuff out they do a cast then what it's going to want to do is line things up in order so effectively I'm going to say I'm gonna cast this over to this case class I would like underscore one to go into the first field of the case class and underscore two to go in the second field of the case class so I can do this cast but it's really a compile-time cast I'm not gonna get a runtime error and in fact if I get it wrong it will complain as I'll show you in a minute then there's another thing that you can do which is called a projection so instead of doing a cast you can do a project now in the case of a project it's not order oriented now the the I'm sorry yes the names are matched if you do this so I'm going to take the original TDS okay this is the one that has all the column names in it and I'm essentially saying I'd like to project this down to this case class so effectively it's going to match up the column names that are in the target case class and ignore all the others so this is almost a way of just scoping down the type dataset to just the field you want and in this case order doesn't matter because it's doing its it's doing a name based comparison or name and type based comparison so this works fine and because I've scoped it down to this case class I've got columns right I've got field names and so therefore I've got column names but once again if it get it wrong you get compile time errors right so here I'm going to put them in the wrong order this is the casting operation so they have it should be long string because I'm going into this thing where order is long string but I flipped the column name so it's now string long so what's coming out of here is a tuple in the wrong order and I get a compile-time error it's not the most readable compile-time error but you can figure it out you get use to it and here I'm gonna go to I'm gonna do a projection but I'm gonna Tate I'm gonna change this from user screen name to user name and once again since it can't match the columns up I'm gonna get a compile-time error right and again this all gets caught when you run your circle CI or you're traversing your Jenkins job which is awesome right and just as the final closing this out here the names match but one of the types is wrong until you're also gonna get a compile time error right which is pretty good and again compare that to the standard data set cast here I get an error but once again it's a runtime error so your your continuous integration compile is not going to pick this up now what about some of the operations on the columns at 6 minutes can we do this all right I've been told I could run a little bit over we might run a little bit over so let's uh let's drop column ID and return a new typed data set now you may have read that already I'm hiding it what's this gonna return what's the type they got type data set the documentation for frameless refers to this as the schema right what's the schema coming back or the type coming back from the select anyone want to guess it's a tuple right it's gonna be a tuple so we can get back a tuple which means we're gonna get weird column names again because doing a drop drop notice it says drop tuple that was your hint right you're dropping a column and it has to give you back some type that captures all the remaining columns but you haven't given it an explicit case class type so it's gonna fall back to a tuple type right and then we're gonna get these weird columns because tuple types don't have field names all right again you can do distinct this is a pretty common operation to do this works exactly as you would expect again you're weird column names back but again all the same operations you used to her here aggregation is slightly less sequel like okay so one of the things you end up having to do here is use some temporary variables if you don't have the temporary variables sometimes frameless can't figure out what type it has to infer into so you'll notice here that the if you tried to do this using the standard chaining you're gonna actually run into some problems okay so I'm not gonna walk through this too much we're running low on time but but in any case if you're willing to live with a few extra temporary variables and what are a few temporary variables between friends right you get you get the compile time type safety you want and you can do the full aggregation on this stuff it's just a little bit more clunky than you might use with data frames you can also use things like with column replaced to replace in place of value of a column right and this because you're replacing an existing column with a new value for the same column that it can retain the case class type so you don't get the tuples back you just get new values for that for that type right which is really nice what about adding columns okay well they're a couple solutions here you can project first let's project this we're gonna do a project down to something more manageable to many columns in this display up here right okay so this gives me something that's actually a little more manageable not sure why I get the scheduled job up here I'm not gonna waste time trying to get rid of that I said let's add a column so what I'm gonna do down here is I'm going to take some data and I'm basically gonna create a new case class that has a new column in here called ID right and I'm setting the value of that column to the literal zero so it's you can almost think of this as a project up now instead of projecting down to an arrow or data type I'm projecting up to a slightly wider data type and applying the literal value I want it to be initialized with as if I'd do that you'll see when you run this display that we suddenly have this ID column that's set to zero and I can condition we replaced the column values here's a case where I'm going to take a particular screen name and so that I'm not data sensitive I basically pulled the top screen name out of here already and I'm gonna say filter it down and then create a new data set where if the screen name is the first screen name we're going to set the ID we're going to propagate the ID as is otherwise we're gonna set it to 201 all right so you can see oh did I not run that I may have dropped that out I'm not gonna put that in all right a top screen name let's see if we can add that real quick screen name no I don't have time to figure that out we'll pretend that that worked okay what you would have seen if I must have dropped that cell out of there what you would have seen is that every single column there's still a zero except for that one right and I'll put that in and correct it in the github repository but if you were to flip it it wouldn't work okay so this is I can't run this again for the same reason but here you'll notice I flipped it and the problem is it turns out this has an inference problem it can't figure out where that lit is supposed to go so you got to be careful this would actually refuse to compile for a different reason right even if I had top screen name defined so this is a simpler way to add a column as well right this is the typical thing you do in spark which is select all the columns so select star plus the ID doubled right and then we'll show one well this is how you could do that the frameless way I'm gonna select now this is actually kind of interesting because it turns out to do something a little bit different what what the frameless one does is it actually takes the entire typed data set and makes it a new column right and then adds a new column so whereas the data frame select basically keeps it all flat all your original columns plus a new column at the end the type data set 1 says take all of the existing columns and make them one a column right and then add the new column so that because to do the proper type management on it right so you know there are some syntaxes for how to drill into that which we can talk about down here in nested schema so the question is what about performance so let's do the join this is our previous join the same join we did before with the same screwed-up filter in the wrong place and one of the questions is you know is this thing actually doing lamb does is am i paying a performance penalty to get all this type safety and the answer really is that if you look at the explained plan and you were if you were to compare this explained plan with the explained plan for the data frame version of this joint it looks substantially similar there's the filter there at number 3 where it was before right now it's comparing against this thing called frameless lid so there's some slight minor differences but the the plan itself the execution plan looks substantially similar to what you get out of data frames not the problematic thing you get with data sets and you know lambdas so you're still getting the same kind of performance that you would get if you stuck to the data frame API right so we've hit 1030 let's see if I can blow through the remainder of this stuff in a couple of minutes so how do I get let's take a look at a nested schema here we're gonna pull this thing in we've got a nested nested tweet data so let's display it real quick so once again you'll notice I've got a row ID and then I've got this structure type as the second column right so how do I drill into like the tweet hash tag in here or the screen name well if you use data frame notation it's dots tweet I want the screen name right so you drill down with dot notation get it wrong yet a runtime error what you're doing frameless is use this calm anything so it's essentially saying string these columns together take the nested then give me the tweet followed by the user screaming right so that's it's sort of instead of using dot you use calm any now again it's pulling back a tuple type but the nice thing here is if I get it wrong user screen NAMM instead of user screen name I get a compile-time error so once again I'm back to getting the compile-time error and then finally there's this section in here we talk about jobs so here's a job where I'm actually doing a take okay I take is typically an action that will trigger something in the regular spark and so is account so in regular spark both of these things would trigger an action but in frameless both of these things are jobs and what I could do then is I can I can string those jobs together with flat mat the same way you would combine any other monad right so even simpler I can put I can say get me to count now that's a job right we I'm running yet and then I'm gonna do this is a poor man's sample right pull the first count divided by a hundred rows back that's not really a statistical sample but I'm getting you know some 100th of the data set back yield that know this is going to give me back another job which I can then run right and they can be combined in a bunch of different ways right so you can you can actually put them together quite simply with basic map calls right this example is very similar to one that's actually in the frameless documentation so the conclusion to this is I'm not going to run this one you can run this on your own but the conclusion to this is then I've got three api's which one should I use right how do I start with this what do I do and your mileage may vary but what I do is I basically say if I got a hunk of data I've gotten from a customer or a client or you know even my boss can you analyze this I don't know what the shape of that data is so I'm gonna pull up typically a notebook or the or the scholar repple with inspark and I want to play with the data and figure it figure it out do some analysis of it so that I can figure out okay what kind of production job can I run on this stuff I'm gonna use the data frame API there why because I can dot combined thing things really quickly it's there's less to type and if I'm doing that kind of analysis I don't really care about runtime errors I'm in the repple but when it comes time to write that production job where I would really like to have the compiler protect me from doing stupid things and in particular protect me from making changes that break the job now time to go to something more typesafe and if I want to go to something more typesafe my first choice is going to be frameless because I still get it still compiles down to the data frame query API so I don't pay any of those penalties so that's my approach use data frames for the quick analysis or Python if you really want to hurt yourself but it means the same sort of thing right it's just deal with the runtime errors because it's fast the turnaround time is quick and then translate it all into something more typesafe like frameless and you can do machine learning with frameless I don't have any time to go into that here but there are some there's a whole description out of the frameless it was a tight level of Doric / frameless I think is the get book there's a whole section in there about using machine learning api's with frameless which can also help you that's a little more problematic given the way the the machine learning API czar are organized but again you get more type safety back without paying a performance penalty which is a nice sweet spot that frameless sits in and with that we're over time so if you want to hit me up for questions or if you want to ask the long cow here the guy who can tell you all about how this is implemented feel free meanwhile that's it that's a you know data sets data frames type data set I recommend playing with it and this this presentation you can use that to play with it and go through the docs at type level that Ord frameless they're really very good alright thanks [Applause] you