SF Scala: Holden Karau: Metaprogramming — making easy problems hard enough to get promoted ....
foreign so I'm going to talk about meta programming Less Fun meta programming this is more kind of the boring meta programming I like to think of this as making simple problems hard enough to still be something where you can get promoted it's very important especially in our current economy that our jobs remain hard enough that we cannot automate them and thanks to the magic of meta programming you too can make your job really complicated um okay cool uh so yeah I'll talk a tiny little bit about who I am um our other goal besides getting promoted and not being replaced with chat CPT is to save ourselves from Legacy software um and our primary challenge for us to overcome is our own laziness as a profession um I'm I'm lazy it's why I became a programmer that and many other things and our tools are terrible and if you look too closely you will find a pearl script but do not worry I will not intersect Pearl on you today um if you want to find the Pearl script come and talk to me later and we can be sad together okay um yeah and so we will try them but only if we scope our goals down enough um this is a constant battle that I face wherein my boss keeps trying to add them back after I take them out it's very annoying um but yeah I'm holding uh my pronouns are she or her uh I've been at a bunch of different companies including Foursquare um right now I am at Netflix uh the data organization at Netflix is hiring um not a lot of roles but some and remote friendly um oh right also very important uh My Views May differ from that of my employer uh my style certainly differs from that of my employer and uh yeah we'll go from there um so it's also pride month um I'm trans queer Canadian I finally have a green card I'm really really happy it took like a decade um but now it is slightly harder to kick me out of this place um so that's really awesome and I'm part of the broader leather Community we always get pushed to the back of the pride parade because it's not the family friendly part um but you know yeah they're they're good folks okay so you probably are familiar with meta programming but just in case you're not that's totally cool we're going to write software that writes our software or more accurately we're going to write software that manipulates our existing software in our case um meta programming can be as simple as that Pro script I was referencing which runs a regular expression on your code you are writing software which is writing software um and it's totally cool if you haven't done meta programming before that's awesome and if you have you might find this a little less exciting but that's okay too because I have a lot of pictures in the right hand corner that you can pay attention to instead so why why should we care about this so for one thing we don't get promoted for updating from spark 2 to spark 3. uh generally you don't get promoted for updating to current versions of software um it's just you know it's your job so you should do it but no one no one really cares about you and you're you're stuck in the basement and they're just like how's it going down there and you start saying I can't find out why this API doesn't work others like okay we're gonna close the door and keep on walking um but if you do meta programming people are like oh that sounds that sounds hard yeah keep doing that looks great on a promo package the other one is more more seriously I like money and right now it turns out the people who are willing to give me money want me to work on spark and that's great I love working on spark uh it turns out the other problem is though they have really old versions of spark uh and I don't love working on really old versions of spark because then I have to like bust out Scala 210 and that is not my happy place that is my sad place I have memories they're not good um and so I want to get us onto modern versions of spark and Scala so that I don't have to deal with this um yeah once again you know all the sadness of doing it by hand yes it is probably faster and will save your employer money but that does not matter because your employer is probably not in this talk and if my employer is in this talk um while this is true for other companies is this certainly certainly for us this is the best thing to do because we have just so so many pipelines and instead we can write Cool Tools and we can be like my dog and enjoy the river uh he doesn't actually like swimming so we had to put him on a little floaty board in the river uh so that he'd actually go in the water with us um foreign but the other thing is like yeah it's it's really inefficient to to write all of these tools but we can use the magic of sharing um is anyone Canadian here uh that is me okay never mind um so there is a Canadian uh essentially the the version of Scouts for for young kids in Canada is called beavers and the motto of beavers is sharing sharing sharing right so open source software makes our life better um just like we were taught if you grew up in Canada when you were about six years old um and so we're going to do that uh my dog still does need to learn how to share though um so that's good okay so we can do this and we don't have to only do it in Scala I only think we're gonna really talk about Scala and maybe a little bit of sequel today um but we can do this in other languages too right those those terrible Pro scripts they don't care what language they're running on now to be fair you will care because you have written Pearl and summoned the devil um but you know there's there's python meta programming tool right and uh yeah that's cool so we're going to use Scala fix um how many people here are familiar with scalafix okay well the scholar Center folks is not the biggest surprise in the world um but for those of you who aren't it's really really cool um and it lets you write things that are much better and even more maintainable than Pearl scripts um for transforming your code now that is a low bar but I am pleased to say that it exceeds that low bar and so here what we what we've got is we've got some really janky symbol matchers and we're looking for people who are calling the Json method on the data frame reader in I want to say like spark 2.1 because we've changed the the type signature on it a little bit and we now have to actually specify uh these encoders when you're doing this on top of an rdd so we also have this utils rdd matcher which matches when on the parameter types when we're passing in an rdd to this Legacy function and it's like oh cool you are calling this on an rdd and we need to tell it like what the encoder is and everything will be happy and also if we have an imported the encoder is add my encoder import otherwise you know if they're calling Json on this and it's like a regular thing like a file path or something it's fine we don't have to patch this code so just do nothing return patch empty um and then here we recurse on a tree because everyone loves rehearsing on trees yay okay uh okay so you probably care less about python um but we can do similar things in Python except we don't have types uh uh so that kind of sucks um I really like the python DJ Community it's amazing I wish their rewrite tools had types they they don't really Implement integrate with my pie in the way that I would hope uh so we're largely we are like slightly better than regular expression level in Python but like still not really all that far ahead um we're just saying like hey if someone's got a function called two degrees let's swap it with degrees so we won't replace any arbitrary string with degrees but we'll replace any function call uh and it turns out there's this inconvenient thing where Spark and pandas share a lot of the like API names but semantically do not agree uh so when we tried to make this tool for python um I succeeded in delegating that business great success uh shout out to the python track lead and I'm sorry okay um and now sequel SQL is actually um for all of its sadness pretty cool um in part because like it's a comparatively simple language right um and there are a whole bunch of tools for it once again we don't have like great type information present but we don't have the same like API sadness um and so we can write like simple rules here uh this one we changed the semantics around Char versus string types because we hate ourselves and we wanted to make our upgrades painful um and so we have to do some things when we find this chart type uh life is sad and we just essentially replace it with string don't don't worry okay now you you might be going like Holden that sounds amazing are there any downsides to building these Cool Tools and yes indeed we could get really really deprecation happy if we built amazing migration tools based on how painful it is to build migration tools I do not think we're going to get there so I'm not too worried about this but if you've ever wondered why Google deprecates a whole bunch of things and just does not seem to care there is a fantastic blog post about what happens when you make your internal tooling so much better than the tooling available to your customers it's a great read really depressing uh I don't work there anymore but I in will probably try and work there again in the future so you know uh y'all are still great cool okay so that's cool okay we we looked at like some pretty trivial things what what does this look like when we try and run it um what kind of changes do we get so and we can see this is from October 14th 2020. um very classy PR and if we go and look at these 77 files change uh okay yeah there's a bunch of things where we're just deleting a whole bunch of code that I didn't need anymore oh wait ah sorry I want to get the individual commit ah here we go we're gonna go to the first first thing um so here the first thing that we did is we applied the existing Scala fix fixes uh which added a bunch of type information which I really should have put in there but I'm a lazy person um and then we went ahead and I had some automated thing to update our SQL Constructor all kinds of fun um I don't know why we changed that but like relatively simple things but we don't have to do it um specifically we don't have to do it about the 20 or so thousand times that I think I've found in in my code base um and so that's cool right it's it's simple we could actually accomplish this with the pro script we just hate ourselves in the morning um but I'm I'm used to that too so okay cool so that's that's great that's like Holden's sample like that's the spark testing based project uh what about like a more real project uh so it turns out that uh those are hard to share because of reasons but I do have a really shady demo which while still not a real project is more interesting than a spark testing base so we are going to run this sketchy demo and we're gonna hope it works and I'm uh uh ignore that okay uh sorry I have a lot of screens yeah there we go okay end-to-end demo Scala this is probably the right directory okay and so uh we can see uh that I am not cheating maybe I'm going to run this this shell script called run demo tastic okay and so it starts off by deleting things and deleting my test table which I called Fart table um because I ring myself great joy um and so what this what this uh relatively simple script does and we'll go and bring this up ah sorry oh it wants to re-import spt changes uh um so we're going to go Spark demo project yes there we go okay Scala run demo okay it does like a bunch of annoying things around sort of like setting up the environment getting an old version of spark getting a new version of spark deleting tables making new tables so that we can run our tests and then it actually it's going to take a while longer um it says what it's doing I did say I was going to hide the pro from you and that is true um I I do use a python script to rewrite the build file because I'm lazy and I couldn't figure out a nice way to rewrite the build file automatically um there probably is a better way but we we take our build file and we rewrite it and the main thing that we do is we add our collection of scalafix rules and then we go ahead and reapply them okay so it says like hey uh this is what I did I added Scala fix and I added these scholar fix rules press enter to continue now we're going to run scalafix it's totally probably maybe going to work I did run this demo like 20 minutes ago so if it's broken I'm going to be really sad very exciting so what do people like to do while waiting for software to build I don't have any phone swords with me oh there were former swords in the FourSquare office um they took them away from me though after uh the incident with the phone swords okay so here there we go okay uh we can tell this is a real demo because it's doing a bunch of print lens and so that's how we know that this is definitely a real demo and not fake uh and so the first thing that it's done is it's applied uh sort of the automatic scholar fix rules I made a bunch of other scholar fix rules for things that I was like I could fix this the right way or I could just warn the user um and those those started to get a lot more appealing after I wrote the first two rules um and so here we see like okay uh you called Group by key which an inconveniently large number of people do it turns out people want to group their data by key for whatever reason uh and also for whatever reason uh we changed the meaning of that API boom like for the better right um like this this this reference to count now like it was a little ambiguous before but you know frustratingly Common uh but I'm lazy and so I didn't actually fix this the right way I just told people like hey here's something wrong you go fix it and this is better than the alternative uh because the alternative is uh this stuff is is not typed and it gets resolved at runtime so here uh we get to find out about it before production goes down and that increases the chances of me not getting fired by at least 10 percent okay whatever there's a bunch of warnings we'll just press enter and Skip them all great okay cool uh now it's gonna go ahead and like kick off two versions of the pipeline for comparison purposes but that's going to take a minute because it's actually running on my laptop um rather than anywhere else so we'll we'll just go and take a quick peek at the code we write really quickly so everyone's favorite word count yeah [Music] no no okay I whatever fine um so here we we have word count and we intentionally call a bunch of deprecated functions uh that are removed in spark three um oh yeah sure how's this okay rocking thank you yeah so we intentionally call things that are removed in spark three um right we can't we can't do things like this anymore so now we're going to look at the spark 3 version of this that has been automatically Rewritten and I'm going to hope that it actually did what I wanted it to do okay and it has once again done these like relatively simple things it's done these these automatic rewrites for us uh and that's kind of cool right um I didn't have to do that I did admittedly in Saving this like 30 seconds take me the better part of a day but who Among Us has not spent a day to save upwards of tens of seconds okay great okay so what does the future hold uh we should add a bunch more rules and there actually is there's there's a whole bunch more rules than when I made this back in 2020. um let's go here uh um if I zoom in it looks more impressive okay but yeah so we have a we have a bunch of rules for migrating different broken things in spark um and that's that's kind of cool right yeah um but we should we should still add more uh and maybe one of the ones and this is one of the ones which I'm both like excited about and a little bit terrified about uh is trying to rewrite inline SQL how many people here use Spark have you ever written inline SQL inside of your spark okay I'm getting head nods but no one's willing to raise their hand because they're just like I do not want you to know um and that's fine that's fine I don't want people to know either but I'm not very good at keeping secrets uh so it turns out though like we have a lot of people who have written inline SQL inside of their Scala spark because it's just so convenient right um the problem is we made a bunch of SQL API changes as well and we should we should probably fix those too now that being said I'm lazy so I'm not gonna like rebuild that from the ground up I'm gonna do it in the classic Unix fashion which is the fancy way of saying I'm going to write it out to a file and call a shell script on it oh yeah uh the only problem is people tend to use templates and those get real real Annoying and if you want to solve that problem I did mention that my organization was hiring okay um we could go through the rules if you're interested though just go check it out yourself I'm not gonna go through a GitHub with us together today um another important thing is like how do we know if this stuff worked um and that's that's a really good question right and so here um we ran a a quality demo um and we had two versions of our Pipeline and we actually produced two separate sets of results and then we compared them afterwards and we use this we did this using the write audit publish pattern of Iceberg but we just skipped the publishing step so we write and we audit and then we just we don't publish unless everything looks okay and the new results are the same as the old results uh then I guess we could publish um but like it's important to not just like upgrade and then run in prod that tends to result in sad people and pagers going off and I have a pager so I don't like that um and there's a whole bunch of different things to do uh my current pipeline to do this comparison is called do magic dot py uh so you should probably do something better uh then do magic dot py uh we are iterating on something that does not involve uh a random python file that I kind of wrote in the back of a conference once and we're going to make something real good and I'm really excited about that but it's it's not ready yet but do magic dot py gives you an idea of how to do this okay so maybe you're like this sounds awesome almost as awesome as vending machines in Japan um how do I convince my boss um so there's there's two things that we need to convince our employers about one is the need to upgrade and the other is that uh the way to upgrade is by spending ridiculous amounts of time making automated rules to save tens of seconds um so the need to upgrade uh is is great um security is a fantastic reason to upgrade because no one wants to be the one who's responsible for losing your company's data so that's a great reason to upgrade uh performance performance is even better nowadays people don't want to give all of their money to Jeff Bezos anymore it's not cool anymore uh VCS want you to spend less money and get more Revenue super weird but we can upgrade to new versions of tools and do things more efficiently and if that's not enough maybe you know you work somewhere where money is not a concern if so like let me know um but you can always be like hey do you want to do AI yeah you need spark 3 to do AI and if you believe that oh yeah um no more seriously there's a whole bunch of really cool tools to do better Ai and ml stuff in spark 3 like a whole bunch of stuff is a lot better there the scheduler got improved a lot and maybe you can use that to convince your boss um here's how we save money once again do not try and reproduce these benchmarks they've already provided us with the correct answer reproducing benchmarks can only lead to sadness um okay right um the other one is like eventually everything will go away and break and then you don't want to be the next Southwest Airlines um who just can't do anything for about a week because no one knows how it works anymore and the software that you're using is written in Cobalt um right okay okay the other one is how do we convince our employers that we should use meta programming to solve these problems right um so honestly if you have like less than five pipelines it's going to be a tough sell right like you're gonna go to your boss and be like hey I should write a whole bunch of migration rules and they'll be like what about if you just migrated that and I don't have a real good answer for that um the other one is you could be like we could do this in the open source and then if we do that maybe other people will join and then over all it won't be that bad that we spend all this time doing upgrade rules I've tried that one pretty successfully as you can see from how much time I've spent on this um and the other one is like easier future upgrades right like if we learn how to do this now if we automate some of our upgrades today in the future we might not hate ourselves um not a guarantee okay in conclusion uh the good news is everything is still broken and we still have jobs uh the bad news is everything is still broken and we still have jobs uh the excellent news is my dog is incredibly cute um this is Professor Timbit I love him more than programming um also if you're like you know what this sounds incredibly terrible let's share it with children uh then I have great news for you about my next project distributed computing for kids um if you've ever said to yourself I really want the neighbors children's coming up to me and asking me about out of memory exceptions then I encourage you to go to distributed computing for kids.com and give me your email address because the PDF is almost kind of finished um and uh based on the early feedback we've replaced the Gnomes that looked like they were on acid with these gnomes who look like they're from San Francisco but not on drugs um and so I think this is a great Improvement okay so that's that's all I had it [Applause] this will be the question for what happened uh yes so at the service University that it's so cool okay so the question was essentially there's been talk about spark 4 and maybe this is a good time to think about integrating some of this stuff into the spark project itself so that we can take advantage of Scala Stewart which is an amazing project that automates upgrades for for libraries um for people in the Scala ecosystem and I think that's totally true I think that's going to be a really tough sell um to people um consensus driven software development is better than many Alternatives but adding new things like this can be really hard because then people in my experience are like really resistive to adding this stuff like I maintain a project called spark testing base uh which I tried to get into spark right and I'm a committer now and I still haven't succeeded at getting Sparks testing libraries merged into Spark uh uh so I I think you're right and I think in an ideal world yeah we would put some of the stuff inside of spark itself and I might just be too pessimistic because I've been around for too long um but I will try it is on my list of things to try and convince people of um and and I would love to be wrong I I'm I really hope I really hope I'm wrong foreign