scale.bythebay.io: Pathikrit Bhowmick, Scala Meta Live Coding Session
Recording: scale.bythebay.io: Pathikrit Bhowmick, Scala Meta Live Coding Session
you alright hey guys yeah my name is Prateek Adamic I'm gonna do some live coding with Scala meta and macros so we'll see alright so about me so I I've been doing Scala for number of years I work at a hedge fund called core to management we do big data analytics and find trading signals in large data sets I'm also pretty much involved in the open source community and part of the scala platform committee and that's my github we can find some other software that i've read an open sourced over there so to set the expectations about the stock it's not about how the Scala meta internals or like the too much lineage or history behind Scala meta it's not about Scala macros per se but there's going to be obviously a lot of that but I'm gonna use column meta to do something useful so the motivation is macros in Scala meta really useful tools and very few developers use it but when they use it the impact is huge like it goes into this method very widely used libraries but that should change everyone who's writing scholar should can and should know how to use macros and when needed and so the stock is to get the fit get your feet wet and see it's not that bad it's pretty cool you can do a lot of powerful things when you can write programs that can write programs so yeah so why macros like some people say like oh you don't need macros you can do anything with the language but that's not true because at least in Scala like you can't do something like this where you can't write a method called debug where you can parse in parameters x and y and you want it to print out the variable net variable name like X equal to 23 in this case y equal to hello on the command line and there's no way to do that use without using macros you can even do some more powerful stuff with the debug macro can print out the line can't even print out the file it was in so macros basically have access to the source code at compile time so you can inject it with which you'd lose at runtime you don't know the variable names you don't know where which file it was and but macros have access to all those information so you could use that to make your life easier when you're writing Scala code so what are macros out of like literally one sentence would be macros are functions that takes abstract syntax tree programs and spits out new abstract syntax tree a new program that's all that's there to us and this is one way the this is a way you can write annotation macros which is the life coding will do so the incantation is you just import Scala dot meta you extend the static annotation so this this is coming from the Scala library so this is nothing to do with annotations the only two things that is new here is the inline keyword that constructs column meta and this meta keyword it comes from Scala meta and so you have the definition it's typed any I will come back to that why it's Annie which is in this particular case would be the abstract syntax tree of whatever you're annotating so if you're annotating a method you'd get back the extract syntax tree of the method if you'll get annotating a class that's what you get back and it turns any I know you guys like he'll hate any but this in this case you are supposed to return Dib stacks a new abstract syntax tree and one one reason it's any is because you could return anything could return like a variable you can return a whole new class you can return you can change the annotations around so it's it's not clear what the return type should be so but I'll get back to that in a bit and this is how you use it like any other annotation you just do at this my annotation and in this particular case I'm putting on a user object that's what I'm doing ok so again the motivation there's a lot of motivation that if you use a lot of these libraries they have they used macros at some point or another and so you already are using macros a lot and this is an important weapon to have in your daily programming scholar repository so macros for whatever historical reason has this notion of being hard to use kind of weird things that people who only talk at conferences do I would say that's not true anymore or I hope to convince you after my talk you would believe that and that's a very good reason for that so before quasi codes I'll get what quasi codes are to write a macro you have to do all this so all of this thing it literally represents list 1 comma 2 comma 3 and what is doing is like okay the whole thing is an expression and there is a this is actually listed apply so it's taking the identity list and it's doing the application with parameters which are literal constant one literal constant - blah blah blah this is this is the reason macros got a bad reputation because what happens here you are very intimately familiar with the internals of the compiler we as programmers we like abstractions why should I know what the compiler is doing maybe there's a different compiler for Scala jazz or something like I should not be aware of how the compiler sees my code so quasi quotes came around and it solved this problem in a very beautiful way we get a string interpolation the cue interpolation and it D structures the whatever you pass into it it's a scholar it you pass in Scala code as string and you can be structure it a pattern match on apply do all the nice things that we are familiar with in this color world for example given a function so assume that there's a def he did a def FN you can do a pattern match which is the left side on apply feature of Scala and you get back the abstract syntax tree of the arguments of the function and dollar body would have the abstract syntax tree of the body of the function so that's the left hand side on apply using quasi codes but you can also create new objects using quasi codes if you can as we've mark and you can create a new function using quasi codes you can pattern match so here I'm pattern matching for the identity function and this is what the identity function looks like you have a payment and you just return that argument so this tree would always match every identity function that ever exists so and it comes with nice helper utils like a board so you can so macros are run during the compilation phase and you can abort the compilation if something weird is having let's say I put an annotation on a class which was but that annotation only works on deaths and you can abort the compilation with the line number with a nice message so yeah I mean this this doesn't look that bad so so two years ago I did a life coding at Scala by the bay regarding quasi quartz you can go watch the video but it's not necessary to do that to understand this talk so for this while I was working on it I found these three links very useful these are gonna be I'll publish my slides you can access this links for now I opened them before the talk because I didn't know if I'd have Internet issues so these are the three tabs I have opened which are exactly the three links and I'll refer them during the live coding part so you'll see how I use those three links okay so actually so hopefully you're excited so let's solve a real problem that I faced so imagine you have a nice simple case class representing a business model in this case a user a user has an ID a name email and a registration time now you're designing a REST API and you're you guys are writing Scala I'd like okay I have to write the case class to say what the shape of the output of the get API looks like so you define a user gate okay you get back the ID you get back the name and you get the back the email well let's say you have like some post API to create a user so when you create a user obviously you don't have the ID is generated by your back-end by your database or something but you need to provide name and email for sure and patch patches are things where you can do partial updates of a model and the patch you can say like I only can update the name and the email and these are optional so okay you wrote this but you can see that it's kind of repetitive if you have like 50 models you have to maintain that's also put another which I didn't put on the slide so you have to mint in four times 5200 case classes and that seems kind of copy pastes and troublesome and what if HAP what happens if I change that type if I change email to email class instead of a string class then I have to go I mean nice thing we live in a static world static compliation world well just go fix the compilation errors manually but still that's a repetitive boring work okay can we do better so here's what I want ideally ideally I want to put an annotation called resource on my case class and I want to say what things are available by via what api's for example I want to say the ID you can only really during the get call with the get API the name the name you can read it during the get but you need to supply it during the post and you can also do a partial update on it so there's an ADD get at post at patch on the name for the email you cannot change your email address because no you registered with an email that's the one you get so you need to but you need to register with it so you have a post on it and you also get back when you read the user back from your back-end it could be a database layer some data access layer you get the email back and this is the registered time is something that is only available to your back-end you don't want to make it available to your REST API at all so this is so this has no annotation on it so given this nice pretty looking user model I want to generate this code I want to generate a case class get with all the things that are get about a case class post with the things that are just returned by the post API and a case class patch but the patch I want to be a little clever when I want to make the since patch if you're familiar or you can do partial updates you don't update everything in the field unlike a put where you have you need to update everything so in patch I want to put options on all the types so in this case I don't can I want to really update the name and I put option string default arguments nom nom okay so let's get this life coding started okay you can see all right so I didn't do a lot of setup before this life coding I know it's a short talk so I did a little bit of setup so so the thing is let's look at the builder SVD sub brand new SBT project I'm using Scala 212 I added the Scala meta paradise plug-in and I only depend on Scala tests to write my unit tests for this purpose of the talk and I depend on Scala Mehta 1.8 there's a newer Scala meta out but for the purpose of this talk it doesn't change anything it will work yes ok all right everyone can see in the back ok cool thanks yeah so provided means that you don't need Scala meta as a runtime dependency of your application you only needed to provide it to your during compilation you can remove it then it becomes available at as a runtime dependency but that's just not needed because when macros are run only during the compilation phase you don't need Scala matter during the runtime of your application so that's what provided means yeah that's it this is builder as we do you have nothing funny going on so this this is the test I wrote this before I started the talk to not waste time I literally copy pasted this case class put it here I'm using Scala tests should compile I'm saying if I managed to make this work user dot get should compile and user dot post should compile and use a dirt patch with a partial update should compile and this put should not compile because I didn't declare any put here so it's just a test and while we life code our goal is to make the test pass now it's complaining like it doesn't do it it doesn't recognize any of these annotations so ok let's start by creating those annotations so I'll call it the resource No all right increase the font actually even before I start so let's start SBT okay so I so I'm doing clean test with a tilde in front what the till that does it watches your directory as you code why do I need clean because you when I'm changing the macro chord the Scala compiler is not aware that you might be actually changing the source code because and so you need to always clean because in this particular case I'm modifying the macro code but if you're not touching the macro code you obviously don't need clean you can just do tilde test so in this case since I'm changing macro I'll do that and let's see yeah I get the same compilation errors that intelligence telling me that all these annotations it doesn't know anything about them okay okay so let's create our first can you guys see in the back okay so let's create the resource annotation okay that's that one and let's create the restful annotations which are scared extents annotation and there's four of them post but but you don't actually need a one for the delete API because you don't specify anything in the body awfully okay and well that should make it recognize the verbs at least yeah so now I'm getting the error saying value get is not a member of object user so I haven't actually generated this get class okay let's let's start with the magic would happen in the resource annotation because that's the only claw at the annotation which has access to the whole the whole case class like the get this get only sees what is annotating in this case the constructor argument ID okay so this is where the incantation comes in from Scala meta so imports color dot meta okay in land F apply and this is the tree and you return a new tree and you use this so this is the pink annotation needed to define a Scala meta annotation macro okay so in this particular case definition is a tree so I can return it and this code would compile so let's try that so this shouldn't do anything yeah okay extend our static annotation so it should be static annotation there we go yeah that that gives me the same compile error but you can let's see if this actually got invoked or not I'll use my favorite debugging tool for a friend line let's see if what does it print ah look look at that look at that it printed the case class this is the tree that this is the actual the class that I was annotating that's very cool I got my compiler to print something okay so I need to first get this companion object out and there are two cases see the companion object may already exist or it may not so you might have to create one on the fly and how do i pattern match so there is this very nice cheat sheet which tells you for anything for how do you create a class how I create an object and you can use it both to create new things new classes new objects or to pattern match your existing extract syntax tree to do something so in this particular case I will pattern match on the definition so it must be two cases one so it's the quasi code it's either a class followed so this is the pattern for the a class followed by its companion object okay and this is the other case when the companion object is not defined yeah it's not like you could it the companion object could be defined before after there could be something in the middle but this this by the time you get here you have this a very good question it doesn't actually matter that the companion object is defined in the next line after the class definition ends it doesn't matter it has is defined as a valid companion object you get this is the pattern match for that because by the time you get to the abstract syntax tree which at the phase at which the annotations run you have all that information nicely collected for you yeah and again there's a cheat sheet linked with links all these different quasi codes to use so okay so let's see what should we do so I want to get the class and the companion from this definition for in this case when the companion is already defined is simple I just returned the object and the class in this case I have to create materialize a new companion object and the way I do it is very simple it's again use the quasi code instead of pattern matching you use it to materialize it so this case it would be object it would take the same name as the class because that's how it is class dot name and that's it okay so let's see what the compiler is saying so the compiler is saying found type dot name required term not name that makes sense because the class name is a type when you declare a new class you declare a new type but when you declare a new object you don't declare a new type you declare a new term like a well an object is a well okay fine so all these constructors are really useful all these ADT's have very useful constructors so I will create a new term and need a term dot name and it takes says okay expected stream ACTU so it needs like an actual string parameter so there's always a value on everything so okay and let's print line class and the companion so in this test I didn't have a companion object I just had the resource class so I should have materialized I should have materialized companion by now okay let's see yeah well uh look I just created an empty companion object object user okay that's progress all right okay so now the so I have I have this companion object I need to create all these case classes for every annotation inside it so I need to go through all the annotations like this get post put patch on the users constructors arguments okay so it's actually very easy like if you have a complete so it's like okay I want to look at the constructor how do I look at the constructor so you start typing okay see tour that's what it's called that's constructor and it's type is constructor dot primary because you could have secondary constructors for the purpose of this life coding let's go with the primary constructor and I want to look at all the arguments or the parameters to the constructor because that's what this these are these are for a case cause these are parameters to the constructor so they should have something called params ah it says params SS the reason for that is you can curie constructor parameters in Scala just as you can construct kuri have multiple parameter lists for a definition you can have multiple parameter lists for a constructor again for the code up for the purpose of this life dog I don't care about that I'm just gonna flatten them so I just pretend I just have one giant on curried parameter list okay so let's print line that I want to see what's there yeah okay so I get back a list of yeah I got the list of all the constructor parameters with all the annotations the names and the type looks like progress okay so let's start our for comprehension so these are my params or param and I want to get the annotation for these params so how do I do that I haven't actually showed you how to use this cheat sheet I it's command F a notation up there we go so that's this is the quasi code to get the annotation to both again quasi quotes are used in two ways to both D structure on apply pattern match or to create new things in your world so in this case you have the mod string interpolator and you do this act thing okay fine so so we go to dot modifiers so the modifiers of a parameter could be access control modifiers public private protected it could be annotations so that's what they're called modifiers so let's go through each modifier and for the purpose of this talk we're only interested in the modifiers if they are these get post put patch annotations so here would better match for an annotation and we saw that the pattern match for our annotation is this more thing so we'll do mod at get or marred at put or mod at post or marred what's the other one patch there you go so this is the thing that we are interested in so we will return this parameter now if there's anything else we don't care like if there's like any other annotation or any other parameter which doesn't have a notation we don't care so none and let's call it the new program something that we're interested in okay and we will yield the modifier saying this modifier is on this new parameter so nothing here is complicated it's just basic Scala I'm writing at this point pattern matching and let's assign it to a variable I don't know perhaps or something and favorite debugging tool per okay okay that's pretty cool I got saying get is on the ID okay this is awkward because I have get twice because when I returned the parameter it returned with the annotation on it I don't want that so these cut these all these ADT's abstract data types to represent nodes in abstract syntax tree their case classes so you can write plain old scholar to modified and to modify them so in this case I want to copy pretty much the whole parameter but I want to nuke all its modifiers because I don't 1 the annotation on it like if I have a get on the ID I don't want another get on this ID I just want to get rid of that annotation on the ID parameter so I will just say copy it remove all its annotation so perfect so now I got back what I want get get ID get name ok so what I have essentially is a sequence of modifier which is the annotation to the parameter which is the arguments to the constructor so I now want to group them I want to get all the gets together get all the posts together got all the patches together to create this case classes so well what I essentially want to do is I want to go from a sequence of a comma B to a map of a comma sequence of B I really wish this util was in the scholar collections library because there is a to map on a sequence of tuples but it just gives you a map of a comma B which is almost never what I want because I want a multi maybe I don't want to just randomly pick collisions ok anyway I have yeah I will do that so I will do a let's call it grouped params start group by the first thing but then those map values have to pick the second thing so yeah this little util I always add it to my implicit sequence double thing because this is such a common thing I do have to do to go from sequence of a B to map of to a multi path I usually call it dot to multi map but yeah this is basic Scala data structure manipulations I hope this line doesn't trip you I trip you up okay so now I got the thing in a nicer format I got it in a multi map which is given a sequin given our annotation what are the other what are the things that are annotating what are all the parameters that have the get annotation so let's print out grouped okay whoops it didn't work so it looks like it didn't actually do the group by because this has a separate element for each get I it makes sense if you think about it so what's happening is every node in the abstract syntax tree are by default not equal to each other the very fact there is a get annotation on ID and I get annotation our name makes them different annotations so this makes sense like these are different things I shouldn't they should have their equality on them should be a equality by reference not by value so P so okay so what do I do one at this point I don't care I'm just gonna do it dot to string live coding okay this actually fixed my thing I collected because since I did a to string and then and the strings different equality is defined as you'd expect so I got all the a all the parameters that have a post in it all the parameters have a patch in it and all the parameters that have a get in it this is this is close to where I'm want to be okay okay so my grouped at this point is a map of string to the sequence of terms okay so map so I want to for each element in that I want to create a new case classes so what I have so this is the annotation and these are the parameters so I want to create a new case class how do I create a new case class well cheatsheet how do i create a class okay so this is how you use the queue Kwasi string interpolator this dot dot mods means you can see like a strings black thing that got borrowed from javascript or something but it's basically saying that you can put multiple modifiers you can put public annotation then you put class type name which is the name of the class new classes are new types these are type parameters again it's a dot dot because you could have multiple type parameters these are constructor modifiers again you can make your constructors of your class public or private and these are triple dot params because again the parameters of a class can its params the plural of params apparently and this triple dot on it because parameters of a class again could be curried you can define multiple parameter lists and experience template is kilometers we are saying something like a body or something else so that they use template so again all of these are not optional obviously but this is how you define a new class ok so I want to define a new class a queue case class I'll give it some class name I don't know yet what so class name I don't know some string I haven't figured it out yet and the parameters are gonna be the same so I'll use there dot dot dollar params okay so was let's see what should class name me class name should be just the annotation at this point right this is this at post I have to remove the act from the beginning so strip prefix strip prefix the at and I want to capitalize it so these are all utils that are in the nine Scala string implicit Scott string library so you can do all these things okay so it's complaining something I saying hey your class name is a string you need a type dot name okay again we know that we can create those nodes in the abstract syntax tree very easily so we do type dot name and pass in the class name okay that's that compiles and let's see we'll call them the modern I don't know new classes or something let's print line the new classes yeah this worked beautifully it created a case class post with name email a case class get with ID okay this is this is good progress I just need to stick this list of classes that I created in the companion object okay and again how do we do that again these no companion object is cell phone or in the abstract syntax tree these are proper case classes you can copy you can do deep copies and change things in them so I do have the companion object from up here that I haven't used yet so I will do companion I just want to modify its body we know the body is called the template okay we have to do this deep copy thing okay what's what do I want to overwrite in the template okay C dot copy and again how do I know all this this is just hitting command B in IntelliJ to see what else I can do are looking at the cheat sheet but in this case I would pretend that I didn't know and I hit command P in IntelliJ which shows all the parameters that I can do in the copy so I can modify the early initializers these are some weird feature of Scala that most people don't know about or shouldn't know about they're gonna be gone in future versions color so forget about it parents yeah make sense okay you can have a object standing object you can have a self parameter you know objects can have a self type and start so starts is the body I don't know it's called stats but that is the only thing that could be the body okay so starts to do like get the current stats which is inside the body it could be empty so get or else know and just append my new classes to it do it and it's an option so just some okay all right and let's return the new companion object I mean that's the only thing new we are adding we're not changing the case class so we can you can return then only the new things to and that would work okay let's see ah it worked almost it's the batch API okay so this is going to be tricky so remember the batch is an optional update so what I do is it's not as simple as the get and post where I take the name of the parameter and the type of the parameter I just plunk it in here the types that go into the batch case class a slightly special I surround them with options and I put a default argument none in them so I need to handle that okay so that's up here okay so we have to handle patch annotation slightly special so we put an add batch we remove that from here so okay I need to return something okay how do I create a new parameter cheat sheet how do I create a new parameter no not those there we go there are two kinds of parameters type parameters and term parameters this is a term parameter obviously so you use the param string interpolator again parameters can have modifiers name the type and the default argument okay all right that looks exactly like what is something I need soap around the name was gonna be the exact same name of my current parameter so prom yeah param dot name it's gonna be option some type I don't know that yet equal to none that's that's what what I want to essentially return with the option of the type of this parameter so okay so so that type should be this params type so the there you go you just I just type and something comes up that probably makes sense there we go the declared type okay so this is funny because declare type is optional that's because pieces of the Scala the abstracts in this ADT that's exposed to us as color programmers is reused in different parts of the language so you could have somewhere in the language where you don't have to specify the types and it's inferred or it could be a generic so it's an optional type but that doesn't fly here because this needs a type and I have an optional type so I will do something that you guys would love I'm gonna do a get on an option it still doesn't like it because it says okay I expected type and actually type dark okay this is because there's a type argument okay fine I'll do something else that you gonna love as instance off okay before I get booed out of here let me explain why this is okay so we all ask all the programmers hate doing a get and an option or doing a typecast because you get runtime failures but the runtime failure of a macro is during your compilation phase no big deal so you can do ugly ugly things in your macro code because it's what your runtime failure is during the compilation phase so nothing bad can happen so don't quote me on that but yes alright okay that seems to this seems about what I want and what yep green test pass that's that's the end and we can go back to the here and I can actually this I can remove this should compile for the things that should compile I can copy it and I can create those things here intelligent doesn't know that this user dot gate is materialized but we can see that the cord still compiles yep and if I remove something it will say that hey it's invalid because I didn't specify the email parameter so this is real stuff this charismatic it was being materialized and it's available for you I believe there is something here no doesn't do that okay they used to have some button here you click and you just show you things being materialized but doesn't always work but that's just an idea issue Scala meta micro this works beautifully okay yeah this is it I mean this is what 40 lines of code during life coding and it's not that bad it's something that's super useful so imagine you can take it to the next level you can imagine you have a data access layer and you can generate all the crud creative plates update delete api's for your data access layer where you want to get my ID create replace and you can do a generation of the full code you can like generate sequel you can generate caching and all that you can take it to the next level we won't have time for problem - obviously it's basically a lens macro you remember I had to do all these copy copy copy copy there's ways when you put an annotation on the def you can reduce you don't have to do all this okay and the code is up on github so the problem one is in that repo problem two is in the second one I'll put the slides up and last but not least we're recruiting I work at a hedge fund we do a lot of scholar spark big data analysis to trade on the stock market so come talk to me if you interest that sounds cool yeah you