SBTB 2015: Pathikrit Bhowmick, Get Productive with Scala Macros
Recording: SBTB 2015: Pathikrit Bhowmick, Get Productive with Scala Macros
alright so the talk is titled get productive with scala macros well that's two of these screens I don't which one to look at my name is a sabbatical atomic I work in a hedge fund i write Scala all the time and yeah and this is my first big talk presenting to a big scholar group and I decided to do live coding which is a bad idea i decided to do life coding with macros so yeah not that's not all right let's see okay so just to set the expectations of this talk it's not this talk is not about Skylab macro internals like how the compiler works with it it's not about compiler plugin see they're all this kind of related if you know macros you probably know compiler plugins it's not about scala meta and that's the new stuff that's coming out so this is literally about how to write your first big not big just first scala macro project how to get started with it so i'll go also with this part very quickly like why macros are useful macros have have understanding of more of the code that's surrounding it so let's say you wanted to write a debug macro which prints out the variable names and the file um as a file name and the line number you can do that if you just do print line it would just print out 23 hello and 19 times too that's all it does but with macros you can do some powerful stuff you can print out the variable names that's a classic example of a debug macro so so what are macros without getting too much into deep about it the one-liner would be macros are code transformers they happen during the compilation phase they take code abstract syntax tree and produces some other apps texture x3 which is some other new code so in Scala when you write macros it looks a lot like deaths it's just a man at least the Deaf macros so for example like if we wanted to write this debug macro it would look something like this the signature would look exactly like a regular scholar def which takes in any Star args returns a string but instead of the body of the Deaf you use the keyword macro and say hey I'm going to implement it somewhere else in another special debug info ok what does debug info look like it looks a lot like debug except you have some other stuff you have the context of the compilation because when you when this thing is running the debug info it runs during the compliation phase of your program so you can know a lot about it you can know the line numbers you can do c dot and closing position that line you know exactly what line is working on you can abort the compilation so if this there's something weird going on you don't know what to do with the macro you can call c dot abort and with a nice message and compliation would fail say couldn't expand the Mac macro so okay that's great you have C so where does it come from it comes from this black box context so without getting too much into the types of macros there are there is the big types of black box and white box macros black box basically means you guarantee that the code you produce would have that same signatures as expected by the compiler so i'll get to that and white box means you can generate pretty much any kind of code so with the black box macros are more well-known i like them they're kind of more strong assertions about them so let's proceed so in your debug info you okay great you got the context this part looks very similar to this part so instead of any star you get the syntax tree of the arguments instead of Arc's you get the syntax tree of the arguments and it says that you're going to get an express a syntax an expression syntax tree of any star and instead of returning a string you'd return an expression tree of strength and we'll implement it in the live coding session so you macros are everywhere you already use them a lot these are my some of my favorite libraries that use macros and they're used everywhere in play JSON is slick but most people shy away from actually using my macros themselves they would rather put it in as beauty imported and be go on it's like someone else should write the macros and let me do some coding because that's just weird but I'm here to argue that they're not scary anymore at least so before squads Kwazii courts macros were ugly because you as a programmer had to know a lot about the internal details of the Scala compiler how this track syntax tree structured and if you're not a compiler writer if you don't work in that level you don't care you don't know you're not interested in that thing so what I mean is before Kwazii codes even if you wanted to do something some simple as inject a list of one two and three this is what you have to do and no wonder no one did anything with micros you have to say that okay the one here is a constant it's wrapped in a literal expression let's put all of them together in a list that are all inside and apply and all of this is an expression with Kwazii codes just forget about all this with Kwazii courses are very nice this you get this nice q string interpolator and it has two use cases you can use it like this so this would literally means a fragment of code as piece of syntax tree abstract syntax tree which ivalice to list one two three great that's it this is equivalent now to what was pre Kwazii codes what this was but that's not the only thing you can use Kwazii codes in many powerful ways Kwazii course are also extractors so you can use them on the left hand side so given a function given the abstract syntax tree of a function you can do pattern matching you can say I know that a function has arguments and a body let me get them out and you can get arguments and the body separately by using something like this is almost like regex extractors you can say that let me do that my reg ex pattern on the left hand side and the actual string regex on the right hand side and similarly obviously obviously you can do this you can also do the regular pattern matching so you can say given a tree this is abstract syntax tree again match if so what is this pattern this pattern basically means it's an identity function it takes some X returns that X back and it's again in the queue string interpolator and you can do something with it you can manipulate the tree Amit and yew tree throw some assertions whatnot and if it's not an identity / function you can take that context and abort the compliation and give a nice message like hey I was expecting another identity function but yeah the point is this is not scary this is what we as software engineers normally are at the level of abstraction at the level of code we're not at the level of the compiler and that's the main disconnect that I want to say that it's not scary anymore so okay so let's look at a real world problem so if you're familiar with lenses this would already look familiar to you but let's take a real concrete example let's write a library that solves this problem so this is assumed that let's see so let's say you have a garage a garage has a car in it a car has an engine in it an engine has a version in it so you see where this is going and scala these are all the lines case classes they are immutable when you want to update something you gotta get back a new copy of it so let's say you have to write some code like this which is upgrade the engine and you're only given the garage so what can you do you can return a new garage back and your coat looks like this you copy the garage you copy the car inside the garage you copy the engine inside the car inside the garage you add one to the version and you return it this code looks pretty but yeah you can smell it like this it's a lot of code like all you want to do is this thing so how can we we can do better we can do much much much better so the first example imagine you had something like this an update function so update function has two things and be two types it words it's a parameter so it takes in our original model so let's in this case your garage it takes in a selector from A to B so from garage to version so selector basically says given a garage how do I get to the version of the engine of the car inside the garage that's all it is an updater says okay if you get the version what do you want to do with it I want update it to some new version and you want to get back a new a so that's all that this is update and this is how you'd use it so you say update you give your garage instance and this is given a garage this is a function which given a garage returns the version and what you want to do with the version you want to add one to it so how do you write this without macros it's almost impossible to write this that's it's impossible you would have to do very fancy type classing where you can say like this is copyable this is updateable and the internal you need to be familiar with this is something that may be shapeless can do with some magic but it's very very hard very hard what I'm going to show you is literally going to be seven or eight lines of code so how are we going to do this we're going to use a very nice property of this update function so what we'll say is ah if you want to do this and if this update is implemented correctly its equivalent of doing this which is basically you copy the garage using without macros using the copy of the case class and you update the car inside it and instead of passing in the garage you pass in this this my garage car and you give the path from car to the version and you pass in the same function so I hope it's clear to you that this will work I mean it compiles so obviously is correct so and okay so what happens next so if you you see where I'm going this is this is recursive what's the base case of the depression the base case of the recursion is when you are updating the leaf leaf node which is the version where X can be any X in this case is version and you you don't have a pot anymore you have the identity function because you're just updating the same thing and you have the f f is this whatever you're adding one to it so this is the base case this f of X so given this nice rule this is called if you from category 3 this is a telescoping of lenses so lens this is a big lens you compacted you compact it's like a telescope anyway that doesn't matter but this is a nice property will exploit so with macros what we try to do is when we see a code like this we will pattern match and see I on this part on this underscore card or engineered version if it is of length or if it has some if it's not an identity function will try to move the update inside if it is an identity function will just simply apply f of X so i'm going to set up the mechanics so the only thing we really need to do like all this is easy the only thing we really need to do is break this up so we have this nice path of underscore car door engine that version we need to tokenize it we need to clip pick out the car so we can put it here and remove it from this part so that we can write this part here the rest is simple and we need to put the correct card in here also so big crashed our car and the rest is simple this this function remains the same this is whatever you got so let's let's see so this is the fun part okay so this is a brand-new svt project there's nothing funny going on it has one dependency on scholar test and scholar reflect nothing I enabled language colon experimental macros that's it and the only thing i copy pasted was the test case so this is I in the interest of time I didn't want to write the test case out but I copied from the slide so you can have the garage car engine you're upgrading the engine this is a regular scholar deaf and and you create an instance of garage g0 it's my garage it has a BMW in it it as a I drive engine and its version is 2014 and you have g one which is using the regular non macro version this guy there's the four or five lines of code and hopefully you got back some different g1 but we're going to try right the second version g2 using our macro this is again copied from the slide and these are some assertions that's all there it is this is the test case ok so let's write the update function so this is cop I'm literally reiterating what's on the slide so we want to update a model of Taipei it has a selector function which take which still see how to go from A to B it has an updater function which tells you given B what newbie want you make and it should return something of the original type a ok how do you implement it well we're going to say it's a macro so we're going to use the macro keyword and say it's implemented somewhere else ok how is it implemented somewhere else so let's look at it so we'll write update info right next to it it again takes in to type parameters a and B its kind of mirror very closely what update looks like so the first thing it needs is the context so it's not the string contest doesn't suggest it's the scholar dot reflect no not remote reflect dot what is it macros dot black box this is the context i want since i'm going to write a black box macro I mean that's the macro we're all familiar with I am familiar with so and it takes in a model in so instead of it being an a is the abstract syntax tree of a so that's what it is it's an expression okay up type a and it's an inner type of the context because this thing can change based on context so but you don't need to worry about that so a selector is again instead of being a function from A to B it's the extraction tax free of the function whose type is of A to B and updater say you guess where it's going is the same deal it's instead of being a function from B to B it's the I think i'm going out of the screen so i'll put it in the next line so it's the syntax tree of the function from B to B and what you want to return you want to return instead of a you want to return the abstract syntax tree of something that it off taipei okay let's do this what's the simplest thing I can do I can do nothing I can return the original model back this compiles and let's see if it works we want so I hope it doesn't download choice okay that's too big font yeah it didn't work because what i did was i just returned the original model back it says 2014 did not because i didn't do anything with it okay let's try to do something interesting with it as i said the what we want to do is break the selector apart like everything else it remains the same updater for she remains the same the selector we need to tokenize it so we want to break this underscore dart x dot y dot z into X comma Y or Z so okay let's look at what the selector looks like so this is the selector so remind you at this point this is not the argument this is again the everything at this point you have is not the values this is expressed the abstract syntax tree of this argument ok so we let's see so selector tree what what is it we know it's a function so let's pattern match it's so what's the pattern matching for a function something like that it has an argument and it has a body and you put it in this queue magic integrator so yeah let's see if that works and let's is complaining that it doesn't know what the interpolator is so you need to import all that using you need to import every mechanism from this universe of this context so you need this thing up here so now it knows what q is so I don't know what the body looks like do you know I don't know like I have a guess it looks like underscore dot card or engendered version what up so what i can do is print line my good ol D Butler so there's this nice little you till that scholar macro paradise provides called show code and you can pass in the body and let's see and I mean I can't put a breakpoint here because this is code that's executed during compilation so okay so tipped it there you go it's printed the body art cool this is exactly what I expected to be it's gonna put this is what underscored card on engendered version looks like to the compiler awesome so I need to disassemble the body so what I want to do is so given an under so given some code that looks like underscored card or engendered version I want to return list c EV i want to tokenize it because i need to take that car portion out and move it in the front so what do I pass in well I pass in whatever body is so body is let's see it's just an expression at this point because so body which is of type c dot expression or sorry it's a syntax tree so it's a cedar tree that's the most genetic thing I can talk about it because if i use expression i have to give it a type at this point the type is hard because i'm going to use it recursively so given see the tree i want to return a list of terms which are these things so some of you might be asking how did I know it was a tree how did I know that it's going to see dot term name there is a nice color doc page which lists all the things possible you can also do instead of show card you can do show raw and this would print out what it was so Sherrod say that it's select select select term term term term so yeah I want to term if you still can't figure it out ask on stackoverflow it will get answered in like one minute by someone from this conference so so yeah there are many ways so that was one of the hotels like you don't know what body is what is the pattern matcher was the correct pattern matcher to use you can use either you show code or so raw I just ask anyway that part is actually very easy so what I want to do is so the what does body look at this point good ol print line well I know what body looks like it looks like somewhere back up here it was list of something dot something dot something dot ok well let's pattern match our favorite tool so it has a parent and a child so in this case it's not Ruby or anything so parent at this point would be if you're given underscored c dot e dot V as body the parent would be till this point in time which are till the till the it's basically the head or the inert of the body and v is the last thing how do I know that well again print line show called parent and child okay so let's we're going to recursively disassemble the parent and append the child do it so this is nothing fancy I'm doing all I'm saying is recursively I know how to disassemble cev how do I do that this is amber C E and append v to it and what's the base case well the base case is this is not the actual base case but I'll go with it so this cases you got back the identity which doesn't have a parent a child but at this point if you're a real library writer you would do some more assertions you want to make sure that this is the identity function and if it's anything else if it's anything else you want to abort the compilation and say where it failed in the enclosing position and you want to give it a nice error message saying expected something like underscored art x dot y dot Z else this macro doesn't work it only works for case classes with nice like or like nice functions which has like food bar dot Z so but that's fine we are not writing real life coding during a conference so we can say that this is what it is I mean worst case it will happen the pattern matching would fail and we'd get a match error which is fine too so back to this trick I'm going to use I'm going to move the update one level down and pick the car up put it here pick the car up put it here change the selector function alright let's do that that should be easy so again i'm going to use q but on the right hand side now on the right hand side i'm going to use it as to create new syntax tree so what i'm going to do is do dollar model so this is the outer model cut copy well let's disassemble the body first sore this semble the body and let's do pattern match on tokens so this is the first token rest of the tokens this is so t1 is the car model is the garage so I want to do t 1 equal to 0 I'm moving the update down that they use dollar t1 so you I don't use the string T 1 so I want to update dollar model dot dollar t1 this is not something magic I'm writing a list a dreadful writing this code this thing in the end against I'm calling model is my garage t1 is car that's all I'm doing so and we need to pass a new function to read a new selector function but the updated function remains the same so the updater pass down the updater and the base case of the recursion is when you don't have any tokens so the base case of the recursion this one you don't have any more tokens you just apply F 2 X so you just say dollar updater on dollar model that's all I did okay so now the one missing part is the selector function so what the selector is does is takes this car engine version takes a car out so we need to read so we already have lists cev we need to reassemble it so reassemble the body back so now we have that this exactly the opposite of disassemble so now I have a list of terms list of C dot term name and I want to return a cedar tree and this is actually so what I'm doing is list of c e v I want to return a tree which looks like underscored seedot seedot be because I'm putting the selector back again so it uh huh ah yeah tokens good point and now it should be fine yeah thank you so how do we do this well this is actually easy we just fold it what is so for left and we start with what what do we fall left with well what's on the left hand side this guy what is this this is the identity function how do you write an identity function so boom that's an identity function and what's the function what's the folder so this is the function and you have tea you need to now just append the tea the last these terms at the body of the function so function again we're gonna pattern match exactly this thing because that's what functions look like as far as quasi cold Scala compiler is concerned and we want to return instead of dollar body we're going to return dollar T where if you see that I'm just reassembling and depending starting with the underscore the identity function I am putting the sea in the end then putting me in the end anything there's a simple fold operation so let's re so here what I want to do is reassemble the tail okay did i misspell resemble see I'm missing a bracket so it's kind of hard to see yeah it's missing a prayer and inside a quazy cord somewhere full of love oh I see which one ah this thing yeah you're right and I don't need a dollar anymore because it's a sender curly and the updater is fine because it's a single yeah the rest of them are fine yeah let's see and now instead of passing in well let's print out what this will this code new record rook looks like so good ol print line and start returning back the original model will return code dot third code just print out the car so now I hit an interesting thing is complaining that hey I'm not returning c dot expression anymore because at this point the type checker gave up and said like I don't know this is off c of expression a so i will say like Ivan if the black box macros let me relax a bit it's going to be c dot tree I'm returnin abstract syntax tree instead that's fine if you can hit alt enter IntelliJ will do that for you which is fine so let's see up and actually return the tree you have a string s string interpolator of the app Thanks having a debugger around ok let's complaining about disassembled somehow so let's look at this assemble yeah right so what I want to do is this yep that work my tests pass and you can see it's recursively resolving the update is doing the first time it's moving the update out from the top level saying let's update the car when it sees that it moves the update inside the engine and when it sees that it moves the update inside the version and this is the base case it just says like hey this I'm going to update this anonymous function which is add one to the version and voila it works and and this is this is not hard code like I was super nervous life coding and somehow managed with the debugger from there but anyway back time yeah cool we don't have time for problem two but this is another real-life problem where you have a business model and you want to write a strongly typed rest api so let's say when you are posting a user you don't have his ID it's created by the database when you get a user you don't want to pass the registration time along and when you patch a user you can only patch the name ok fine you wrote for case for class is great but now you have to add phone number ah now you have to go put it in all of these classes figure out how they map to the database because of how they map to the original business layer and these are your presentation layer models so can be solved as macros it's about 30 lines of code you can use instead of def macro sanitation macros you can say this ID is only in the get model this is name is in the get post patch email and registered on it's never exposed to front end api's that's possible so there's more to macros like all these things implicit macros fun dematerialisation and they all sound complicated no one wants to get into it are the people who wants to get into it are like the few people in this conference but if someone shows you how to get started it's not scary it's actually very easy it's something worth doing a scholar workshop on like going through all this life coding and one day everyone would be an expert and it would be awesome to use it in your own libraries yeah that's my talk the real these are the actual solutions to problem 1 and 2 these are real life libraries that are available on Ben Tre I use them in my projects every day some other people also use it so thanks any questions yeah it's it's it's as long as as long its sorry since if it's if it you did something bad it it will not compile so that in my definition is good the code yeah right right right you can make actually in the libe in the link i posted this is actually is evaluate its of its evaluated as a block which applies the function and that's a small change you can do but yeah i agree but the generated code looks very similar to this and and you can do a lot of powerful things like you can you can pass you can create a lens operator which just has this too when this the third one is anonymized and you can reuse that lenses to apply em on the same object yeah it's very forgiving you can you can put this thing in wherever i use the identity somewhere i think i've been in my fault you can put arguments around it is fine yeah yeah yeah yeah it actually uses like the scala AST transformer to actually parse the quaza card out into the pre Kwazii codes color st so if that works it will probably work so it's not out even sees like I'm not the macro internal expert but it works it's very forgiving all right okay