Compiler Plugins 101

SF Scala: Seth Tisue, Scala Compiler Plugins 101

Recording: SF Scala: Seth Tisue, Scala Compiler Plugins 101

[Music] [Applause] hello everybody thank you to rally for hosting and into cellar and Lexie to inviting for inviting me so the subject tonight is compiler plugins I hope to make this interesting or relevant to kind of two different groups of people in the in the first half of the talk I'll be talking about what compiler plugins even are what compiler plugins can do which ones are popular so that the whole first half will be from just a Scala user perspective and then in the second half of the talk I'll change gears and and talk about how compiler plugins are actually created and I'll show you a sample plug-in I should say something about timing maybe I the first time I gave this talk it was 30 minutes with you know no questions last time I gave it people had tons of questions and and we ended up doing 490 which was maybe a kind of long but I it's it's fine if you have questions please feel free to ask them I more than happy to have this be interactive you know I'm the only speaker tonight so we've we've got we got plenty of time I've been programming install a full time since since 2008 I joined the Scala team at like Bend in 2015 moved here to San Francisco two years ago the reason that I selected this particular topic to do a talk about is that I've written several compiler plugins while working at light Bend one of them is called sculpt and that's open source now I've put the the repo URL up there if you're curious about it there's I have a separate talk about it called tools for taming tangled code video of that is online it's basically a plugin that analyzes the structure of your of your code so not dependencies in the sense of external libraries that you depend on but dependencies within the code like from class to class or from source file to source file if you're in a architect role or doing a lot of refactoring then sculpt Asst is is something that could help you so that was my first compiler plug-in the 2nd compiler plugin I wrote is called fortify SCA for Scala so that's a commercial product from light Bend that we did in partnership with microfocus who fortifies their product it's it analyzes source code and tries to find security vulnerabilities like security mistakes in source code they support tons of programming languages there's more than 20 and so my role in light bends role was to add Scala to the list of supported languages and we did that by writing a compiler plugin so this is an example of a compiler plugin that's an alternate back-end to the compiler it translates Scala into an intermediate language that microfocus invented and then they do their their scanning on that so those are my first two plugins and then the third one I wrote came about because I'm the primary maintainer of the Scala community build which is like a big met ibill of lots of Scala open source projects that we compile together in order to test changes to Scala itself as well as to help the ecosystem make sure that their that their libraries interoperate so I wanted to write a blog post about the community build I wanted to know how many lines of code were in it so I wrote a little plugin called C c-loc and that was surprisingly easy and it turned out to be only 45 lines of code so 41 lines of Scala 4 lines of XML that's kind of the the minimal example of how easy it can be to write a compiled plugin so that's that's kind of where that I got the idea for the talk and that's the that's the the code that I'll show you later on one of two example plugins that that I'll show okay so so part 1 of this talk is you know what what are plugins what which ones are popular and how do i how do i use them as a scala programmer I I should start by just defining my terms briefly I mean that this this question will get answered in you know more fully as the talk progresses but a scholar compiler plug-in is an extension to the Scala compiler that allows you to alter its behavior without having to change the compiler itself so it's an API that you write to and then users can you can use your alterations to the compiler without having to use a totally totally different compiler as we'll see the variety of things that you can actually do with with a plug-in is is pretty wide I did a kind of survey to try and find out what open-source plugins were out there and which ones were widely used so you never really have to have good data on usage I used github stars you know which is of course an imperfect metric but I would say that the results I got that way you know kind of informally matched my own impressions from what I've seen people using what's out there I won't explain what what every one of these plugins does obviously but but on on the next slide I'll sort them into some categories to give you an idea of the space of things that are popular but right away you'll notice a few things at the top of this list that are really important and that lots of people are using so I especially want to call attention to zinc I put a star next to this one because I'm cheating a little by including it it's not technically a compiler plug-in but that's purely for historical reasons it should be it ought to be we'd only have to change a few lines of code so it's sort of morally a compiler plug-in zinc is the part of SBT that does incremental compilation and it's it's separate from SBT because other build tools like maven and great I'll also use it in order to so you don't have to recompile your whole your whole source code you know every time you just change one file Scala is installed stalled native or you know well-known non JVM backends for Scala and somewhat farther down the list here but but definitely rising is Scala fix I've put a star next to that one as well because it's actually not a compiler plug-in itself but most of the a great deal of the functionality and it is coming from from a compiler plug-in called Scala meta which is a little farther up the list Scala meta is a plug-in that's the the basis for the current and next generation of Scala tooling so it sort of extracts syntactic and semantic information from Scala code and makes it available to tool authors so I think that means that over the next few years this list it may actually shrink right because we'll see that that some of these some of these plugins kind of predate Scala meta and might be better implemented on on top of Scala meta if they were starting if they were starting today okay before I go on I want to you've had a while now to sort of look at this list is anyone using a compiled plugin that you know of that I did I missed no okay all right good I've I previous times I did in this talk I've had to add a few and I tried to remove one the author was there and he was like no that's actually not a compiler plugin so yo sorry hopefully the rest are accurate okay so all these plugins what what do they do what's what's possible with these things I've tried to separate them into some big categories that are separated by the divider lines and then even further subcategorized so the the top group that the top three bullets are plugins that actually change the language that add add things to the language itself make new kinds of make writing new kinds of code possible a compiler plugin can extend the syntax of scala kind projector which is I think in almost universal use among people who do pure functional programming in Scala adds better syntax for type lambdas the other syntactic extensions that I found are more experimental I'm not sure there is widely used but there's sort of experiments with improvements to what kind of for comprehensions you can write semantic extensions actually change the not just for you know provide a different surface for for writing something that is otherwise already possible more verbose ly with semantic extensions these are actually enabling an abling semantics that weren't possible before Scala continuations adds delimited continuations and two tails adds mutual tail recursion finally mackerel paradise is a compiler plug-in it's so you may be familiar with macros in Scala or at least have heard of them so macros and compiler plugins are definitely in the same space here what macro paradise is is it's like advanced extensions to Scala macros that are that that didn't make it into the into the macros that are built into Scala so macros of all kinds are experimental in Scala but that you know there's some the macro paradise ones are like a separate module that are especially experimental so that's that's the top category extending Scala itself the second category is tooling so these are plugins that don't change what code you would write but just inspect and help you with the the code you're already writing the the biggest category is linters Scala fix which I already mentioned fits into that category it does not only linting but also rewriting but it's a it's a linter as well and the others on this list are are all linters of various kinds some of them are general linters I think wart remover is probably the most widely used and then the others are do more targeted kinds of linting fortify as I mentioned is like security linting sculpt and a cyclic help you with linting program structure and so forth other kinds of tooling include plugins that improve errors and warnings coming out of the compiler help you do documentation or in or in turn internationalization I already explained zinc that's kind of a category unto itself what wasn't sure where else to put it and then the final category is alternate back ends astrology yes of course from piles to JavaScript and Scala native compiles Scala to LLVM which can then be further compiled by the standard LLVM backends to native code either for you know for iOS or for command-line tools and so forth so hopefully this gives you an idea of the the space of things that that are possible with with these plugins in order to actually use a compiler plug-in the the easiest experience you can have with this and also a very common one is that compiler plugin authors will make their plugin easier to use by packaging it within an SBT plugin and so then adding it to your build just requires this one line add SBT plug-in and then you specify the the you know maven coordinates to the jar for the plug-in the the usual SBT way that makes it easier to add and then also the the an SPT plug-in that wraps a compiler plug-in it may provide extra settings and tasks for for internet for configuring or for interacting with the plug-in so you may you may have this in your build and not realize that there's a compiler plug-in behind it if the compiler plugin doesn't have an SBT wrapper then adding it to your build looks like this there's another convenience method called add compiler plug-in again you take the maven coordinates and then you you you typically specify this prospers and patch thing so that's mysterious-looking it's it's maybe worth mentioning because a useful thing to know about about compiler plugins they usually depend on Scala compiler internals Scala compiler internals can change even in Scala minor versions so we we try not to you know change things around too much because it's it's disruptive for plugin authors but we don't have the we don't have the same hard compatibility compatibility guarantees on the compiler that we do on the standard library so compiler plugins should be published according to the specific Scala minor version that it's intended to work with and that's what cross version dot patch gives you it's common for a compiler plug-in to take in configuration options and so this - capital P for plugin is the standard way that that plugins accept those options so it's it's - P the name of the plug-in and then the name of the option equals and the value and then of course a plug-in can define as many of those and and with whatever meaning that that they want if IDI compiler plug-in didn't exist or if you're not using SBT then you know you won't have access to it then you you may need to know the the fully manual way for for enabling a plugin and I've actually given it an SBT syntax just because that's what I'm most familiar with but you know every build tool has its own you know has its own syntax for adding a dependency and for specifying a compiler option it works works with command-line Scala C as well so the this is all that that edie compiler plugin is doing it's really a very you know thin wrapper for this the jar for the compiled plugin needs to be on the compilers class path and we see this extra thing with configurations and this obscure maven syntax for that what that does is it it ensures that the compiled plug-in is only on your class path at compile time right you don't want to have it end up being a dependency for your users at runtime and that's that's how you tell the build tool to do that if you're curious what compiler plugins might already exist in your build there's a flag to Scala C X plug-in list that will just dump a list of which ones are enabled it's also possible to just look at your list of dependencies in SBT and see which ones have that that magic plug-in default compile string on them and that that's an example with an actual project of how to get how to get a list like that I just before I move on I'll just pause and ask if there's any questions about about anything that I've shown so far this is this is kind of the easy part okay another slide of advice for you as a prospective compiler plug-in user if you're thinking about using a plug-in these are questions that you can that you should consider and the one I've put first is is the the one that I think is most important because it's just be people should do this and just don't so check whether it affects your build times you can and you know that the concern on this is is I don't know the the more popular the plug-in the the the likelier that the author has kind of checked that it doesn't slow down compile compile times too much but lesser used plugins that's more likely not to have been checked so I would recommend measuring that Scala in general has a reputation for slow compile times and certainly a lot of that a lot of that reputation is deserved and and is its it's true of vanilla Scala but you know it light bend what we are often working with customers who you know have slow build times and want help doing something about that and it's it's incredible actually how often it turns out that it's coming from the sole compile times are because of either compiler plugins or macros the the it's surprisingly rare that it's just that is just the vanilla vanilla Scala that's that's taking so long if you add a plug-in to your build you should ask whether the plug-in supports all of Scala and in that there's a whole that there's a whole spectrum of answers on this to whether it's a hundred percent or whether this is a really serious concern so and some of this advice really overlaps between compiler plugins and macros so the a plug-in or macro is especially likely to you know have unpleasant surprises about what subset of Scala it supports if it's actually rearranging code so something like the Scala continuations plug-in which does a continuation passing style transform on your code writing a plug-in or macro that does that is it's really hard to get right it's it's hard to get it right even for 90% of Scala code let alone a hundred percent so do but do be careful with that on the on a more positive note the category of plugins that run that are alternate backends like Scala Jas and Scala native and fortify those plugins almost all the time support like literally a hundred percent of Scala or the you know like first college yes it's like 99% there's like these really obscure edge cases around you know behavior of numerics for example where the semantics can be a little different but basically the the later in the compilation pipeline that a plug-in runs the later in the compilation process all of the the diversity of the Scala language all of these features that exist have been eliminated by the time you get to the end of the compilation pipeline your Scala code has been transformed into something that closely resembles Java I mean not even Java current of it resembles like Java one Java 1.1 and also is only a short distance from Java bytecode so it sometimes surprises people that that's college is install native support of the full language but that it's that that was something that sort of pleasantly fell out of the inherent structure of the compiler you should check if there are binary compatibility concerns that's usually not the case with a compiler plug-in but something to check on you should be concerned about tooling support so if I use this plug-in am I gonna have lots of red squiggles in IntelliJ because it doesn't know because it doesn't know about this change to the language that the plug-in makes this is very common with macros as well jetbrains does offer an API that lets macro authors and plugin authors extend in IntelliJ to understand these language extensions but you know it's relatively rare that people actually do that would do the work of abusing that that API to add support it's also possible to confuse not just an IDE but also other kinds of tooling like like Scala fix or linters and what-have-you you should also be asking yourself is this going to how is this gonna fly with my team right at my company how does this affect compatibility with with people and this is especially a concern for anything that makes some sort of alteration to the language is at that point you're no longer using vanilla Scala you're sort of working with a fork of the language and that's a pretty serious liability for a team to take on you know maybe it's it's worth it something to think about okay any further any further questions anyone will ask the first question about about the stuff so far yes yeah great great question so the the question is is a common for for plugins to conflict with each other what what happens if I'm using lots of them I would have to say in my experience it's surprisingly rare that there are conflicts I I can't even think of a single common well known conflict that people run into so yeah is there anything else to say about that it's certainly possible but I I just haven't seen it be a concern in practice as well like why that might be I I think will maybe become a little clearer once I start talking about the internals like how plugins what the API looks like because there there are enough restrictions on there are enough restrictions on plugins that are sort of designed to keep them out of each other's hair any other questions on on part one okay yeah dude good question question is ours our scale are the compiler plugins written in Scala yes I think they all are so I don't know of a counterexample you the API I'm not actually sure if it would be possible to write a plug-in in a different jvm language I would have to really go over the API and look what and look and see whether there's anything in it that that is Scala specific as we'll see in the sample plugin there's a sort of nesting we have the plug-in which has a component which has a phase and boy I would really have to study it to see whether there's there's some whether there's something that that would where you would get stuck writing in Java or something else but as far as I know this is something that everyone uses Scala itself for the Scala the Scala compiler itself is is a hundred percent 100 percent Scala there's the Scala standard library is also you know almost 100 percent Scala there's a few sort of fragments that are that are in Java any other questions okay okay so suppose suppose you what you want to actually make one of these things what does that look like I'll continue with more cautions for scaring you away about whether whether you really want to do this as I mentioned Scala Mehta is a major alternative that you should consider if you're doing tooling so you should look at Scala Mehta asked whether it can submit it can support what what you're trying to do and definitely know that that route instead if you can Scala meta these days is not so new it's it's been around what is it two three years but that there's a major piece of it that only fell into place this spring which is semantic analysis so if you need to if your tooling actually needs code which has been type checked and has the types attached to every node Scala meta didn't didn't do that until until fairly recently I said when I first gave this talk in March it was like super brand new and now it's now it's like it's been out for about six months and this actually seems quite robust our experience with this on Scala fix has has been great that it's the Scala meta powered ability to to understand typed code seems seems really solid and that's it's it's it shouldn't be too surprising that that it worked out that way because it's actually it's it's they didn't independently write a separate type checker for Scala meta it just it calls in to the type or phase of the actual Scala compiler in order to get the information out so this was like a glue job rather than a building job if you're interested in extending or altering the Stahl language or doing any kind of meta programming certainly your first choices are art to do macros if you can including the expand expanded set of capabilities that are in macro paradise so macros are less powerful and I should I should try to explain briefly like what you know what is the difference a compiler plug-in has much more freedom a compiler plugin can do global analysis of an entire program it has free rein to arbitrarily inspect and and rewrite whole programs as well as parts of programs macros are a much more targeted tool so anytime you someone calls a macro that macro is only free to alter the meaning of that one piece of code that's that's inside the body of the of the macro call with macro paradise you have somewhat more power because macro Paradise adds something called macro annotations or annotation macros that people seem to say it both both ways around so ordinary macros are def macros which means that that when you use the macro it syntactically resembles a method call with a annotation macro you cue attach a annotation to a piece of code which you know looks the same as any Java or Scala annotation but then that the macro is then free to is Frieda - to not only to rewrite the thing that you annotated but also to introduce new definitions so for example if you want like a common use of annotation macros is to do things that you might otherwise accomplish with with code generation like supposed to suppose Scala didn't have case classes and I wanted to add them to the language I could do that with an annotation macro so that involves not only rewriting the the definition of the class that I've you know annotated with this imaginary case annotation but it also involves introducing a new definition which is the companion object so doing something like that it is within the power of of macro paradise so those are alternatives to consider and as always in programming you should choose the the least powerful tool that will that will do the job I'll if your if anyone is wondering at this point about Scala 3 and how this might change I have a slide on that later it's a question I'll come back to ok so you've decided you've ignored my my cautions and and decided to dive in and do this anyway if you google Scala compiler plug-in you'll get a page on the Scala site which you'll see here hasn't been updated since July 2008 so that that was the situation when I started preparing this talk so a side effect of me writing this talk was that I actually went in and updated that so now on the Scala documentation site doc stalin org where there used to be this blake's blank space under meta programming there is now a compiler plugins section so took that page from 2008 stuck my name on it and and next Alexis and brought it up to date so it's it may not be perfect now but certainly everything that that I could think of or see to to update us now is now current and accurate again okay leading in now to actually writing the code for your plugin in order to plan how your your plugin is going to work you need to know kind of what the the shape of the api is that that you're going to be writing to so i've contrasted compiler plugins with macros by saying they're much more powerful they can do anything that's you know not entirely true you don't have to because this is the JVM you you don't have the ability to just arbitrarily reach into any point of the you know any class and the compiler and just change anything you want so you don't have you don't have that totally arbitrary freedom you also are not completely free in terms of deciding when your code will run the code in your plugin so the power that you do have is to insert a new compiler phase between existing phases that that is that is you know basically the one thing that's in the API add a phase you can decide when that phase runs but it's always going to be it's always going to be a separate phase that that's waits for other things to finish runs and then is done and hands-off control to the rest of compilation so that those are the those are the limitations however what you know once your compiler plug-in once your new compiler phase is active you have complete power you have access to NSC global NSC stands for new Scala compiler because it was new in like 2004 or something like that so it's it's still still NSC global is an object in that package that basically it's like the it's the God object for the compiler it has access to everything so from there you can you can get to everything and you're you're free to access and manipulate anything so to write your plugin you're going to have to decide when you want your new compiler phase to run if you run Scala C with this flag dash X show phases it will print a list of the existing compiler phases and then exit what I've put on this slide is actually not the full list so you you'll see some of the numbers are missing I shortened it to what I would consider the major phases like that the ones that anyone doing compiler work is is most familiar with and most likely to have to deal with so parsing comes first of course and then I've highlighted two phases that are especially especially important in terms of deciding when you run phase number four is typer so that's when type checking happens you'll need to decide whether you're whether it's appropriate for your plugin to run before typer and process unto untyped trees or run after and deal with types the advantage of having types obviously is that there's you know you need types to do many tasks but it does also increase the the the difficulty of what you're doing just because the you know two types are added information it's more complexity and and they're also rather difficult to manipulate so if you need to if you're just changing the syntax manipulating you're rewriting untyped trees it's really a ton easier than than then the type ones and then I've also highlighted the last phase which is generate JVM bytecode for so for back-end plugins like Scala Jas install native those will run usually immediately before the JVM phase or maybe a little bit but a little bit sooner before how how much sooner you might want to run it depends on how high level or low level the language that you're compiling to is so for example in the fortify plug-in I was compiling new language that doesn't have lambdas so I wanted to wait until deal and if I had gotten rid of that to make it easier for me but like in Scala j/s they're compiling to javascript javascript has lambdas and so you you you want you want to keep those and in the code that's that's the input to the plug-in I didn't do a like a full survey of all the the plugins like in what what phase that they run after but I looked at some of the most common and popular ones so the the easiest ones are pre typer those are to do syntactic transformations count lines of code the sample that I'm going to show you know that also that the obviously it doesn't need types so that that we can run right at the top linters typically run post type err anything that does errors and warnings also typically runs immediately or almost immediately after typer that might be a bit surprising in the case of errors and warnings just it's like well you know what if the error happens later but it turns out that almost any error that you can make in Scala code is going to come out of it's going to come and come out of typer or earlier it's actually pretty rare that invalid code is detected later I think almost everyone who writes Scala actually kind of has a intuition for this just as you know you hit compile and it's like either you get an error in the first like three seconds or the computer is just sitting there and you're like yes okay my codes right and it's just like grinding through the the process of getting all the way through to bytecode so that it's just another more evidence of that property of one when errors happen I mentioned back-end plugins run pre JVM I only found one plug-in that runs after byte code generation which which is s coverage so s coverage is a tool I think I've been think pretty much the standard tool for measuring code coverage and Scala it's doing bytecode instrumentation where it's actually inserting stuff into the end of the bytecode so that that seems to be you know fairly rare category but but it does exist okay let's actually look now at an example of almost the simple possible simplest possible plugin this is the URL if the Reap of the repo if you're interested in taking a look at it afterwards and if you're going to write a plug-in I would suggest like literally copying and pasting this repo just just borrow the whole thing and then and then rip out the stuff that you don't need so I won't show everything that's demonstrated here but there are I'm basically just going to show you two files but the the rest of the stuff in the repo actually is useful in terms of illustrating build and and testing so the important files here are in the resources directory you need to write some XML but only for lines so that you you just need to tell the compiler what the name of your plug-in is and the main class where the where the code lives putting this in source main resources SBT loops SBT will automatically add that to the jar in the right place so that the compiler will find it so that's we've spent four lines of our 45 line budget and now in source main Scala is the actual Scala code is this well I just going to say is that font size large enough but that Blanc was the browser telling me that it's as big as it gets so I hope you can see this I mentioned that everything in the compiler is in this NSC package under Scala tools there's a sub package plugins which is where the the compiler plug-in API lives so if we sort of let's just get an overview of this entire file so the stuff at the bottom is artillery code the that's just this is just a code for invoking this external tool I I got a last time I gave this talk I got a question of like well you know what why do you why do you need this external tool to count lines of code so I'll just take a slight tangent on that to say this is this great program that you can apt-get install or brew install that actually knows lots of different programming languages and can can filter out comments and in blank lines and categorized the code by by programming language so you can run this you can run this and get a breakdown same sort of breakdown you might see on github of what languages you're using and how many you know source lines of codes like real lines of code not not counting comments and so forth so that that's just helper stuff the structure of the plug-in itself is is nested in three parts so we define our plugin which extends plug-in within the plug-in we have a component which extends plug-in component and then within the component is a phase which extends standard phase I don't know the answers to all the questions you might have about this like why is there this separation you I I looked at some plugins and asked some people and and you know I was like well why is why are these things separate and it doesn't seem to be you know nobody knew so I I think it's certainly the plugins that I've written all just have a single component and a single phase but but you're certainly free to have multiple phases if you want multiple components I have no idea what that's for okay at the plugin level you're repeating the name of your plugin which was also in the XML there's a little description which will show up in in X show plugins or whatever that flag was called and then you have an it method that where you can do any initialization but also this is where you do options processing so we saw that when you're using a plug-in unless you're using it through some sort of SBT plug-in facade that provides settings unless it's happening that way you pass options with this - P colon plug-in name thing that this is where those options will actually turn up in your code so that this plug-in doesn't doesn't take any options the plug-in specifies a list of components normally there's just one of those the component is where you specify where you run so REMS runs after and runs before are how you specify that here's another place where there are questions about about compiler plugins that I can't answer but that also I know that nobody else could answer either so I can I can assure you that it doesn't matter at least like 99% of the time so if you look at the compiler plugins documentation you'll see a reference to a PDF of a long paper that describes this elaborate scheme in the stuff in the Scala compiler for phase ordering so you might expect you just you would just say I want to run after this phase but but know you there's a constraint solver and you're writing inputs to this constraint solver and the solver runs and and orders the phases don't worry about that every plugin I looked at simply used runs after and before you need to be you need to specify both of them because the semantics are runs of runs after are runs sometime after and so I actually screwed this up in the fortify plugin I said runs after this phase and then like six months later I'd was investigating some bug and realized I was running at a different a different point then I thought I was so definitely specify both and you'll notice that there's a method if you looked at the API there's a method called runs right after which sounds very appealing don't use that because your plugin will work but if any two plugins use runs right after then the compiler will say no you know chant do that so be friendly to other plugins and don't use runs right after that was a mistake that I made in this plug-in before before someone corrected me okay within next level of nesting um let's see I've been yakking for a while without asking for questions yes that is a feature of this elaborate constraint solver that you can say and you know said runs after has runs sometime after right so since since there's that much freedom you can say well I need to be sometime after these three phases I couldn't find anyone who knows why you would ever want to do that yeah any other questions about this stuff before I continue [Music] there's something of it [Music] since you're asking now let's take a brief detour and and talk about dotty because we're actually I'm going somewhat deep now into details of the Scala to API it's it's a good time yeah let's let's talk about dotty a little bit because I have more details to show on the the style to plug in the API but you've already seen the basic idea you know you you had a compiler face so what about Scala 3 Scala 3 also has a compiler plating API and if you're curious about it I you know I'm not I don't work on dotty so I can't answer like advanced questions about it but I have looked at these two pull requests one of which actually adds the API so that landed about six months ago on dotty master and then there's a separate PR that adds documentation for their API it's clear from reading the docs that this is very similar so you you know you're adding a phase and you can and you specify when it runs and then you can do anything one difference that I'm aware of is that the dotty has this concept of research plugins which Martin is interested in discouraging people from from essentially forking the language by having plugins that run early and change the syntax around or change the semantics so there's there's like a there's a check in the API that will like refuse to run if it thinks that you're in a release build of dotty instead of a development build of dotty now obviously the stuff is all open source so you know this there's really nothing to really stop you from doing this anyway if you want but at least that that sort of extra check is there at least it as a way of communicating to people writing these things like you know hey if you're if you're in this part of the compiler you should think twice so and yet you asked about about micro phases so one of the architectural differences between Scala - and Stahl 3 is that the Scala 3 compiler dotty has lots more phases so in I don't remember the number but like in Scala 2 it's about 25 and they have a ton more than that so in order to make that work they say if you did this if you did this in like a naive way you would make your compiler very slow right you'd be doing sort of you now I'm doing 100 passes over the trees instead of 25 that would be really slow so you can many of those phases are not full phases but are these little micro phases that are basically you express a transformation that you want to do on certain kinds of trees so the micro phases are written to a much more a much more targeted API and that that makes it possible to run all of the micro phases together so it's like the dotty compiler it's walking the it's walking the tree of your code and when it gets to each node in the tree can say oh I see that these nine micro phases apply here ok so that that was I was sort of explaining your question to the to the group but I don't actually have the answer I don't recall seeing anything about micro phases in that documentation so I would expect that adding a new micro phase ought to be trivial and if I didn't see it if I don't remember seeing it there it's probably because it's easy and I mean and that's what this was that's what this whole micro face thing was designed for me Faison you said I would hope and expect so as well it was clear it was clearly designed to be better for this so yeah Oh any other any other Scala three questions while we're there before I pop the stack [Music] generation systems those stages [Music] Oh [Music] it's a separate project community [Music] sometimes [Music] yeah so this is a complicated question about a complicated situation we get what what is the what is the future of macros in Scala 3 and how does that affect your decisions around about around what to use the situation on this is still evolving like even like still evolving as as in even just last week you know Martin announced I worked all summer on this thing called transparent methods that was going to be a replacement for many kinds of macros and meta programming and then just like week he's like well we decided not to do that so yeah the situation is still evolving it's confusing to everyone I think that the most you know and and this this has people worried right I think that the most reassuring thing that I can say about this is that there have been as the the dotty team you know tries to do a design for how meta programming should work in Scala 3 it's all based on having surveyed users and macro authors and finding out what's out there so you know they've looked themselves and there have also been multiple calls for macro authors library authors and users to weigh in and say well you know that I'm using this macro this macro this macro so the people on the dotty team have a really clear and solid idea of what people are using macros for in Scala too and they it's clear from everything that they've said that they intend to support most of it right so the area where its most unclear what will is white-box macros where which are macros that are allowed to okay with black box mat with black box macros the requirement is that your code type checks before the macro alters it white box macros are free to alter the the code and then say this all needs to be type checked again because it's different now it remains genuinely unclear how much of that will be will still be possible in whatever meta programming facilities Scala 3 ends up with the good news is that most of the macros that that people use are either black box macros or are easy to support by other methods like shapeless for example has a macro in it that the need for that macro is it's going to disappear entirely install 3 the dotty team has just hired miles part-time to work with them on type level programming and made sure that that whole that whole area of meta programming is covered by but by whatever they do let's see I've gotten a bit lost in my in my own answer here it takes a while [Music] I see yeah so III have a definite opinion on this which is you should you should not be afraid of macros in Scala to you you will not make your life easier by choosing compiler plugins instead and what why do I say that first off what what so when you're running a macro there's a sort of surface there's the the okay you're writing a compiler plugging in you're writing a macro those are two different api's but once and I'm you know I'm showing the the compiler plug-in API but once you get past this sort of first level API of like here's how I write a plug-in here's how I write a component here here's how I write a phase once you're inside the phase you're you're just dealing directly with compiler internals and that's equally true whether you're writing a macro or compiler plug-in so for any non-trivial macro or for any non-trivial compiler plug-in almost all of the work is in that guts part and so if you had to change your mind and and and change a macro into a compiler plug-in or or vice versa it's it's just not that different the there is a hope in Scala 3 to sort of lift meta most meta programming to a higher level where it's not just compiler hacking with us sort of very thin API around it right so there really the the reason that it's taking so long to decide on what the best meta programming story is for Scala 3 is that they're working on really hard on on doing a good higher level design that keeps you away from compiler guts much more of of the time you know so on the other hand if you're writing a macro or a compiler plug-in now in Scala - you really don't have the choice of waiting for that new mechanism to arrive so I would say just just just - just do your thing if you're if you're lucky when Scala 3 comes around your job will be much easier because if if these new higher level API suit your needs and if not and you have to rewrite your macro or rewrite your compiler plug-in under dotty the other I guess good news that I haven't shared yet is that the the dotty compiler internals and the Scala to compile internals they're not that different like if you read the documentation on internal so watch the talks you know all of your expertise almost all of your expertise on Scala - internals is is going to help you enormous ly once you're working install of 3 it's like the concepts are the same you've got trees you've got types you've got symbols it's just it's it's not radically different so that that's a that's a big that's a really a big top and I'm really glad you asked that because it's like it in a lot of ways it's like the question about what is the the future of that stuff so I hope I you know answered well without going on for an hour about it which which someone from the dottie team I'm sure could easily do how're we doing on time it's almost 8 o'clock so yeah let me wrap up let me wrap up the walkthrough here because there's almost nothing left to show once you're in your compiler phase once you're inside the compiler phase light so as I was just saying at this point you are now talking directly to compiler internals and it's really the the code is going to look the same regardless of whether you're writing a macro a compiler plug-in or you're making a pull request to the to the Scala compiler the main method that you implement in your phase is just run with no arguments and you have added for for a trivial plugin like this the only so here I'm calling out to count lines which is the helper code that actually that actually does the work sort of the only the what's interesting here in the apply method I'm getting compilation units one at a time so these are generally source files but it's a compilation unit there's a slightly more abstract concept because the code could be coming from the repple for example so I wait until I have all of my until all of my compilation units have have been supplied before I actually run the only sort of tricky thing I hit here is that oh and I'll mention it's just as I think it's sort of funny but also you might hit the same thing which is you need to know that that Scala doc is part of the compiler when you run Scala doc it's running the compiler and so your plug-in will run during Scala doc generation unless you take care to disable it so when I first did this I was getting line counts that were twice as high as I expected until I inserted that check ok so we got to the guts part of this but there's almost no guts in this one to show I'll briefly show one other example plug-in that does something a little more interesting so I need to go back to yeah okay so Jason zog who's one of my colleagues on the Scala compiler team has this sample a sample plug-in called boxer and it's it's a I don't think it was written to be useful it's it's another example although it certainly certainly could be turned into into something useful what it does is it tries to is this is like a linting plug-in that tells you if you're using value classes it finds that the places in your code where your value class is getting boxed so the whole purpose of value classes is to cut down on overhead of boxing so this is useful you know and unfortunately valued classes don't succeed at that 100% of the time this plug-in will tell you when when you're failing so the structure of the plug-in is very similar with a plug-in with a component inside it with a phase inside it and here we have the actual guts that are talking to compiler internals so we have a fourth level of nesting here inside our phase we have a we have a traverser extends traverser so so traverser is coming that this is like a convenience class that we imported from from NSC I'm actually surprised not to see the import I think that's where is the import coming from oh yeah so import global Jason sort of did a shortcut here and just like imported everything yeah so so traverser is coming from global and the so the nice thing about this is that it shows actually how easy it is to write a tree walker so this is something kind of kind of comparable to the micro phase stuff and dottie not efficiency wise but in terms of how easy it is to write the code so we're here in this example we're interested in boxing any time boxing happens on a value class the ast node that represents instantiation is the node called new so the traverser API runs this traverse method for every subtree of the entire program and then just lets you pattern match against it so when you so when you see the kind of node that you're interested in processing you catch that case and then you just you can just let everything else fall through so we look for those nodes and then we look at the class that's being instantiated and we check to see whether something is actually Ashley value class if it is we issue a warning you might be wondering how you might know how to write this and I have a few slides with advice on that what one thing that I enjoy pointing out about about about this code is it it uses it in what in this one place it uses this concept that exists in the in the compiler of time travel so even though your phase is running at a particular time and even though the compiler is constantly mutating trees as it runs it also remembers the states of trees at other phases so editing typer is like time travel saying well I'm running later but so that I need to go back in time and see how something looked earlier it's cool it may be a little scary but that if you see exiting or entering that's that it's it's that time travel thing that's happening any questions about this about the details of this code before I pop back out to more more general we're near the end here but a so but I do have a few more general slides [Music] yes everybody's mind [Music] the situation [Music] the the the two most most typical cases where a value class get boxed what one is when you're using it in a generic context right and then the other is is collections when when you're storing things in collections so and just as an aside about value types so that there's a active proposal for replacing value types with something called opaque types and the the you know reason that proposal exists is exactly because value types didn't achieve this goal of guaranteeing that something is never boxed opaque types will give you that guarantee and I think the hope is that opaque types will they're they're already implemented and dotty I think that's right and they're expected to land install it too and in Scala 2:14 yes I don't think there's a mechanism for that so yeah you know that that's a that's a good question and a funny question I guess that's a sort of sub question of the ones above the one about compatibility I don't know I don't know if it blows up or if you just get funny behavior before so that that is I'll just revisit an earlier slide so that happens with AD compiler plugin so the I add the the jar that contains the compiler plug-in as a dependency in the jar it knows the name and so the the compiler will look at that XML file and in the jar in order to see what the name is because it's you know you might expect it will probably be the same as the artifact name but there's no that there's no guarantee there the name in the XML file is actually what determines how the options are passed so yeah you know normally it'll be the same name in both both place but it's this one that actually matters in terms of using the plug-in and that's coming from the XML file okay I already mentioned that when you publish your plugin you should publish it by scala minor version I've seen some plug-in authors just sort of take their chances on this and it's sort of usually it works but there is there is a possibility of breakage and I have seen breakage happen I think in the interest of time I won't talk about testing except to just say that there's a few slides on it that I invite you to that I invite you to to look at and for debugging if print line debugging fails I'd I want to make this recommendation because I I think it's some something that people don't necessarily know or don't think to try the intelligent bugger is has this marvelous feature where you can use the intelligent debugger even if you're not editing or building your code with intelligence so you can intent you can attach the IntelliJ debugger to any JVM that was started with some special flags to enable debugger access so this is that this is you know often how we debug the compiler is we just just run the compiler the usual way with these special flags and attach the debugger it's you can step through that works for stepping through plug in code just as well as it works for stepping through the compiler code it's a trick that's that's worth learning okay so compiler internals of course there's no time for an introduction on that so you know yes this is one of those where I'm not going to tell you how to do the hard part but what I can tell you is is I can refer you to learning resources on how to do the hard part and you know ideally that this would be a page online but actually you know unfortunately these learning materials on compiler internals are just scattered in different places so I've collected the most important links and a few pieces of advice here for learning the phases print out the list that you get from X show phases and study it the probably the single most powerful you know ratio of like learning power to size tip I can give you is run some sample programs through the compiler with this phrase with this flag X print all so this will show you it will pretty print the code that you pass in after every face and it'll skip any phase where the code didn't change which is handy so that that isolates just the things that are changing so you will see your your whatever sample program you put in gradually transform from full-blown Scala to something that looks very much like Java 1.1 and is almost at the at the bytecode stage so just just I mean it's it's educational and also also just cool III think that you have this this easy window and into what the compiler is actually doing in order to get a deeper understanding the document that everyone doing any kind of compiler hacking whether it's compiler plugins macros anything everyone should read symbols trees and types it actually applies to stall a reflection as well so it's like this is really the Bible for this this whole space there's a document on the Scala site called the Scala hacker guide that walks you through the entire process of making a modification to the Scala compiler so that's a little more focused on the a little more focused on compiler hacking itself in terms of like you know here here's how the build works here's how to make your change but there's a lot of material in that that's also relevant to plug in authors and macro authors I have two videos to recommend one is by Jason Zod who also wrote that Boxer plugin this is a pretty long video of more than an hour but it's it's super I mean if you have any sort of affinity for compiler stuff at all this is like super entertaining and fascinating talk he takes it's not a lecture at all so it's not you know it's it's not like going through concepts on the whole talk takes place in the Scala repple there's a feature in the Scala repple called power mode which I think was added by Paul Phillips some years ago that enables all sorts of extra compiler hacking goodies so what he what he shows you how to do here is how to interactively explore compiler internals and actually by playing with them by actually trying by actually trying things in the repple super cool overlapping with this somewhat and also very good quality is talked by Chris Burchell called a deep dive into Scala see the stuff that you'll learn from these learning resources will help you with all of these different kinds of programming besides just compiler plugins one more talk to recommend a talk by me I'm called urs Scala contributor if you decide to you know use your newfound compiler hacking knowledge to work on Scala C itself or also if you're interested in helping out with the Scala standard library Scala modules the Scala website I this talk is about how to become a contributor to any of those areas of Scala there's a video online of one idea this talk in New York earlier this year I'll also be giving it one more time here in San Francisco in November at scale by the bay we already talked about Scala 3 so we're good there and that's it thank you [Applause] [Music]