scale.bythebay.io: Jon Pretty, Magnolia: Generic Derivation 2.0
Recording: scale.bythebay.io: Jon Pretty, Magnolia: Generic Derivation 2.0
thank you and thanks everyone for coming it's good to see so many of you here I will be talking about a library I've been working recently called Magnolia that the talk is gonna be in three parts of which one part won't exist because it's normally a 90-minute talk and I'm squeezing it into twenty so I'm gonna start off by talking about very fast about generic derivation in Scala what what that is I'm going to introduce Magnolia which is a macro for generic derivation and I'm not going to tell you how it works so I try to summarize what generic derivation is it allows us to generate type class instances for types we don't know beforehand composed from type class instances we have type class instances that exist in implicit scope it's probably worth me asking a question to the audience how many of you here have done have used a library which provides generically derived type classes so I'd say more than half probably 60% how many of you have written your own derivations with I presume with shapeless for your own type classes okay so probably closer to 10% okay that's that's a good good indication for maybe how how hard it is so I'm going to use a very simple example a show type class throughout this I'll tell you what what this does if you I'm sure you all know a show type class will basically convert a type a specific type into a string you give it a type that gives you a string representing that type and we would use it by implicitly getting that instance so we'll get the instance for for example a show of int we would call show on it passing it the thing we want to show which is an integer and it will give us a string it's incredibly boring but it serves as a useful useful example for this talk now if we wanted to manually derive a type class for a case class say we would write something like this here's the show type class here are some type classes written actually with with Sam types but these are our primitive type class instances and we want to compose these into a type class instance of show or a person the way we might do that is we define the implicit we construct and you can all see the pointer we construct a new show of person I mean we implement our show method with this this string here which uses the string show instance it uses the int show instance which it gets from up here and it just calls show passing them the parameter components of that case class joins small together puts in the string prefixes it with the word person what is this is that has never happened before if it doesn't happen again I'm scared now okay let's let's concentrate so we have we have our new instance of show for a person likewise if we had a co-product type which in Scala that there are seal traits as a convenient representation of these what we essentially want to do is use instances of these type classes for the the components of that that sealed trait the components of the co-product and basically delegate to them so an implementation might look at an entity which is the suit the super-tight the seal trait based on its type whether it's a person or an organization we delegate the task of showing to either that that show person we created on the previous slide or some some other one we might create for an organization case class so it looks a little bit boilerplate the most interesting one was the was the product case but nothing nothing particularly complicated but this is a very simple ADT where we're doing this for and you can imagine that for many of these which may be deeply nested and recursive for complexity is greater and greater in the actual code which is very boilerplate e there for coding use right gets quite large so just gonna make a bit brighter for myself there we are so in ER in a world where we have generic derivation what we might do instead is import some sort of generic derivation implicit with with this line here and then we would just be able to request with no more information than we have above here just these primitive primitive cases we will be able to automatically derive an entity for show without any extra extra code in our source that's what we're aiming for and Magnolia is my attempt to to provide this so it is basically just a macro it's a larger macro than than the macros in in shapeless but it allows type class authors and and users to easily write fast and debuggable generic derivations for arbitrary type classes so I've italicized three words here easily fast and debuggable these are three key things I wanted to improve on the the state-of-the-art so bit of background it's it's about four hundred lines of code it's not a huge macro codes pretty dense but it's it's not enormous it's open-source license but under Apache 2 it supports a variety of different type class shapes and different structures we might want to derive for it it's it's early days it needs more real-world testing I've only done simple simple examples so far first I mean 0 6 1 was released this morning with a with a bug fix but there are people using it and the feedback has been exclusively positive so far I'm happy to say probably people just don't tell me if it doesn't work but I I would like to know if you try it and something doesn't work as expected the website is Magnolia dot work and there is there's also a quite a comprehensive tutorial if you have your own type class you want to derive for but I'm gonna give you just a quick overview of how it would work and what you can do so I've already already totaled you about sale traits and case classes it will also work for case objects which are a little bit like case classes with no parameters it works with value classes which are a little bit like case classes with a single parameter and two tuples which are like generic case classes with no with no prefix so the top case is the co-product case they look the the bottom for our products cases and it'll do all of these and you can actually disambiguate between them with with the api api if you want to but we observe that there's no standard way to combine arbitrary type class instances as show a number of shows corresponding to each parameter in a case class will combine in a very different way from a number of equal or each type classes we work differently with each one so we need to provide some specification to to the macro so it knows how to do the combinations two to construct the the type class instances Magnolias api aims to be flexible enough to deal with the vast majority of type class shapes and structures while while also not being too complicated and being easy to use and that interface I put interfacing in quotes because it's not an interface in the in the object-oriented sense it is a basically a list of methods that need to exist in some form on an object for Magnolias to be able to be able to do a derivation they are a type constructor for defining what what the type of the type class is and which parameter in that type class is generic which one we are narrow sizing with the macro a method which defines how we do the combinations how how we bring together the type classes from different parameters into a new type class instance a method for choosing which type class we delegate to in the case of sealed traits and a very boilerplate method for binding to or invoking the macro that's a one-liner in I think every case this is that this is the type class that the type constructor you need to write very simple it's called type class it will always take one parameter and it will be equal to the type class you want to generate if your type class has more than one parameter then you would have the other parameter in here but but fix to some value so this this enables you to specify effectively which parameter is going to be the generic one for show it's it's pretty trivial and then this is how we write a this is how we define the combinations and I'll spend a little bit longer on this slide don't worry if you don't understand the detail of what I explained the main takeaway point is this is a this is a sort of representative size of implementation for for a combination so you can see that we are we see the definition here it returns a new instance of the the type class parametrized on T which is completely abstract we don't know anything about it so any references to T have to be written in a way which assumes very little about them we do know that we are deriving for show you can see show appearing in the parameters here and we implement the show method the abstract method in the show type class and the way we're going to do it is we're going to take the parameters of the case class now the case class is passed by the macro as a parameter to this method to the combined method and it's sort of a lifted representation of a case class it gives us access to the parameters here are the parameters their pram typed on show and and the abstract type T and for each parameter we're going to create a string and with that stream we're going to join them all together with make string use the name of the type which is simply the case class object type name put some commas between them and the interesting bit is really this this line here now remember I said we know very little about the type T we know even less about the type of each parameter in there we don't know whether it's it's in real life or run time where there's gonna be a string or an int so we have to write without knowing that and sure we could get a label we can get the name of the parameter and that's always gonna be a string nothing complicated there but we also have access to the type class that is the show type class corresponding to that parameter that Magnolia will find for us we don't don't worry about how it finds it at this point but we assume that will either find one or it will be recursing and we'll generate it for us and with that type class we are calling show on it because we do know that it's a show we know that much we pass it a dereferenced T now T is unknown tears abstract but it's a case class we have a way of dereferencing it do you referencing the parameter inside that case class we have accessed that parameter the type here is is actually P dot type it is a type that is existentially existentially quantified on the type P now that's a complex type we don't want to we don't want that type hanging around for too long so what we do is we passed it to a type class of show which is also parameterised on the same type that complexity cancels itself they cancel each other out and we end up with a string so this is a string all is all is good and that will that will be types a if you can't just put any value in there in fact by parama tricity the only way of combining these is the one I've shown here and join them all together we get a string so that is our implementation for a show type class for combine obviously it'll be very different for for different different types you might be producing and what the macro will do is it will it will simply call that combined method passing it a case class instance I've sort of prefixes with the dollar to indicate that it's it magically generated the macro will do that it will pass it to the combine method and it will give us this this type class result so that's the code we write the code in that the price to code at the bottom the code the macro will generate in the case of seal traits very similar thing we we have a convenient method dispatch which will dispatch to a a subtype I won't spend too long on that but again look at the size of the code it's small it's reasonably simple I'll take questions the energy so we put all these into an object derivation object which I'm calling show derivation and we edition include an implicit method which I'm calling gen which which references the macro because this is defined on this object it knows where to find these methods so so is what I would call a contravariant type class the generic parameter T is an input to the show method and the same is true of something like eat these these two will have a similar structure in the implementation but we can also support cases where we are producing a result where we're producing an instance of the type T default will return a default value for a particular type or a decoder which is I've call these covariant type classes decoder you actually need to produce a new instance of a case class in in implementing the the abstract method in your type class so there are ways to do this with the API which I won't show you you can also get type classes which are parameterised on a higher kind of type such as functor we don't support this the API doesn't have a way of writing it I think it's possible but I haven't spent long enough trying so that is that is a feature for version 2 you might ask what happens when it fails if you've if you've used shapeless and tried to drive for type class you don't you just don't get a result back you literally don't know where to look used to do searches through the the structure so if we had a an ADT like this we try to get a show for entity there's lots of types in here if for example we didn't have a show of boolean we would get a stack trace that looks like this it points through the structure to telling us step-by-step all of the all of the types it couldn't find and the ultimate reason at at the leaf node why it couldn't and we then go back and fix this maybe we get another sack trait on on the second error we had and we'd go and fix that but there's not much work to be done in in any any complicated searches recursive data types these are supported these these work without any special magic on the user's part you can see that if you have a if you're trying to generate a generic type generically derive a type class for a tree tree is a co-product it has two branches to two parts leaf and branch and the branch has tree within it so if you were naively following an algorithm it could potentially just recurse indefinitely and never produce any any result we what shapeless does is it has a thing called lazy which deals of this with magnolia we have a greater knowledge at the point it's doing the derivation and we can identify when we would have been recursing and simply point back to the the point back recursively to the derivation for that particular type there is a story on implicit prioritization I think we all probably know that it's quite complicated the rules around it but you often have a situation where you want to you want certain implicit state priority over others and Magnolia has a story on this the jury is still out I think on whether it is the right story but it's I think maybe easier to reason about the Magnolia implicit works like any other implicit it gets called at the same priority it would as if it were a non non macro implicit that works hopefully as you would expect but any for any recursive call to Magnolia so when it's deriving a nested case class for example that will be beaten by the existence of any implicit that's in-scope evening companion objects even if they would have had a lower priority than the Magnolia definition that gives you a what I think is a sensible default and there are ways that are quite easy of overriding that whether it it handles all cases I don't know but I've got some ideas on how it could be tweaked just to mention we also support ta DTS and we also support recursive ta DTS which I think we were a problem for shapeless it it could do I can do two att's papers can do recursive types but the intersection of these two was was problematic now compiler performance I think this is this is something that a lot of people complain about with shapers and the reason is it generates huge amount of huge amounts of code when it when it has an implicit that calls an implicit that calls another implicit all of which are producing more and more code there is a large amount that you just don't see at the you site that that gets generated and this this is the reason why the code is so slow to to compile Magnolia does some deduplication it's aware when it's doing the same thing twice and will reuse reuse instances where possible it also caches implicit searches in the same position for the same type so it doesn't need to doesn't need to do that work repeatedly and this is this is the the results for one particular example the the three three lines up here this is a cold JVM and I repeatedly do the same compilation I think about 25 times they all improve as the JVM gets warmer the top is shapeless with kittens using deriving a show type class for some complex ADT I wrote 50 times in the same file and that the bottom lines are magnolia magnolia ends up around two seconds for this particular example for the full compilation shapeless with kittens sort of I think ended up around fifty seconds maybe a little bit faster so it's it's quite a bit faster but I mean a lot of that has d2 plication it depends a bit on how how many times you have the same type being derived in the same file my worst case scenario was four times faster but it's somewhat arbitrary how much how I define my adt is to what what benchmarks I want to get and it the different different lines are four different versions of the compiler type double scholar is actually quite a bit faster for both Magnolia and for shapeless it's this this bottom line here and the bottom line their runtime performance I I've not looked at this in much detail but you saw that we had instances of a lifted case glass lifted parameters these are all objects which creates get created on the heap this is garbage that is maybe maybe worrisome if you have a lot of parameters so I wanted to wanted to look at this I've not tested it thoroughly carefully or even accurately but I did I did try to construct a million show instances for a particular ADT and it was 52 times faster than shapeless which got me excited and then I actually tried to call show on that on that instance and it was only twice as fast so I think I'm aware there is garbage produced if I'm if I'm twice as fast and that's that's the best I can do it's it's an improvement it's maybe specific to show as well so it depends on a lot of factors but this is an area I'll be spending some more time on once we have version 1.0 released so I'm hoping that could be done in the next two weeks I'm not I'm not Precious about 1.0 being perfect if it's not then version 2 comes out a bit later and we'll we'll move from there so rot runtime performance is probably my my next priority after that and they're looking into supporting higher kind it type parameters and that is what I've got for the talk because I'm already over time but I'll be around outside later for anyone who wants to hear about details of the how it works behind the scenes that's a longer conversation but I'll I'll be available please come and talk to me and do I have time for a couple of quick questions now a very good question okay who's got a quick question you seem to do know so that's the long question I bet it was gonna be a long one no one think I guess nobody thinks their questions okay go on [Music] you could derive it for the it's one that way isn't it there so the implementation of this one would be almost identical in every contravariant case so the only thing that would really be this implementation that this is called to show here but it might take more than one parameter so freak you would need to cast the first type parameter and the second type parameter so that the first value parameter and the second parameter you would also need to check that if your dispatch disambiguates based on one value with the each implementation you need to check that the second value is of the same type if it's the same type then it's not equal so false return false so to quickly summarize in some cases you could come up with a sensible sensible implementation automatically I haven't provided it because I don't know how common those cases are and how hard it is really to write just those three lines and just have a consistent way of doing it for every single every single case okay thanks I'll be outside now [Applause]