Scale By The Bay 2020: Shameera Rathnayaka, Materialize Typeclasses with Magnolia

Scale By The Bay 2020: Shameera Rathnayaka, Materialize Typeclasses with Magnolia

Recording: Scale By The Bay 2020: Shameera Rathnayaka, Materialize Typeclasses with Magnolia

[Music] hi hi yo uh welcome to the uh type classic magnolia session um so uh thanks my point receivers uh introduce me and who am i i'm shawn with nike and scene engineer [Music] um so today's agenda is um uh we gonna talk about like what is uh type processes and what is that why we need it and and then we are going to define designer type class and then there's a tiny library we are going to implement called jsonify uh then we talk about magnolia how we use it and then talk about the converters that we are using in spotify and then some benchmark results yeah the type classes so what's it what is it in wikipedia it says like a type class is a type system construct that supports add-on polymorphism the most important like it's a way that you can program a technique that lets you add new behaviors to the clause types without using inheritance and without having access to the the original source like if you have some files in some libraries then you don't have access to those also you can actually without in without using inheritance you can actually add some new behaviors so that's uh from the advanced scholarly uh cats uh so uh i don't polymorph just right it's a kind of nice word uh so what is it it is like method overloading for functional programmers in like uh here we have a thread for show and has the uh uh method called show as definition and then it returns a string so the same like case class that we have here in person that actually um uh extend show and overwrite that show so a better example would be like if we have some um something to add up like we we can have like uh multiple uh multiple ways to like add two uh elements together uh in either we can add it as an integer either we can add it as strings so in that case we we can actually run like we can define this different at adding methods for the same class without touching the source code like on the original source code and they give this different behavior based on the type plastic so this is uh that that we call like android polymotion and in general polymorphism actually we like keep the behavior and the data in the same place but with the adult polymorphism it gives us a way to like we can decouple like we can keep the data uh separately and then we can keep the behavior separately out of that uh like in a separate class that's really like a clean uh clean way of doing that this is here is the example that we uh try to use it like the people i use person class so here person uh class is like just a case class doesn't have any show method but we have a uh show type class which explains like it's given like a behavior the show behavior to the person uh type case class but the logic car is in the separate place it's not inside the person okay designing a type class so designing a type class there's a three different design uh uh roles here like when we are defining designing a type class uh one is define po abstract types it will at least one generic parameter like we need to have at least one generate parameter like type t and sure bracket t is like t is a direct parameter we need at least one we can have multiple and then we need to implement this uh abstract type for the uh to create an instances that we we need to support for example we need to support the person like type to support this show type class then we implement type uh show a person uh that's implementation we can actually so there if there is no any implementation then they we can't use that like behavior on that top of that uh the type so that's we can limit the scope for the only supported type and then we need to create an operation like we we need to create a method that which take this type class and then the object type and then uh call this behave like behavior uh on top of that uh type uh so this is like so how we define like this design this is this color is like we can use those color built-in features like traits uh case classes and implicit and then we can uh achieve this advocation like define design this uh type classes according to these three rules uh the the the the slides we have a show and the semi group it is kind of combining two uh types and uh producer one uh the same type so we can actually um uh use this like trick and then like this is the right where the the the type class is the one direct type parameter which is pretty uh with this both the show and the semi group so um we can use uh like i said like trade classes like and uh to you know design this type class let's talk about uh the json file so jessica is a tiny library that we want to implement which convert the scala class passes through with json representation so in uh we so we have this uh the the the pre-given data like we have a adt uh json adt uh so we have a couple of case classes implemented like json spring the base to cover the basic uh json uh types properties uh so we have number uh the boolean cover and array and object so object especially like take name and then the map which is uh properties of that object all these types are json and then we have a type class so we have a json file type class with a type of t and then it has a method to json which take the value and then output this json type and then we have the json printer which is the like which has a print method uh which take the uh type t and then the context form jsonify and then uh it convert the uh type t to the like string which is a json representation so we have uh we have decoupled this logic like how how how we convert the t to the json and then json string so for that we have another private method uh called print json which take rotation object and then the return the string so once we know like how to convert this key to the json type and then we know like how to uh convert that to the like just the string representative representation so design the type class so we need to design type class for the json file so first thing is like we need to define so we we have the trade and a few abstract type with at least one parameter so we have only one parameter here and it it has a two json which take t and then return json type and then the implementation like uh you know jsonified object we have this uh this uh auxiliary uh constructor like spec function which make it easier for like uh uh have this multiple instrument reduce the code uh so we have string jsonify which take a string and then convert it to the json string and we have uh uh just notify uh integer so uh we which took the integer and then convert to the json number and and also the boolean so we can have whatever like we supporting types uh defined here uh so so we have the implementation so operate so uh as we see like json printer is our operator like it has a print method which take the type and the jsonify and then uh can do that or use that to like uh radio some uh behavior so short recap here so what values json printer can print uh it it only can print all the the values that the implementation the instances that we define in the the json if i object but so that's only limited to those types but what about the user defined classes like if i want to use this json file library and then i have my own classes so i need to just simplify my uh my types so how we can how i can do that will this work like i have my address and person case classes implementation and then i create a person and then i call the pin or this won't compile because it doesn't find the justinified person so either i have to define it here uh uh so oh i i need to get it from the j75 uh object which is not possible because that's a library so it doesn't have user defined types in implementation so we had to define it by ourselves right we defined the jsonify address and person because person has an address as a one of parameter now we can call the jsonprinter.print with the person and it has all these uh contextual implicits uh so can't we simplify this law so it's a really cumbersome to like define uh jsonify for all the custom type like if you have a bunch of custom like like types like you would use the case classes and you had to provide the implementation of jessica all of these types so how about we derive like json five for any user define types that's that sounds awesome uh so how we can do it there's a multiple let's do it first is we can use low level macros we uh manipulate the hd tree and we in the compile time we get the thai tea and then we like generate some uh classes so generate some classes and also just if i construct so it will be available to the then we have shapeless which is where like another uh like big library uh which also can use to uh accomplish this uh and also magnolia which is uh not broader like it's it's just focus on like how to derive these uh uh type classes the shareplaces it's a the terrain type plus is one new skill like you cover a lot of the other use cases so background is a really really simple uh library that uh uh specific to the driving the type classes let's talk about a bit about magnolia so the the magnolia the professor's uh the the documentation is like magnolia is a the generic macro which automatically derives some meteorites some type classes for a product and the core product types so uh here we have the product and product types so the country is the core product uh which is a trade have a multiple uh case objects and then the address is a product type it has some parameters and one of these is so if it is support these both cases then actually i can use it to uh generally like derive my which is in pi for address and same for the person as well so and it's supposed because you defined data type so uh so it's actually like uh when you have uh type the json if i drive some justify type t and then you encounter the rounder chain like again same type and it doesn't like uh derive again like it use the length it's kind of user caching mechanism like it's you keep track of like what what types of drive and then you use it and then uh if the duration fails the error message or detail and information that's really awesome but like if you're trying trying this new library and then it's everything is like in macro it's really hard to like dig deep in magnolia which is this is like a wrapper for like high-level wrap-up and underneath that like use a lot of macro like low-level macro uh so it's really hard like like uh debug if it doesn't support like use like knives or where it fails and stuff like that so it's good at the rarefield like really easy like it's say like which type you can't like it couldn't like grind so that's really uh awesome so you can find the manual and the proper the github professional slash magnolia it means open source so you can uh use it uh okay so it magnolia like let's dig deep like how we can actually use magnolia to like derive our jsonifi object so the basic construct here is the structure of this uh the derivation is like this object like this is this this coordinates snippet so first we bought magnolia and then we enabled the macros because magnolia we're working with the mattress and we are going to use macros as well so the migration object has three types three things one is the height constructor and then the combine method and then the dispatch and then we have implicit general method which is actually called the macro uh so we need to have this all uh type constructor combine and dispatch method in the same place because um on the compile time manual i need to find these things so you should like uh put them in the same place an object and so let's like let's use magnolia for our json file so type constructor so what our type construct is json in five t so we define it and then we have a combine so the first block uh is like a person block percent case class and then so we need to derive the teaching five for the person to to do that we need to implement the combined method in the magnolia uh derivations uh so here uh if you see like for a given type t i get a like case class object from the magnolia and then we i need to uh return a type class uh t which is the uh uh the type we call it our type construct which is just in 5p uh so i i have i'm creating a new jsonify and already in the 2.json and get the value t and then i i put in json type so in in the the important thing is like how we do what what we are doing in this two generation body so first we access the parameters of the passing uh case class and then for each case class and for each parameter we create a map because the the what we try to do is like we try to convert this the case class to adjacent object which take the case class name and then the properties the parameters property property map so we're constructing that with the the the for each parameter we get the name label is the name of the parameter and then the v each parameter the the as you can see in the the third block we have the like structure of the case class the the magnolia and nolia case class which uh uh if i uh walk through like this example like when we are getting a case class in the second block in the combine what you are having is like the case class with the type class d which is we defined type construct json if id and then the type is the percent type so and has some properties like type name which is the uh you have the package name and then the type name uh as a type term and then the sequence of parameters uh and uh these are the like parameters that uh you need to like to construct the case class so the for uh so i we accessing that parameter list and then we connect that to uh uh the name parameter name and the json authentication type for each parameter and in the parameter trait like if you see the parameter itself has like a type class for each type param like parameter type for example for the first type i have a person is a name if the string is a parameter so we will have just if i uh and then the type is spring in the fourth block i'm referring here so the type p type would be the string and then the p label would be like name but that's the name of the parameter and then type class is the the type class of that the jsonify uh string and then the difference is like when you pass a instance of the case class uh and then it returned the the value of that parameter so we we are uh yeah we are using the that in our third uh uh second block in the the combined method uh you can see here so uh we are using uh uh the case class parameter and then we are labeling type class and so and construct the property and then build and you can see like for the address we get the address and then the show method in the the fourth fourth block and and then we get that an address itself like we define like the same thing same pattern same same thing going through like uh then magnolia derived the type class for the address uh case class as well cool and derive uh type class for product uh so i get it like it's i'd be i i uh take it to the different slide like that that's only dispatch method like it has a lot of code so uh dispatch is like uh handle like where the magnolia handles the co-products so we have the country uh still trade and it has different implementation u.s and sweden so uh we only support those two uh uh countries so we can uh implement uh so just if i for the country so um we are limiting that no one can actually uh implement any json fight for different countries this is like we are scoping the countries to us and sweden and we are here dispatch uh what is what we getting like same trade like it's a sale trade that's kind of represent the country and and you can see that uh in the third block like the seal trade is like a take two type parameters uh type type class and uh with the types and then then the actual type if you check the like the the interesting point here as like uh also is like two json body so how we derive the json uh json uh type from uh the what we have with the scene grid so here we have a secret or dispatch value uh we have to cross the value that object that we are working on and then it return of the sub type object so if the subtype we will like so we pass the country they need to return like which country are subtypes we have uh right now and then uh and we cast that to that type then we cast the value to that type and then we get the two string so so this is simply like how we uh convert the same trade uh object case case object to like this json string so interesting to know like how this the data is looking like a sale sale trade uh the type name would be like the same one like same thing uh we have the object the package object and in the country uh the type and then the each dispatch method you can see we passed the actual type uh and then we have a different return types uh so we need to pass this handle and the second uh parameter is uh and it has a subtypes and data type so we are using this to like construct the uh the body of the json jesuit so so it's more work up here again in the jsonifier so now we have all these things like we have the type constructor defined and we have the combined define and then we have the dispatch defined and we put the uh implicit generated here and and we have some uh primitive implicits that we we are going to support like spring uh boolean uh uh and those stuff so that that's our primitive implicit we have to provide those things like uh that's like mongolia is like uh only working like case class and trade level so we have to provide other uh implicit like for the premiums and and we have an extraordinary class simplify that okay yeah it's time to draw some uh tech classes so it's really easy like so we have a test using e5 method which uh sma method and it import the json api so like we this is how we bring the uh the derived classes like to make those variations within scope uh and then we we construct the person object and we we then we just call the jsonprint.print for p and this will work like this will output the at the json uh uh representation of the like string represent of the json uh and uh it found out with the pretty print work so we don't have we have we don't have implementation here but you can assume that that's between our json so basically this json is like kind of a string uh the the output of the json print right or the print p and it find the json file because json if i is derived from magnolia so we we don't need to uh define that we don't need to get it that we don't need to define person or either address so and also for the you the country so everything is derived and you i'm a pretty happy uh use of this tiny library like we do i don't have to define any any of my uh uh just if i have any of my custom types i don't automatically drive that's really awesome so uh when when we are using magnolia there are some things we use we should notice like maybe we should uh uh like know uh that the magnitude support more than one type of you know like for example if you have kind of encoder which you take the type t and then convert it to the encoding to the type r and then we have with that encode and it doesn't doesn't support like natively it doesn't support but we have uh some work around likes uh for what we can do is like we construct the type type class for encode and for the type for all the type t and we need to define uh uh the r type for all the json types we have an encoder so this is like we are uh converting the multiple type parameter a type class to like one type of parameter one type parameter class so we have to define this encode like magnolia generation for encoding a different for the different types one for json and one for something like you want to open and another thing is like implementation heavily depends on the type class variables so uh so first one the json file is this is like the controller controversial type class so it uh it taken type t and then output the string it's a fixed it's present uh depending on the the tube and then the covariant types is like you output the t so you have to construct that t inside that method so you have to uh uh really get family with the magnolia uh apis the clk uh the case class or seal trades so you can construct pass the the uh required uh parameters to uh uh to the the constructor method to like construct your type it's pretty uh it's it's a bit harder than uh the contravarian thing like where you get the type and you produce something uh so but it's doable so the interesting like where we use magnolia in spotify uh so we use uh in shio which is a idiomatic scala wrapper for patchy b uh so we use that for code iteration so if we uh you know like if you have family with that budget then you have to define code for or each input and output types so uh it's a really cumbersome like uh if you want to direct write the quota for every custom type that you are uh using so we you provided us like if you want to provide us that's fine so if there's no holder fine uh finding the scope then we will derive it for you and then we use for testing so generate materialize some type classes um and we then we have the mag spotify magnolia file which is a add-on a nice add-on collection of add-ons to common type plus iterations uh if we are using scala checks for testing and we have property-based testing deterministic generator like we generate the deterministic uh genes uh for scala checks using the uh magnolia and arbitrary types derivations and then there are we the magnolife has all that cats type classes derivations and uh some some data converters that we heavily uh we we we are using uh in the spotify uh such as we courier and big table um let's we talk about the converters it's really interesting so uh so deriving case classes for common serialization format like so we when we have a converter we have the type t and then we have a red and right so this is like we are con we we we working with a multiple type parameters and and this is we have converters for all these the data types the korea pro big table data store and tensorflow and all stuff let's talk about like how we use that in every time so ever like it's a work with the generic records it's basically like you have a put and gate methods uh so you put some key value value is object types and then you get a object using that key it's in string key and then uh so generic record has a default implementation call uh so which has the schema and then the list of objects and uh the an associative schema is look like this like enough types recording number a map union and then the field which has string schema docs and default values and so we can uh we can use this uh cell tray to represent a primitive field so we are using it like arrow field which is uh we have two types uh from three and two t that's depending on the like implementation and uh we have schema we have a bunch of methods that we need to use uh to work with uh uh field uh which is a type class um now we have uh enter generate report right so so we have for the type class for each fail a row field and we have a a type class for the entire general generate record avro type uh which is a converter with the type t and it cannot read from uh uh generic record and it write to the uh generate report it's like read generate record construct t and then write uh uh convert that to the uh uh generator record and write it so that's that's that's it like that's why we have these two uh uh uh constructor auxiliary constructors here uh get the generic record uh originally and then that get the tea and get become the generic record um yeah so and sometimes like for the overall the the the types that we read and write are not exactly the same every time for for some it's the same like for integer it it is the same like we the read values are like the the what why overall types are integers it can we can map it to the one to one like in teacher in the scala and uh but not for all the cases like for the string like in avro uh well uh it the string is kind of a char sequence so char sequence and when we are reading we need to read from chart sequence and then construct the string type and then we're writing we we work with the string so that's uh like this bit different like what the read and the right types so we have a uh implementation like observing implementation for that uh type and then we have the important like here we have like combining for ever type so we get the case class and we know we need to provide like uh you know to provide the error field like here we are like implementing the avro field so because arrow fail is used by the road types uh so here we have from key we define in the type types and then we implemented this using the case class that we are receiving so uh so in the from method you can see we use the case class construct to construct the type t and then the two method we use parameters to actually construct the generic record uh from the chip so this is like the cover controlling and the covering both both example in one place you can see like we use different uh apis to like work with them and we don't support uh uh dispatch right now with the type class uh zero type um in future we can work with the union and and other the uh type types that need uh dispatch like uh co-product type support right now we don't support it uh future we might so let's see how our type uh in action like so if you are using uh so we have a type like a duration with the magnolia uh magnolife so we import that into the scope and then we create new tests like our case classes and then uh we create instances and and then we call the avro type then fu and it will derive the type for us for this foo and then we can use converter uh to convert from the type uh full to the generic record and then generic record back to type 2 so here we are get another generic record and we are accessing this schema and then the next next like uh last we uh get the generic record and we connect back to the full so we will get like uh one to one uh like machine like uh you know the same foo object pack from the direct record so it's round trip should be equal like for instance convert the generic record and generate code back to four it should be it should be equal uh so we have other over the converters uh but uh other than the the avro we have for the bigquery it converts from type t to pi table row and we have protobuf a converter and we have a tensorflow network in magnolife uh and we have data store converts and a bunch of more uh other connectors uh for guava we have and cats we have we talked about it uh so there are multiple uh magnolia derivations uh for all these uh types so uh let's talk about some benchmark we this is not officially the benchmark this is something that we ran in like uh uh in our computer local computer one of my colleagues ran this one and i i i got that benchmark results from him uh so the with the so we used to have shapeless uh in uh our uh the show and uh the libraries that we use before magnolia and so we replace that with the magnolia and then we get we see uh compile compartments the differences uh for arbitrary and the converters is really big like it's really fast in compile time uh compared to shapeless uh and also we did uh some converter uh nanosecond operations so it's also like my body is pretty fast uh compared to shape this implementation uh so uh it's pretty fast like because i think that's only reason it's like it doesn't derive the same recursive it doesn't derive the same type types again and again so it's and also some uh improvements uh yeah in the background so you can find references here uh magnolia professional blackberry that it's a really good documentation if you go there and you will see like other than a really good example like uh uh instruction like there's a tutorial so you can follow and then use magnolia and if you if you want to like use make notify it's available in spotify github repository uh and then we have a show uh and there's a talk done by neville and claire uh in uh northern india 2020 you can uh you know access that and thank you [Music] you