SBTB 2015, SF Scala @Nitro: Paul Kinsky, Generating Immutable Case Classes from Avro Schemas
name's Paul Kinski software engineer at night try work in the platform team so while earlier earlier this year we were trying to decide in a serialization format the options were protobuf thrift and Avro right so they'd each have different strengths and weaknesses protobuf we mostly right so let's do this proper all right Paul Kinski computer say tada at worcester polytech over on the other coast moved to SF from boston just done with winter platform engineer at nitro right so I'm favreau so we were deciding between Avro thrift and protobuf earlier this year different pluses and minuses fridge ended up getting to the decision between protobuf and Avro an Avro is great support among data science tools all there's a lot of ecosystem there's park great support for park a sparks equal tools in Kafka Camus chemists Camus but the problem was at this time there Sarah jumping around a bit and like all these different serialization formats they're all fast they're all reasonably efficient they all pack your data down to binary they all have simple types right they all have enumerations boolean strings and salons floats etc they all have complex types the few different wrinkles so Avro specifically has maps of strings to type T a razor type T in unions which contain some like bundle of types right it's sort of like in any type you're scoping it down two types with ABC etc so Haskell has this as just like a built-in but if syntax it's really nice to use Scala sort of has this you have to use libraries like shapeless but they're really great there's some you don't really need them most of the time but when you do there's really no substitute for having an actual union type and these are all these protobuf thrift Avro etc they all support schema evolution by optional default fields right so that's the good part so here's the problem the reason that we were trike on the edge between protobuf an afro so protobuf doesn't have this great big data community support but what it does have is some nice type safe code generation so avro it's only Java code generation you don't get something that's immutable you essentially get like a bag of objects which you can put and get via string keys and you don't even have type safety for union fields right it's just an object like everything most other things so we decided to do about that was write our own code generation tool its generate Scala case classes it uses shapeless co-products to represent union types you have you can see there you have this sort of like an H list you're essentially creating at a type level a list of types and enforcing that you only have one of those types instead of just using any that's really handy when you need it when you don't you can just ignore the type level VODO but when you need it there's really no substitute so this is all based on scala PB it's a proto buffed cogeneration tool it's actually one of the cool things about open source right if there's something that works really well that you like the implementation of that you like the design of but it's not quite it doesn't do quite what you want it to you can actually use that as a starting point for it you want to write it so that's what I did right i took scala PB i tore out some of the base to the base infrastructure it's an open source project put out by true record and we retooled it to work with Avro and to generate different different classes and such so this is what a Avro schema looks like it's JSON here you have the one that's highlighted that's a line so you can see it has its a namespace that's just a package it has a type that's record that's essentially a class named line and some fields which is just an array with objects so there you have a simple field start and end they're both points a more complex one you can see name it has a list of allowable types right that can be null or string that's how you represent an option in Avro schemas then you have some other types here polyline in there you have an array of points instead of just two of them circle you have a radius etc and then at the bottom you have draw a request this is a bit more interesting it has just metadata field nothing important there but what is important is geometry right this type is a list of or sorry it's type is an array of Union where it can be a circle it can be aligned it could be a points so you want to be able to draw a bunch of different things and you want to be able to represent like in a very type safe way that this request can only have a point align a polyline or a circle so here's how here's an actual Scala code here's how we deal with this stuff right so we're generating point line polyline circle etc here you can see we're creating a type alias for the co-products of point line polyline circle and that there's a bit of a shapeless voodoo there that you don't really need to use for most of the code gin but again we need when you need a union type is really handy to have so yeah so it's basically just scholar right you have case classes they have named parameters you can see here actually it works with reserved keywords i just added that to show that so if you have type you can actually it'll correctly generate code for that it won't just blow up maps vectors subset or all that like just nice Scala stuff so here's what the actual generated code looks like and what's interesting here is I copy the approach from Scala ppb of actually generating code and not using macros because macros are sort of hard to work with especially if you want to create companion methods or just companion objects methods on them etc so you can see here it's just a case class extend some boilerplate generated message message you have some like with name with start with end convenience functions just for creating a copy with a different name start or end those are automatically generated from the different fields on the line type and what's really interesting down here you have an arbitrary instance who's familiar with Scala check show events so the idea is you with a scholar check is that it provides the machinery for generating random instances of types and then creating tests that attempt to prove a property over all randomly generated instances of those types as opposed to testing with a single value so you get from your generated code here including or on top of like the case class on top of the helper methods is Jen instance so what we can do with that is actually just generate random lines and I'm going to drop out of PowerPoint for a second to show you that it's not a console or yes right so let's just so that's the arbitrary instance online we can just sample it to grab some arbitrary lines and you can see some of these have named some of these don't have names that's an optional field they have points points are also generated from the schema those have some of those have names some don't and then you just have like random nonsense strings and like huge numbers as they're like x and y coordinates yeah so if you use these for your tests you'll hit edge cases right you'll get like zero you'll get negative numbers you'll get one and then you'll get just like random large numbers random strings maybe you'll get like some characters you didn't expect it's it's just a really nice way of structuring your tests it helps show where you have just basically weaknesses in your code base so we can do this for a more complicated type right so let's do that for draw request that has a union type that has some arrays so we're generating here again a random draw request it has a randomly generated metadata field randomly generated array of Union types it's all it all just works so that you can use this in your test code and that just saves a lot of time having that automatically generated right so a few more of these are just to show that like there is just a ton of stuff that you have to do to generate code right so I'm showing this to highlight one of the coolest things I took from a scala pv or used from scala PB which is this functional printer so that lets you just in a very type safe way in a very without using mutability with allowing you just use like single expressions for all of your functions to generate code or to generate strings rather it said sorry right so let's see how that actually works in practice so here you can see this is the master class for a printed Eric you can't seeks it's still quite small right so this is the master class for printing records so you just grab a start with a printer you add your final case class indent indent then you callin that takes the printer and add some sequence of fields you out then add your list of what you're extending etc you go down you add some utility methods and this is all just one expression right there's no string buffer here we're not appending to anything we're not like going and mutating anything it's really just very clean and functional and so again I want to like call out like true record and the people who put together scala PP that's where i grabbed the pattern from right so that's what the actual generator code looks like let's take a look at what the code that we would use to add this to our project looks like it's quite simple first you just go to the plugins right in project you add well this is currently not published but you add Avro cojen compiler that's the class that as part of a compilation creates these like classes in your source managed folder of your inside target let's check it out right here right so this is this is for some kind of enormous schema that we use to represent PDFs then in builded SBT you just add the runtime component and this is the companion objects it's the traits it's a little other machinery like for example Kafka encoders and decoders so what's cool about this is a since you have the companion object for every generated class the companion has two mutable from mutable methods for going from this immutable representation to to your Avro class that all the existing Avro infrastructure knows how to work with knows how to write to disk knows how to serialize and deserialize so you can just implicitly pull in the companion object just given a type and use that to do your serialization and deserialization it makes it really easy to create for example Kafka consumers and producers that work with like typed Kafka topics so that'll be a part of this one it's open sourced as well back to PowerPoint back so yeah it's you don't you don't actually have to deal with this yourself right there's a lot of boilerplate that you need to go back and forth here it just it's just really complicated it's a lot of writing you'd have to do your fingers would get tired you get carpal tunnel etc so it's much easier just to generate this so this is used internally at nitro we've been using it for a few months it's been like users across several projects we open sourcing it in the next several weeks and yeah so any questions right so two questions plus the general case bottles they play well with civilization like choir chalice organization so no question you mentioned Swift the what made you job from the evaluation um all right first question how well do these generated case classes play with existing serialization and deserialization frameworks so the trick here is that they kind of don't or that's that's actually incredibly radley phrase let me continue to explain all right Wow okay so let's there's a type in Avro that you have called let's see where is that a generic record that's how Avro all the machinery around Avro all the stuff door integrates with parquet that's how it serializes and DC realises Avro schemas or instances of Avro schemas so what you do is you transform your case class into a generic record and then you serialize that generic record the rooms right yeah so you have a spot like we're doing a shovel one which is serializable space classes into fights which is its supported by bicycle it is yes but you wouldn't get as compact representation and if you wanted things like Union fields enums I don't know if that would if the serialization machinery accepts to that this is my people having cycling or weird objects it's nice everything that use here at Street bizzle what the other fields as here I wish I haven't really worked with cryo before but I don't know if it's necessarily as compact as something like Avro oh you don't have to do it with this either and also it has reflection I don't like reflection it's personal opinion but I honestly didn't really look into cryo I hadn't really hadn't experienced that before but a second question right what made us drop drift from consideration so it's more that we chose Avro because it has such a good data support among data science tools like our data scientists really just wanted to use that and it has this great support there's an existing ecosystem folks there's a lot of existing support from Kafka from spark that you don't really get with a thrift or protobuf I don't know I mean theoretically were hypothetically we could have also built out like the same tooling in thrift or in protobuf but it's ready there it's easier just to work with that so that adds that answer your question sorry no essentially can we drop off generator because you know if I'm is parking so I don't want to see realize all of these classes in jars to the clusters right so it's very cool for testing and so on of course right I completely agree but you know right time prediction I want to get rid of that Oh totally uh well all right to repeat the question first so one of the things with this is that scholar check is comes a run time dependency not just a test dependency because you have these Jen instances on the companion object that is something like adds a bit of heaviness you'd might have to have those jars if you're on spark that is something that I've been wanting to take care of to somehow move that from the companion object to something that would only be required in tests if anyone has any ideas how to do that I'd love to speak with you after and sir submit a pull request when you open source it but yes that's a very good point that's something we do want to do before this is open sourced or after adds any other one else have any questions