Scale By The Bay 2018: Joe Kachmar, Production Haskell Demystified
Recording: Scale By The Bay 2018: Joe Kachmar, Production Haskell Demystified
you I'm trying to set up showing off how maybe a hypothetical sir someone might want to build would have some sort of notion of a user that could be either an administrator a moderator or sort of a basic user so in which case each of these members of this user type is represented by its own type a record showing an administrator that has a username a password and email address and all these sorts of things one of the things I kind of wanted to really emphasize that Haskell makes interesting and useful is over here we're saying that an administrator may be promoted by another administrator and so the administrator data type is actually a member of the administrator data type is an administrator itself so an admin may or may not contain an admin it's a recursive data type designed to represent the sort of a decision very easily for the user and so sort of you see a similar structure with a moderator a moderator is sort of a record of all of these different fields with the yeah so pretty similar only a moderator is definitely promoted by some kind of administrator and further down you see sort of this very basic thing a basic user is just someone with all this information associated with it username password email address etc and for all three of these components were doing something called deriving these different type class instances so we're saying GHC the compiler for each of these data types derive equality derive what they call as generic which is a way to generically program over these data structures and then derive a way to show it to the user and that is what's going to sort of power the rest of what we're going to do afterwards here we're telling GHC allow me to generically operate over these data structures without having to actually write concrete code that does things with them and so one of the most I think useful ways to get started with working with data and Haskell is actually seeing something concrete about it so we have these data types but like I don't have necessarily a user handy in the code somewhere without making one making you know some kind of admin with a username password email or whatever so what I can do is I can say GHC you know the shape of all the data in this data type gin something up for me like give me some random administrator type user here and to do that all I need to do is basically say an instance of the arbitrary type class for this basic user type where the arbitrary method is defined by this function generic arbitrary u which says use the arbitrary data generation functions that exist over the basic types that this record is comprised of and then create a uniform distribution of randomly generated data for it and for the very simple types that is literally all I need to do I can say my moderator and my basic user are very very simple and so just give me some random stuff done sweet and in the instance where we have something a little bit more complex specifically with the administrator case we can say well you've got this recursive data structure and if I give you something random maybe it just never terminates maybe I just get an admin of admin of admin of admin forever and so in that case the function that I'm using here actually lets me tweak that generic stuff a little bit so I can say mostly give me something for free but provide some kind of base case and under the hood this function generic arbitrary req is actually tracking the recursive steps in producing this data structure and decrementing I think some kind of counter internally such that when that counter gets low enough it says then shove this data structure in there where each of the fields and the record is still arbitrary but the field that could be recursive is the nil case and so I've now found a way to sort of guarantee that this data structure that could potentially go on forever is actually going to terminate and complete and we can actually even see this in action so if I look at the user type in the GHC I repple over here this is sort of user this is sort of admin and if I want to get something out of this I can write this function which is basically going to generate an arbitrary basic user and then pretty print it so it looks a little more legible to the screen and we get a really sort of garbage Unicode text randomly generated stuff for the name and password and everything else because we're just generating text and then we get random time stamps and a random boolean and if I keep executing this function I'll get new data out of it every time and so this is I think personally a really great way to just start working with a concrete example of data that you might want to have but you don't really have any good examples for it so like it allows me personally to model the problem that I want to solve just as a what is the shape of the thing I'm going to work with and then immediately start working over stuff and then get an idea of what it looks like and so I just think that's that's really neat in general I can modify the type here and get like a moderator and you can see here that it's actually generating the administrator that promoted the moderator and then putting that in there as well anyway so that's one example sort of of how you can get a bunch of code for free and Haskell the other one and I think what is probably near and dear to the hearts of a lot of people here is being able to automatically generate Jason serialization and deserialization models and that's that's it now I've got the ability to create Jason from those data types for free and if I want to let me see I've got some code here that makes it a little bit easier and now I am generating an arbitrary user and then printing out some JSON that reflects the structure of my data type exactly and I've for not very much code so far been able to declare a data model for this kind of contrived structure that I want to talk about generate some way to randomly create that data and then in the very very trivial amount of extra lines automatically generates serialization and deserialization instances for Jason I think this is this is also really awesome and so one of the things where this is especially useful is we have all this stuff and we can sort of play with it but if we can randomly generate data we can create property tests that automatically guarantee certain aspects of of what we're talking about and so people have actually written libraries that allow you to in Haskell derive complex property testing for free from all this information and so if I want to create tests that the data types that I have can serialize to and from or can serialize and deserialize to and from jason and that those will always work back and forth I can write a test module that looks like that and that test module will actually go through all of those types that I defined earlier and then verify that it can roundtrip the conversion to and from Jason and that it all sort of works out there's an added bonus it will use the arbitrary instances also defined to generate what are called golden files which are an example of the generated test such that if I change the model afterwards so that it breaks the serialization or D serialization guarantees that will also be a test failure so I can not only guarantee very easily round-tripping of serialization and deserialization but I can also guarantee that if my data model is ever broken that my tests will fail before it actually fails when someone's trying to call my service and so an example of running these tests here would be if I run the test suite I've got a whole bunch of tests that I've just gotten from that which basically are running like a hundred different property tests on each of these data types and actually creating these files golden user admin Jason that have random seeds put in there and now if I change the data model a little bit for the moderator and I rerun these tests the example I'll have somewhere else will fail so instead of that I'll just add a new field now if I run the tests one of the downsides of having the compiler do a bunch of work for you is the compiler takes time to do all that work now here we have this field is not present in the golden file we defined that guarantees our serialization and deserialization work and so because we've changed the model in such a way that might break with the guarantees we have with other clients we now have a test failure that will automatically show up when somebody adds a feel that they haven't accounted for and so with not I think very much additional work we've been able to put together some pretty useful characteristics from GHC that let us get more stuff out what I'll say is for free but as a downside this isn't particularly idiomatic Jason having some kind of object with a tag and contents and all these camelcase fields probably isn't what you want to present to people because that's not what they're used to they won't probably think it looks very nice and so as fun as it is to not have to write a bunch of serialization code this isn't gonna fly with with I think a lot of people and so what you can do is instead of just saying mimic the data structure exactly we can actually write functions that operate over the generic representation of our data types and then have these automatic generation these automatic instances that generate stuff change the way that it generates it for us so here I have a function that I've called idiomatic JSON options which takes as a parameter a function from string to string which is what I will use to modify some of the fields in there and then perform some additional modifications so here I'm saying the constructor tags for some of my types turn them from camel case to snake case by replacing the camel cased letters with an underscore and then filter out any back there any of these little tick operators which might not we might not want to put into jason saying do the same thing with the record fields however here i'm passing this function in to the record field so i'm saying do some modification that i'm going to specify below and then government snake case and then these are just a couple of extra things to maybe make the sum type and product type encodings in a house-call more idiomatic in json itself so if i uncomment some of these things here which are just modify the some encoding for user for to and from Jason and then for the other types literally just pass in this function generic parse jason with these options I have in this case because basic user had all of its fields prefixed by bu the modification function I have it just dropped the first two elements of the string and then perform those transformations which if we look at how it actually ends up working out gives us what I would say is a much nicer JSON output that you would expect from a handwritten serialization instance and now this is what our jason actually looks like a moderator is a top-level object that contains all the things of the moderator and it might be promoted by an admin who is structured in a similar way and so that I think is a relatively small code change but now we have idiomatic Jason and as an added bonus if we rerun the test suite again the golden files that we generated from the last test pass are all going to say your serialization format is totally different and they're all going to fail and what you do is when someone actually does this kind of change in a production code base you could say these tests are failing get somebody to review it make sure that the change we're making to our protocol is correct and then just delete the golden files we run the test suite again it'll regenerate a new set of golden files and then that will become your guarantee of what is correct behavior here so that was a lot and it was pretty fast there have any kind of questions before I go into sort of the next thing which is I think a little more crazy and I think cool about what you can sort of get for free from this stuff yes so the you mean the the arbitrary or which one I guess sorry so arbitrary so there's another aspect of production Haskell uh Haskell requires you to import a lot of stuff to do complex things so in this sea of imports the arbitrary stuff is coming from tests quick check gives me the arbitrary type class generic dot random gives me the generic stuff that I'm using to generate a bunch of these arbitrary instances and then some of these other ones quick check instances gives me orphan instances for all the things that quick check doesn't define arbitrary for so UTC time I want arbitrary time stamps but yeah not built in the GHC but pulled it in from one of the many libraries okay so yeah right so the question was are the golden files checked into version control and the answer is yes they should be checked into version control I'm not doing so here for the sake of example but what you would do is you would check them in and you would probably have CI run these tests and then if CI fails then somebody could make a change that deletes the golden files and then it regenerates them and that becomes a new golden master for your format so a little short on time so what I want to do is I want to show you off a final demo of what this all kind of looks like like what you can actually get when you really go overboard and so one of the things I've done is used a web framework in Haskell that will let you generically derive a whole bunch of stuff for free from it to actually create a server that shows off some of this behavior and so just kind of as an overall thing the whole demo is roughly about 250 lines of code and from that we can get from this data model and not very much extra Pascal to generate swagger documentation for our entire API with examples for what some of these different types do an explanation of the schema associated with different responses and all of this nice stuff all from a very very small Haskell module and even create a mockable or I shouldn't say mockable even create version of the API that you can hit and get all of these like an actual response from somebody back from there so I'll zoom in a little bit but this is just being served up i refresh the page it'll generate a new arbitrary instance and then give me a new JSON payload and so in about 250 lines of Haskell I can just create a random little server that I can serve and that have people maybe try and build a front-end on top of it without actually implemented me of the logic and make sure that all the wiring to talk to Jason is correct and that they can decode it and understand it and get all of the documentation sort of for free which i think is really nice and so there's only a couple minutes left but I just want to kind of very quickly gloss through what some of this code looks like I want to say that sort of the beginning part I would this stuff is probably relatively basic the instances stuff specifically overwriting bits of the JSON stuff is a little more intermediate and then this stuff is advanced so just kind of maybe a taste of what you can do and so this is basically what it looks like to define a web api routing specification instead of descriptions as a Haskell type and so this is actually a type that the compiler can do stuff with and so this is basically the get users route where this is the base route for users that accepts or that responds with JSON for a get request and then returns a list of users and then it can sort of similarly be done for the different routes and I can encode in the type level a summary and a description of all these different things here and for my actual implementation all I'm just saying is create a record of functions that handle the routes that correspond to the record accessors for the actual routes themselves and all these things do is generate arbitrary responses for all of them and then that actually generates the server and then beyond that I can generate the swagger specification from it where I have example versions of admin moderator and basic user and I could use the arbitrary instances to generate these as well but for the purpose of documentation I want to be a bit more readable so an example admin is Chris himself and then I'm a moderator and I govern this talk and then I use those examples to actually create swagger schemas with that information in there and this is all going to be I've got this repo publicly everyone wants to go through it and the master branch has a lot of comments if you want to drill down and sort of understand this a little bit better but all this is basically doing is saying generically derive a schema using some of the Jason options we had and then override certain fields so override the description field with this description and then override the example with this example of an administrator which if I go back to the page here you can see this is the administrator example in this data structure up top then it just gets wired altogether and then served in this little sort of main server file so a lot of stuff but I think that this is exactly the type of thing that Haskell can be used to make real production applications a little bit I won't necessarily say easier per se but let people have to do less actual hand wiring and hand creation of a lot of code that isn't necessarily tightly coupled to what they're trying to solve if you're really just trying to express your problem as like a data model for something then Haskell lets you get a lot of stuff for free from just the model of your data on top of that and that's pretty much all I've got so there's like about a minute left does anybody have any kind of questions that I'd like to ask yes [Music] so the question was can you specify rules so that the data that you generate abides by certain field validation is that is that correct you could probably haven't tried that out with the generic stuff I believe if you create I think that there should be a way to do it even if it means you have to manually create the arbitrary instances that generate it there should be a way that you can tweak all the stuff I think if there's one thing I want to sort of drive home that's useful about this is I can start from nothing and then get a whole bunch of stuff and then as the constraints of my problem become tighter and tighter and tighter I can override those automatic for free generations of things with the concrete implementation that I actually need and so even if in this instance there was no way for me to do it very easily generically then at that point I could just say well the generic solution it doesn't let me do exactly what I want so I'll just implement a solution concretely myself but you have the flexibility to do a little or do a lot yes at the top yep so all deriving generic does is it derives a way for the compiler to generically operate over the record itself generic is just sort of like my understanding of it is not great but there is a library called GHz generics which defines how you sort of interact with generic functions generically or data types generically and deriving generic basically says allow you to do generic operations over these data structures yep you could just do instance to schema for all these data types as long as they have adjacent instance it'll also work as well I have to double-check there might be an extension I would have to turn on but I think yes you could just in there it gets derive any class and you can do deriving each generic show to Jason from Jason schema and arbitrary I think cool well and I think I'm actually over time so thanks very much [Applause] you