Ninny json: type classes that know that none is not null

Ninny json: type classes that know that none is not null

Recording: Ninny json: type classes that know that none is not null

great everyone thank you for coming to our Meetup we're on to our next uh speaker now and the topic of this talk is on nin Json type classes that know that none is not now uh by Nathan fiser uh uh data fit is a San Francisco based developer using Scola to increase productivity and push errors to compile time over to you Nathan thank you for doing this talk all right thanks Sol so yeah um like salar mentioned I'm a scholar developer in San Francisco um the team lead at light form uh we make projection mapping software like this glittery thing in the background you might be able to see um um giving a shout out to the company because they gave me some time to work on this uh during work and they're also really good about open sourcing some of the Scola stuff we built internally and giving me some time to work on um some open source projects in the Scola Community like Mill and stuff like that so this talk is about as we mentioned that none is not null and what that really means is in Json libraries most of the ones in scholar today um don't make a distinction between a null Json value and a field which isn't present and so we're going to talk about kind of why that matters and what we might do different so nin Json is a library that I wrote which has some new type classes that aims to address this issue making a distinction between what none and null are or absent fields and and Json null and so we'll talk about why we care what the difference is what the libraries do today just a survey of kind of the three most popular or three of the most popular Json libraries in Scola some new proposed type classes in nin um go over some questions and if there aren't too many then we'll jump into some usage usage examples of what using the N Library actually looks like so what is a type class um I know people from the from the Java Meetup and maybe some new people to Scola so we've got this kind of scary looking Wikipedia blurb and it talks about uh ad hoc polymorphism and we're like that's kind of a a big term but really all that means is that it's allowing us to Define abstract Behavior Uh outside of the type hierarchy so in kind of more simple terms usually we think of abstract Behavior coming from a super type so we say this thing is a sub type of that thing therefore it has this Behavior type classes let us say I don't know where this thing came from but I know that I can Define this behavior for it so an example of that would be if we have some associative Behavior we can combine two things of type A into another thing of type A and for integer we can Implement that for addition so just add two integers together easy um and so that would be a type class and one of the big things in type classes is composition so if we have another type class sum where we've got a bunch of instances of some a we want to be able to combine all of them together not just two and so we can say things like okay well as long as we have an associative for a then we can build a sum for a and we can just you know reduce it down and now we've got a sum for our list of of any anything not just of integers but of anything and so that composition and that ad hog polymorphism are two of the big attributes of of type classes and we'll see this carry over into Json um as well in terms of building serializers and D serializers to and from Json uh for complex types so for the Java people do we have type classes in Java we definitely do um you might have seen comparator Compares two things of type T until you which one's bigger or if they're equal um you don't see type classes in Java as often because the implicits that Scola has and the higher kinded types make type classes a lot more useful um and so the the reward of using them in Scala is is a good bit or the implicits make the the difficulty of using them lower and the the higher kinded types make the reward of using them higher um so they're definitely more popular in the style of functional world so who cares um null absent field what's really the difference in scholar we like to say that that uh null is none anyway so one reason and maybe the most minor one but is testability so if we've got some class and it's got an optional member and we turn that into Json we might want to make an assertion that says hey once I turn this thing into Json it shouldn't contain its it optional member if that optional member was none right so we going to say yeah the value should should not be in that field or another way we might write it is that hey if I try to pull the value field out of that Json I should get none but with the libraries today that's not actually what happens so these assertions look correct but if we run them they're not actually going to be so our uh our output Json actually might contain a field called value and we might not get none we might get null and so instead we need to write a test like this that says hey if we try to pull it out does it contain Json null which covers both the case of it's not there or it's null and that's not the end of the world but that's definitely not the first assertion we would write when we're just going through our tests and so that's kind of a sign that things are a little weird so another case where we care about the difference is in Json RPC I don't know if anybody's worked with it but it's a a language independent transport independent RPC um standard and it uses Json as the serialization format and in Json RPC you've got requests and responses and in the responses you either have a field for the result or a field for an error and the client will look at those two and say okay you know what happened with my request and the spec specifically says that you can't have both of these fields in the object it's got to be one or the other full stop and so if we use a lot of the scholar libraries today and we serialize something that maybe has an optional air and an optional result as a case class this is the Json we're going to get but this is the code that a lot of clients are going to write which says hey does that Json contain error because the spec told me that if it does then it was definitely an error and I shouldn't expect that there could be both of these guys or that error could be n and so that would be a correct implementation the client but it would break it would try and parse the air and it wasn't an air it was actually a success and our air was just a null [Music] placeholder so the last example um is an RFC called Json merge patch which is pretty cool and it lets you kind of partially update resources and it says you've got some resource some existing resource and some Json object you can take all the fields that are in the Json object and update the corresponding field on your resource and if the field isn't in the Json object you just don't update it and if the field is null in the Json object then you try to null it out or delete it or whatever that means for you resource so there's kind of an obvious um obvious issue there where if all of the fields are always present and they just happen to be null if you weren't putting something in there then you're going to end up deleting you know the majority of your resource when you try to go update it so this is the case that we'll use as an example when we go through look at our different Json libraries we'll try to implement an algebraic data type uh for merge patch and see if or how we're able to do that in our different libraries so do we want to write another Library another Json library for Scola we've kind of got a million of them already um so let's dig into like what are the ones what are the most popular ones we've got and will they work for those use cases that we talked about already so play Json is a big one um we use it a lot at light form uh currently it's got some pretty straightforward type classes we've got a rights that writes a value into Json and a reads that reads some Json into a scholet type of some kind and that's a JS result so that basically says it could fail um but in this there's not really a way to say hey there should not be an output here so rights when it's implemented for option in play Json just writes null and N or JS null is you know a subtype of JS value and there is no reads for option of anything because there's no way to know from this type signature if the field was absent only if it was if it was null so that's a bit strange um play Json has some macros which are handy so we don't need to implement those type classes by hand for each of our types so we have my class we can just make a read for my class by just saying json. reads my class nice and easy um and it uses implicitly a reads for string because that's the type of the myfield member but we see something kind of strange if we have an option number because we said we need an implicit reads for our field and we said in our last slide that there is no reads for option so how does that work how are we able to resolve that um it's because option gets some special treatment in play Json on the read side and this macro basically generates this code and validate opt is a special method for option that is able to use the normal type class for string and decide whether it should use that or return none based on the context of looking up a specific field in that so that works fine when we're working with option in like our normal cases but if we go and try to implement kind of a algebraic data type for patch where we talk about wanting to update the field clear it or ignore it based on if the field is absent or null we can't we can't Implement a type for patch and there's no support for patch pch in play Json macros so what that means is that we have to do the same workaround as what the macro does uh and basically Implement our own reads and writes uh for the data type for every single data type that contains a patch member and so that's kind of a drag um and we can basically see that if we want to ignore the field there's no input value that we could find that would lead us to figure out that that's what we need to do and on the writing side if we have an ignore there's no nothing we can return that's going to tell it that that field should be absent so in summary play Json is a little weird because there's a type class for option on the writing side but not on the reading side and we can't use macros if we care about if the field is present so that's going to lead to a lot of boiler plate Cersei is another really popular library for Scala or for Json in Scala um it doesn't use an a for its Json or it doesn't expose it at least it just gives you this top level Json value type uh it uses these things called cursors and you can think of the cursors um as something that sort of looks at a specific part of a larger Json object and the a cursor is kind of like a try in Scala so it can either be a success or a failure and depending on which you can kind of infer different stuff so it's type classes the encoder looks pretty much just the same as it did in play and it has just the same uh drawbacks it you know writes none as null and yeah the decoder though is interesting because it takes a cursor rather than a j value as its input and so that cursor potentially can be a success cursor or a failed cursor and there's some subclasses or overridable methods in that decoder type class that let us take the a cursor rather than the success cursor and from that we can infer that the field was absent so that means we can Implement a decoder for patch even if we can't Implement an encoder so Cersei in summary we can't Implement that encoder we can implement the decoder and it also has macros just like play but again if on the decoder side you want to or on the encoder side sorry you want to do something you need to go and Implement that whole um parent class by hand so the third one is Json for s and Json foress is a project that aims to have one uniform algebraic syntax tree for Json and Scola that different libraries can then use but it's also got typ classes for reading and writing and these look just like the ones in play with the difference that these throw exceptions rather than return uh success or failure as data and the interesting thing in is that they've injected this J nothing in and there's no concept of nothing in the greater Json spec but in Json for S they kind of put this guy in here and it lets us Implement patch the way we want to and so now we can use J nothing in those places where we didn't know what we were going to do with play Json um so that's great so we actually have a way forward now with Json fors but there's still a little bit of drawback so if we have some objects some Json object that we've built by hand that has a j nothing field we get something common kind of reminiscent of our weirdness in our tests where we say hey does this object contain a specific field and that returns true and if we go to look up that specific field we find oh there's nothing there and so that's kind of a weird in consistency and we could just write oh let me look up the field and check that it equals nothing but ideally our first attempt wouldn't compile at all so that's something we like in scholas that if you do we do it wrong hopefully it doesn't compile and you know the compiler tells us about our error so yeah um hopefully we can do a little bit better than this a even though it does give us what we need to to implement our patch you know maybe we can do it maybe we can do it better and how do we do it better so what we want to do is move that concept of nothing from the where Json fors has it and move it into the type classes so in nin the as for Json is very vanilla it's just got all the normal types you find in the Json standard but if you might have guessed from the title we've got options when we're going to Json and options when we're coming from Json and if we look at what that looks like when we go to implement patch it looks a lot like it did in Json for S except that now we're able to say okay we can say it's just not there so we can just say ignore is just none like don't output that Json field at all and it won't be present in the either so if you're Eagle eyed you might have noticed that the patch 2 Json has a has a type class that I didn't mention to some Json and basically sometimes we want to guarantee that our thing is going to Output Json and not just nothing right because we can't return none to our API consumer or something like that need some concrete Json to to give them so to some Json lets us do that and it also lets us bypass um you know the awkwardness of of always having to deal with options uh and as a bonus it means that our patch to Json only works for an a that definitely outputs Json and so we prevent the issue where you might try to serialize and deserialize and end up with something not symmetric um if you tried to write an a that was not going to Output Json that would get interpreted as as ignore rather than update to none and you can't get that with uh J nothing so like I said it's kind of a drag to always deal with option and to some Json kind of the implementation is here let's us let's us bypass that and from Json that's not really an issue we can just add an extra method to the existing type class so sometimes we said okay we definitely want some Json but what if we definitely want some Json object and not just any old Json what if we want both how are we going to do that with the type classes we showed well we can't and I lied those aren't actually the real type class signatures there's a couple different and basically the type of the Json value is something that we have as a generic here and we have type aliases for these different um specific cases and this actually works pretty well so the implementations for patch that I showed before those don't actually change in light of these new types that we have they work just fine with the type aliases so largely when you're using the library this is kind of behind the scenes you don't need to know that there's a 2 Json value you can just use 2 Json of a directly so we found lots of different problems with our other libraries and we went over some solutions how' we do in Shoring up those problems so trying to get a patch from Json was a problem for play trying to get a patch to Json was a problem for Cersei and play the As was kind of weird and Json for S and having null floating round was a problem in all the libraries as the default way that you write uh none so we've managed to fix all of those and we also actually have macros thanks to shapeless so we don't need to write any of that boilerplate overhead that otherwise would have brought us onto the same level with play and that will wrap up to any questions and if there's no questions or we have extra time after questions um we can go through some usage and code examples of using mini awesome any questions everyone I'm keeping an eye on Twitch one question for you have you considered uh using something like Magnolia instead of shapeless haven't used either of them Ser but I but I've um heard from others that on benchmarks Magnolia is faster I have not um I have not checked it out but I definitely would be curious to uh there is a GitHub project and a PR uh showing what the Magnolia implementation would look like would be awesome cool anything question from the chat um was curious if it was supported in all scol versions yeah so on GitHub there's a issue open marked as a you know Help Wanted currently this is implemented for 2.13 and it should be pretty easy to get AC cross building to 212 I'm not so sure if it'll be as easy on 21 um but I definitely also want to implement Scala 3 support once that is finalized so 213 at the moment I think the person said they were using play framework FYI uh framework or uh Play Play Jon I asked I asked the chat question of what's everyone using and he replied that is using the play play assuming that he meant play oh that was from earlier okay sorry yeah yeah this should be a pretty easy drop in uh replacement for blade Json for a most use cases actually um you can just do a simple like control you know replace and just swap out the macros and the Imports someone's asking now is there any other external dependency for n Json good question uh I believe it is just no it's not just shapeless I take it back uh it is shapeless and [Music] um actually this is at the end uh shapeless and John so JN is a Json parser so we offload all the Json parsing to John where it's a really fast robust parser and we don't have to worry about messing that up ourselves I thought about bringing in cats but didn't do it great uh I don't see any oh here we go is there way is there a way to allow and forbid which Fields the client can update in a Json patch definitely um I usually implement it by having a data transfer object that is a case class that has all the fields that are allowed to be updated um and then has that you know something like that patch data type as the value and so then you take that data transfer object and apply it to your resource and so only the fields that you put in there are going to be updatable on the other end great any more questions if not we look forward to your examples that you mentioned you've been thanked on the chat awesome yeah appreciate you uh coming and presenting and taking the time to prepare all this um yeah look forward to uh hearing more from you if you want to come back we like I said definitely want some more Scola uh on the SF jug um so you're certainly welcome to present to anytime both of you so um yeah thanks Nathan app yeah I'll maybe do some more more educational uh beginner schol stuff did do you have any more to show today Nathan or is that the end of your presentation because I thought you mentioned you had there was something you wanted to demonstrate yeah just some quick uh quick showing what that code actually looks like in use cool that's awesome you've got some really straightforward you know you've got some variable or value you can just say to Json and it will give you that Json value or to some Json if you definitely know you want it um you can take any Json and convert it into your Scola class with a just plain two and that works both on an option and the actual Json um to and from json. Auto invokes that macro and it just builds the type classes for you there's some cool Dynamic syntax where if you parse out your Json you can invoke you know a method like hello directly on the Json value and that'll if there's a field called hello it'll return you know the Json node at that field um there's also a non-dynamic syntax in case you need to pass it a string but I think Dynamics are one of the more fun and underutilized scolar features that uh people don't realize you can safely you know have have funky syntax like that um there's definitely some play Json inspired construction so it's a lot similar to building a map in Scala or building a sequence and if you do want to write out marshallers or UNM marshallers by hand um they return a try and so you can just build them out in a for comprehension that's pretty tidy uh you could also write your own automation to just generate this um if you're had some other external tooling like a like an open API specification or something like that would be easy to generate this kind of thing from so yeah ultimately I hope that uh other scholar libraries adopt this capability in their type classes and that we can sort of you know lay lay n Json to rest but that uh that may may be quite a while off so in the meantime I'm aiming to get it production ready I mentioned it uses John it's got quite a bit of test coverage and I'm going to be building out some Integrations with other tools in skull ecosystem um AA HTP will will probably be first and I'm contemplating adding support for binary Json um we do a lot of Internet of Things style mqtt communication and light form so uh some binary support there will be something we might build in house and open out and it's definitely uh definitely open to feedback or contributions on GitHub encouraged you might even say and that is all I have great thank you Nathan any new questions from anyone thank you so much seriously it was wonderful talk I'm glad to see more options for Jon par in scolar uh nothing in chat anyone on Zoom got any questions for Nathan I think we can call it w