Scale By The Bay 2018 Unconference: Devon Stewart

Scale By The Bay 2018 Unconference: Devon Stewart

Recording: Scale By The Bay 2018 Unconference: Devon Stewart

so anybody use auch HTTP in production HCPs okay so good yeah couple HTV for us yeah okay cool alright so but a good spread on both sides so this is a heavily acha HTP geared talk because we use our case should be it works so yeah it's the HTTP for us stuff we have pretty extensive test coverage so I'm pretty confident it will work but I don't have direct experience with it because I don't use it in production so but I am interested in getting more people who have experience with HTTP for us evaluating the code that we're generating and seeing if they can contribute back in some way so talk to me afterwards so this is a standard routing layer of HTTP we have this very interesting DSL for composing routes Anaka HTTP these directives are very convenient to just kind of slam out a server but very difficult to reflect out specification languages from so there are a number of projects that have in the past attempted to actually annotate the the akka HTTP infrastructure annotate the the the the the case classes that you use to back these things and then use that to reflect out some sort of swagger specification from existing code this is wrong this is wrong for a number of reasons mostly that it doesn't really work and it won't get it won't save you if you make a mistake you find out when your customers complain to you that your specifications wrong right this is backwards you need to start with your documentation first so the fact that we have code that's undocumented is not helpful so we can start with this as a baseline of what we want to generate from this project but again it's it doesn't have if you start annotating everything then first of all you don't know if you've missed an annotation and then additionally the the annotations end up cluttering the actual codes so now you're dealing with this tremendous amount of a plate on top of your boiler plate so you've got like both levels of problems and you're trying to just document something so instead we we saw I like first of all implementing functions because I like having a specification that I'm implementing I like having strict types for the thing that I'm actually writing and I want my compiler to help me as much as possible HTTP is a giant function from string to string or string to async string so if you're talking about actually trying to derive types out of HTTP that's also a losing strategy so starting with the actual starting with the specification and constraining your view of what you can even do in HTTP is critical so this has ended up working tremendously for us on my time on my team in particular we actually found some pretty significant security vulnerabilities so I was talking about security earlier I I I love that we found vulnerabilities from the compiler just migrating code into this system exposed bucks so being able to say no actually it's impossible to it's impossible to do this it's it there's there is insufficient information at this point and we thought that there was sufficient information but it only came out of trying to migrate into these more strict types so that will find me after I'll show you the actual difference it's awesome so if I implement this Handler trait the handler trait comes with a bunch of a bunch of abstract methods so if I don't if we look at what those methods look like so in this case would be creating a user so we have one parameter which is the user this over the specification would be JSON the first argument though you'll see is this thing called respond respond is a singleton type of the individual response type of an individual function so I cannot respond with a different type from a different function to this function you see that the future at the and is a user resource create user response type so that's it it went from a it went from that singleton type and then produced the resulting types so there's there's no way at the compiler level to respond with the different types they're just completely completely different so we can actually look at what this looks like in terms of generated code we have the the response type itself that closes over the status code this is describing which status code is for each one of the branches of the create user response class or after class and then we have okay and forbidden as two valid responses notice that neither of those take Patra take parameters because okay and forbidden both specified that they did not have any message body so okay could have been no content in this case forbidden also has no body so you're just responding with like a like a 401 or something so if we implement a simple version of this function the the easiest thing is to just take the user object and return immediately with a future successful of respondent okay that satisfies the required type we additionally if we're if we start adding actually I think there's a slide for that so let's go the next one if we yeah so if we if we swap out the the static implementation that we had before with something that writes to a date the to a database returns the future of long we we we've now used that value we flatmap that thing and now we're gonna yield it responded okay so the the idea is that no matter what your your business logic is you're still constraining your edges to these these well-known types so good good separation of concerns there then we can extend it a little bit further if we try to get from a database this gonna return a future of the option of user and then we can just fold like you would normally you normally do but you can see here that are not found has now added a method by a a parameter to not found that returns an error code so it's not only going to return 404 but it's also going to return the text of the like text plane response of no user found otherwise if we did find the user then we respond with okay of the user so we can see that we've actually changed our specification to have these new parameters as well something interesting that I will point out here is that even though we're using okay and not found as symbols there's no there's nothing tying me directly to http here they are not actually seeing anything on the HTTP layer so the the implementation of this trait this implementation of the handler is completely abstracted from the underlying transport medical transport mechanism so if you wanted to write some sort of tests that tested just the function in isolation you have this implicit separation of concerns by by construction like the the handler itself you can test the handler you can supply a mock handler to test the routing layer if you need to the whole the whole platform is or that the what you're writing is separate from the routing layer to the handler those are completely separate concerns so you can see that that actually ends up creating first of all readability I don't I no longer have to worry about the the the particulars of the routing DSL of the framework that I'm using right now I or if I'm switching from one framework to another framework most of the types that I'm dealing with are already mapped for me so I don't actually have to worry about I'm gonna migrate from akka HTTP to HTTPS or HTTP for s sake HTTP if I'm using some if you're if you have some specific use case that needs a particular framework right so your you can very straightforwardly swap from one to another because most of your types are actually abstracted from you in that respect so we can actually see here this is a complete mission of a server this is the create user get user and get friends functions are all stubbed out all you do in Aki HTTP is you call user resource so there are two concepts this resource and handler the resource object has most of the support definitions and the function called routes you pass your user handler into the routes function and then you get a specific type response like a type of the framework that you're working with so in this case we're getting a route because we're working with akka HTTP so off HTTP has this route concept so we get a route we can compose those routes with other things in our case UDP so migration is also very easy to migrate into the system because the idea is you it's it's sort of at every level you you have at the edge there are no more dependencies between guardrail and the underlying frameworks there's just you get the underlying framework back and you can do whatever you want at that point so you can use whatever testing frameworks already exist for the libraries are using you can compose them with other things you can compose different servers together it's at that point it's entirely up to you so you take that server that we discussed or that we created earlier rather there and then let's say you have some routes that you're migrating let's say normal routes so if you combine normal routes your existing routes with your new routing infrastructure you combine that to port 8080 and now you have a completely running server we've partially statically well typed code generated from guardrail and the partially handwritten code that you're not not yet migrated so this is a useful and good working working pattern so you can actually do this so it's a sort of migration in a very straightforward and well supported way so yeah well so we can see that if we if we change the schema if we add this this body and like I think that yeah so if we if we add a response body to this 200 then we can actually see that now instead of used to not require a string now it does require a string we can actually see that the response will require us to supply this okay here so we we actually if we attempted to compile this I wonder if this will compile yeah so we'll see that in the in the code sample shortly so the the string that is supplied on that okay is now required so if we had compiled this without changing that okay to accept that string then this would not have compiled so from changing a swagger specification you're getting compiler errors in your application this is super good this is what we want so if I then pop over to yeah so let's jump to the actual live coding part of this which is going to be interesting and also this I work best when continually interrupted so if you have a any sort of questions or you want to shout things out at me while I'm typing that's that's totally welcome I encourage it so I have this routing DSL taken from the akka HTTP documentation this is what their standard routing DSL looks like but we're actually going to start by creating this 2d resource with an empty to do handler so we can actually see here that if we compile this will actually say whoops not even found because in port right so if we attempt to compile it now it'll say hey object creation impossible since method create item in this trait is not defined so what we can do is we can grab the error message and I like when there are more than one method not two not defined errors because then it will print the entire stub for the class for you which is great so we can actually see here this is this is a reasonable at least to start definition of this thing and I've forgotten the that no parameters for this function and it returns a Scala concurrent future which it does so we can compile this it will compile doesn't do anything yet but at least it does run so if we attempted to run this we can actually first of all attempt to call that create item function and it will explode because opes because I did not so let's take a look at the specification to find the route so to do so so if I do that then we get an internal server error which we expected because we had that [Music] implementation is missing so excellent let's provide an implementation for this function that just ignores any parameters that we would have gotten and returns a future of just an okay so when it's before you do that we don't need this anymore and we can say future successful respond okay so if we write this now let's start back up we can actually get will see that the responses that we get are very empty HTTP responses so let's log this the verbose chatter talking to this server did I do a 1 of course so we've got and we'll see that the when I make a request I just make a Omega post and that will respond with a 201 immediately yeah and that's also true because I specified that this returns a object so let's just stuff that out so if we now run this this will every time that the plug-in detects that you're trying to compile anytime the compile action is run in SBT we also have a maven plug-in but anytime that you try that you run compile it regenerates all the code which means that if your business logic is completely abstract from the writing layer if you're writing layer has bugs or if there are any bugs in guardrail itself or missing features or something in the future one of the big problems that I have with most code generation frameworks especially in swagger land is that you have to commit your generated code this is a tremendous sin in my respect all right my opinion because this ends up preventing you from continuing to iterate on the back end the because now everyone who has ever generated something from your tool now they have this locked in time version of this thing which may have bugs or may otherwise have missing features or things like that so being able to generate this every time is is absolutely critical so we can see I'm going to post verbose Lee and we can see again the tooling created so that's that's what we expected so if we now put back this definitions item we can actually see first of all the definition of an item is it's just an object defined in in open API on a swagger specification it has two required fields ID and contents the idea is an integer and the contents is a string so if we change this created two we're going to import from to do definitions as well and we're going to say yeah we want that one item and then the first thing is long and this is an item right so we've got that and it's if I thought I had I would have made it so that you can hit enter something and cancel this bind and I did not think that 4head so if we make this request now we should be able to see a JSON object come back with just the item that we created we can see that the ID is 1 and the contents is this is an item it is normal JSON there just great so in on the backend we use Cersei and we use Cersei for akka HTTP and also HTTP for us it does the derivation of all the encoders and decoders for you and does some special mapping around read-only fields that you can specify in swagger as well so that's it's all on the back end it's it's code that you would be able to write yourself it doesn't use reflection doesn't use as instance of it's all like reasonable code on the back end and we've had a couple of situations where during the development of guardrail it's made sense to just take the generated code directly move it into source main Scala modify it to fix bugs like it like credit like critical issues where it's like this feature doesn't even exist so they migrated code into their codebase patched it sent an issue to me I patched in guardrail and then they were able to delete the code once it was fixed like once the missing feature was added so yeah there's it's a it's a good iterative strategy additionally I want to talk about if if I have a couple of minutes at the end I want to talk about the the iterative nature of Scala meta so I'm Scala meta is what's being used on the backend here and we're also expanding into Java parser so the let me actually do one more change here I'm going to add a property to this to this field and see that my my code that previously compiled and ran will no longer compile and run so user is actually going to be instead of a item it's a user type is an object and I'm going to say required ID name and properties are ID type integer and name is type string right so if I do this and I say the user is actually a ref two definitions user then this should no longer compile so we can actually see I'm not gonna fix this compiler error because this is the the whole point is that if the specification Lane language changes if you're if you're whoever is designing your API is or if anyone who is part of your build process oh because I didn't mark it required ah good okay cool so user so now the user field should be required you can see not enough arguments for apply you can see ID long contents and then user is a user object so this actually did what I expected it to do this means that now even people who are not interested in writing code not interested in writing scala not interested in collaborating with you directly in your project can write your API specification they can start writing their implementation against their spec then you generate your service based on that specification and implement the business logic so this fundamentally changes how teams work together at a large scale so I highly recommend that you use this thing because it's cool so if we have a couple minutes so if we want to add custom support for a type that is not supported swagger if we want to add like business logic that is specific to Scala to help help you write better Scala then you can define your custom types or if you're using the refined library or anything like that you can already specify a refined type and it's so long as you have a refined adapter to Circe it should automatically pick that stuff up we're going to write something custom to do this right now where we can say noon on if actually we can say so we have a string and then if the string string is empty then we can say I can't remember exactly what the syntax of this thing is so Matt flatmap yeah so if we say decoder yeah cool so if we if it's not empty we can say decoder Const of new non empty string string else we can say decoder failed and then oh okay that would be good I'm gonna continue with this but that would have been helpful I don't typically do this low-level stuff anymore yeah non-empty lists or a string required so I'm very interested in making it so that you can use the the standard decoder encoder strategies that you would use to say like a cat's non empty list or anything like we use cats under the hood we use source the end of the hood so if so long as you have something that's capable of encoding or decoding from that thing so this is I'm gonna actually do the game app thing because decoder string map cool so we can actually do that much easier than EMAP so if that then we can say right of that value otherwise we can say left of that and as a result of that it should at least compile okay plus a decoder so it is actually looking for an encoder now so right so contra map on non empty string so that should give us something that works and then again we don't have all of the the definitions but if we make this optional again then we can see the generated code yeah so it it asks for now where we need it to be a non-empty string and you can you you saw the definition for that non empty string it's just a map on top of or it's it's some sort of handler on top of string so adding additional business logic into your types and then that's all folded back into the writing layer of the underlying framework that you're working with so so that's good additionally I want to extol the virtues of multi-level testing techniques I hate writing tests and I hate tests obsession general I want my compiler to do everything for me but unfortunately if I make a mistake then that impacts all of you so we have five levels of tests so we have first of all of the compiler we write we write good code but then if the compiler starts complaining then obviously that's not good so we've got the compiler as our first level second level is we actually have tests that run against the compiled code so we have tests that test like it runs the generator with some stock strings like static strings and then it verifies these the structure of the generated code then we have the third level of tests which is running the code generator so we actually have as part of our CI build process we actually run against a number of known tricky swagger specifications and then we on the fourth level try to compile what we generated and then we actually have tests that are written against generated code so we actually test way too many times but this ends up it's ends up finding some really cool regressions and and other things and and enables to call back to a previous presentation from I think yesterday the has set us talked on having a successful open source project we have a platform in the way that the tests are designed to enable people to contribute without fear this it because people can come in and start just throwing code around and it just won't be able to be merged because the code will not pass the tests like this is the best thing and then if somebody comes in and says well I have this really weird edge case they send me a swagger file I drop it in the thing and then I look at what the output of it is and I literally copy and paste the generated code into the tests so if we can take a look at like backtick test so backticks there used to be a bug in scala mehta where scala mehta would not correctly encode special parameters in scala so you would have to like if you have a variable or value with spaces in a type with spaces in it in scala you can surround it with backtick but it's called a meta had a bug that did not have this capability so I have this very unpleasantly structured swagger file with loads of like dashes in routes dashes and packages like spaces and various places and what I want is I want to make sure that the generated code exactly matches this ast from Scala meta I want to be able to say the the client here has to exactly match this structure even if this is undesired even if this is undesirable if we look at this class name like - e - package capital C client like it it doesn't we don't even know how to correctly transform this into like a human usable or expected class name like there's this this like dashes underscores like we usually handle snake case converting that into traditional like Java style package and like all camel case but in this case we just didn't we didn't know how to handle this - but even so the test verifies that that hasn't changed we want to make sure that we're exposing a consistent interface to the user so that because we have this structure and because we can walk this this AST and say this these are the set of things that should be in this thing this is the set of things that should not be in this string then we can actually talk about comparing the structure of the entire class like the the generated class structure should be exactly equal to the the structure that we that we have in our test so the Scala meta is an amazing project if you don't use scala mehta and you're doing coded by hand please consider scala mehta it's fantastic like the the structure of it is is just such a joy to work with so that's that's pretty much all I have to talk about right now but I think I have a couple of minutes for questions okay so any questions where we actually allow any F to be passed in yeah I would expect so again don't use it so I can't say anything about that yeah yeah of course so as you're supposed to when you're going into a demo you have dirty code all over your system so ignore the fact that there are probably going to be squiggles everywhere but while I pull this up I saw another question yeah yes so we have as of I think three weeks ago we have support for all of we're adding support for the more specialized forms a validator using we're looking at using or fine for that it's not currently supported but I'm interested in discussing with people on whether or not they would first of all use that feature since it's been mainly demand driven development but then also how they expect that feature to work because that's I have not needed it so I don't know how I even would want to write that so yeah if that's something you're interested in talk to me definitely not today no no we do have enumerations and we have olive so we have especially with the the ability to reference other yellow files from Gamal you can already do like code reuse and like splitting definitions apart into different services like there's a lot of structure that you can get out of the way that yamo works and that swagger or opening open API works today that kind of gets around this it's something that I definitely know is it's it's a glaring hole in our implementation but we started from a ground up and like-likes ground up because when I tried to run the official swagger coach and there was a bug that has since been fixed but they forgot a it's it's all mustache templates internally for them and they forgot a new line on one of the files so all of my classes were on one line which is somatic okay so we just ground-up rewrote using Scala meta and and we're adhering to a strict philosophy of AST is for every support of like every supported language so we can see here these in Scala meta you can actually write abstractions that they that look like normal Scala code but using quasi quotes they're string interpolated so if you're familiar with with macro paradise or anything in Scala it's it's very very similar to that except you can actually do things like this where you can say defu string is like triple question so I can get that reference but then I can actually say I'm gonna actually pull out like the function name right I'm gonna actually pull that out of what I've just created and now I've got this reflected term named foo that's just staggeringly powerful so we use this in a couple different places to make the code easier to write and easier to read as the as the Cochin evolves this is otherwise it's just managing huge trees of AST and it's just not pleasant so there's a there's a lot of that in there and I can show you more after the talk if you're interested any other questions yeah oh yeah represent represents awesome yeah represent is a SBT powered power that the uses power features under the hood to to do that and it runs code on each slide but I didn't set that up correctly because I mentioned that those slides from oh that's six months ago so yeah yeah it's your use is suppose to be able to go to a slide run the code on the slide and it'll actually show you the results in the thing so it's it's just super cool yeah any other questions yeah cool thank you all right yeah okay yeah thanks [Applause]