Scale By The Bay 2018: Jeff May, Play on Dotty: Design Patterns unlocked by Dotty in a Play...
here we go rally health we're mostly scholarship I learned Scala they're been working on Scala for a little over six years our first task was to for me was to migrate from the Zend PHP framework on to lift web framework so naturally my next project was to migrate off of lift web framework and on to play but more recently I've helped move from play to three to play to five and surprisingly this ended up being one of the harder migrations we've ever done namely because of binary compatibility so oh yeah we had started open sourcing a lot more of our code you can find it at github comm slash valley health so what's the plot of play web framework it's a popular Scala and Java web framework maintained by light Bend it's modeled after some web frameworks like Ruby on Rails and Django in some ways but it's a lot you know less prescriptive about your models so it gives you controllers templates filters these are all just sort of functions that produce some results either in a template case use a function from the template arguments into that resulting template in the case of controllers it's a function from a request header to some essentially iterative you know byte chunks that come in to a result they've changed it over to akka streams but essentially the same thing it looks a lot like this you have your controller will have methods that take in some parameter from the routes file you can decide to choose a request or not if you do you can make it implicit you can also pass a body parser here parse JSON and that will return a 400 if the request doesn't have the appropriate headers and if it does it'll try to parse it as JSON and if that works you'll get your your body will be request will be a jlu all right so what is Dottie you're probably familiar with this by now but you know it's gonna become Scala 3 it's the compiler back-end for that it's emphasized emphasis is on type safety functional programming patterns and sort of streamlining a lot of the common things that you would do with Scala but making a little easier a lot of new features I monitored sq1 over enums yesterday and some other interesting things like extension methods in the past he's talked about implicit functions which I'll be going into more detail today there's a lot of other really interesting stuff you should check out match types are quite interesting to multiversal quality you know these types of things so there's also a lot of features removed XML a canonical example but also generic type projections existential types and macros for now at least for this talk have been using version 0 10 0 RC 1 which came out not too long ago and it's supported with SBT 1.2.3 and above using a plugin so I was inspired to work on this based on the talk that Marta nerd risky gave at Scala days last year in Chicago and on what to leave implicit so I highly recommend checking that out so what I wanted to explore was this idea of context he was talking about you know implicit SAR to represent context so alright what is context I kind of think of it as like you have some input say a config file or your request headers or something that is available as input and then anything that you can sort of derive from that that you want to keep available for that lifetime of that input so in the case of an application if your input is a config file well it's the lifetime of that whole application until you shut it down or restart it with some new config so here's some other examples I come up with obviously the application config example request headers could be some input that you would want to keep around you know maybe you want to access some from the request at some point later or maybe a database transaction roles and permissions authentication information etc so one way you can pass this context around or really the only way is through parameterization and there's many different forms of parameterization so the way that you would do it and a lot of languages out there is you would just add an extra argument to your function so you know take in some argument X and then you need the context you would just add another argument to your list you also get in scala another argument list you can create one to do what's called currying and and the Scala's format of doing that you can also return a function which is essentially the same as currying but you don't have to name the second argument and you can also abstract over the return type and this allows you to do a lot of interesting things with type abstractions another one that often really you know you know hear much about is returning a module so it's sort of capturing the argument the first argument and then providing a set of operations you can perform on that argument and then here you can supply your context if you wanted and then lastly and probably most commonly you've seen is taking an implicit argument list and this is not quite currying it looks like hurrying but actually the compiler looks for the context the moment you call f you can sort of thread context through by making the thing that calls it require an implicit context but this send of bubble up if you started with say a function value and you want to pass this context well if that function value has to take an implicit well then you have to make that a def instead of a val and so this can you know changing something to require an implicit argument list can can propagate kind of in ways you might not want so what's new in Scala 3 is that you can put the implicit keyword into the function same same as you did with returning a function and its relationship to the explicit method currying where you don't have to name the implicit argument and you can abstract over in the return side so this has a lot of interesting properties alright so I want to talk a little bit more about modules because I think this is kind of relevant you know everything is Scala is an object you can think of it like that every object has can have methods and a module you know it's kind of like an object that is created in memory or it could be a package object and you can invoke methods from that the apply method is sort of a special method that allows an object to be invoked like a function then of course you know that's how function is actually implemented it has other methods like and then and compose and so forth but the main method is apply turning a module is a great way to capture some of the dependencies and return a set of named functions based on those fixed dependencies modules that have no dependencies you know you can use a singleton object pattern for that using the object keyword a good example of when you would use modules is say you're building a dependency injection or you're building your system through dependency injection you know you want to decouple the components from each other and instead have them depend on abstractions so that you can modify the implementations without having to recompile the things that depend on them so here's an example of like I don't know some silly web framework for our web application you might build you know you have your services module your collection module and maybe your collection module depends on a database module and you have an implementation called Postgres module and these sort of these modules can be you know dependent on each other and you know the constructor of user service may depend on the user collection for example and the user collection may depend on Postgres connection that's sort of pattern of building everything outside of these classes is inversion of control so for dependency injection I highly recommend Adam or skis talk on dependency injection he goes over you know the various ways of doing it through using constructor based dependency injection as you would sort of do in Java and showing how this is actually for the most part it's what you should start with it's it's really easy to do and it's it scales well and it's uh has a lot of benefits but there's one reason you would want to you might want to use a monad and that is or monad or a function and that is if you want to put the argument into you know the context you want to take you want to put that into the argument of the method so that each method becomes a function that you can parameterize so this is just you know visualization of the object graph ideally you know if you're doing manual dependency injection you want to be a cyclic if you if you have a cycle and you're using manual dependency injection you're either going to end up with an infinite loop where you know the things trying to construct a is trying to construct B is trying to construct C which depends on a and you just cycle or if you're eagerly evaluating you'll get a nullpointerexception but you know once you have your whole application wired up everything in your application should be you know configured as needed there you can recreate some of the sub modules with new inputs ok so I'm moving a little fast because I realized I've less time that I thought and I want to get to the demo so bear with me but we have you know first example case of how we might parameterize passing this context we could just use explicit functions so here we're taking a function of execution context and returning a future event never mind the fact you probably would never need to add two integers in a future that makes no sense but just for demonstration purposes so you would you would you know take your implicit execution contexts call your future that requires an implicit execution context great passes it along in this second method here we're doing a little bit more with with with our results so we're saying okay we want to take some result and report it on Io T I actually don't know what IO T stands for but I saw an atom or skis talk and so I figured I'd just throw it in here now whenever I look up IOT I see Internet of Things and I don't think that's what it means but well here we go so it's reporting to the user or somewhere that this is the result of calculating that all right so now how do we use it let's say we wanted to like capture these arguments and apply it at some point in the future so I create this function of execution context and IOT - future int so I need my arguments I declare one of them is implicit and I go through and do my flat maps over the future returned by ad and you'll notice that you know like we have to take the argument there we don't really care about EC we just have to name it we take these two arguments here it matters because you know on the right hand side we're executing or passing EC we have to take them both here as well and then we have to declare it as implicit here and then we have to pass it explicitly in a lot of these places so you know passing it with functions it's pretty good no external libraries necessary you can extract over the return type with an alias but you know function arguments can only be declared as implicit when you're defining them when you're passing them in and you know you have to sort of explicitly thread them through the code and every time that you call a function that returns a function you're creating a closure so some disadvantages okay so you can avoid having to pass these things explicitly by capturing the context in a reader monad a reader monad a-- basically is just you can think of it like a function in which you can map and flatmap over it keeping the the initial input so this is just a very simple implementation of it and and so you know how how would we end up using this so we can create our type alias which is one nice thing about having it all represented in the return type and just to make the type inference a little easier i created this async method it I had to keep explicitly putting the return type so doing this help to avoid that and then you know now we can define our add method and we take an implicit execution context and return future sorry those I thought that was another arrow and then down below where we're using it you can see that hey great we don't have to pass the execution context explicitly in fact we don't even have to declare that it depends on it because async off the type has sort of captured that knowledge for us of course now you know if we want to this this whole thing got moved into a for for yield comprehension so our future monad has to also be you know it has its own for yields comprehension that we have to then get the result of and you know so but we can take our implicit execution context and great now flatmap for future works inside that block as well and then when we want to call it later we just pass our execution context and passes it all the way through there's a little bit of knowing that we have to cut you know put EC here but that's not a big deal it's kind of nice but we actually didn't implement everything that we did from before we actually had the IOT part that I left out so if we add that back in it gets a little bit more complicated we have to first off our add method in order to keep it in the same for yield comprehension it has to take the same context as the report results one which report results needs the IOT in addition execution context so you have some coupling that maybe you didn't want to have there but you know for the sake of this demonstration you can just we can just bite the bullet there and say that's fine and then now you get that monadic for you can call add and after you've calculated your result you have to get out of the sort of future monad and get into back into the reader monad and that's where this ace you have to call async again here and take the arguments explicitly report the results it looks kind of messy we've now we have you know a lot of explicit arguments because you can't declare a single argument or a tuple of arguments as implicit in Scala - so we have to make them explicit now refer them here to pass it explicitly here you know again for the async that we define and then we have to make it explicit here so you get to the end and okay maybe I use the wrong abstraction for this so let's see what other advantages and disadvantages here it's great that we can thread the arguments through the code and you can use various tools to like lift this abstraction into more interesting abstractions like you can use either T was talked about where that's sort of you know it I think the the model for using monads in is better for other things but for context maybe not the greatest model for that and if you add an argument it could kind of ripple throughout the code so you you know changing what is required in your context can be kind of difficult and you know there's a little bit of a performance cost with closures being created and if you're going to use an external library that may be another factor that is important to you if you care about that alright so this is probably what you're most familiar with using an implicit parameters you know you take your implicit here and they're threaded through the code for you of course you know there's a downside you have to repeat the name of these parameters everywhere that you want this implicit in scope you know this the so that a downside I guess but you know it's similar to similar to taking explicit functions it's just sometimes you don't have to be you don't have to name them so alright sometimes you don't have to pass them in where they're needed again you know we have to declare these things as implicit but we no longer have to do we no longer have to do a lot of the sort of footwork of getting the right values into the places that are needed so all right yep so I there's a good reason why I think this was chosen for the future library it's you know it's sufficient it you know and and if you're defining all the places where you need the execution context there's not really much of a problem with that you know there's there's a bit of verbosity and boilerplate miss when it comes to declaring that you need these implicit argument lists but for the most part all this gets threaded through the code efficiently and you don't have to depend on external libraries to do it okay so what is implicit functions give you so this is in this case you know we we now are returning an implicit function that takes our context and produces some results so an async context is something that takes an implicit execution context and returns a future of T and our iot context is something that or our I Oh T function something that takes a tea and returns a tea so now our ad just returns future of X plus y and because we have an implicit execution context in scope we don't have to name EC we just it's it's assumed that that's what we're going to use it for and when we report the results we return an async context Iowa coyote context of unit and we have both the IOT and the execution context and scope so sorry when we call it we can you know it works the same as the implicit parameters form all these arguments are threaded through implicit but we don't have to declare their names at the top anymore we don't have to like assign them to implicit and I don't know if you noticed it but I actually swapped the order of this so that you know ones returning async context IOT context of unit and the other ones returning IOT context async context of int right because we're returning the results at the end here but it doesn't really matter because the Scala compiler is still figures out that okay well we have an implicit iot and we have an implicit execution context so that's all this thing needs it down here so we're good and you'll notice we didn't declare any unneeded variable names anywhere and if we wanted to add another context you know it propagates throughout the code and we'll find all the places where you don't have the implicit context and scope so this seems like a pretty good fit the only downside you have to upgrade to Scala 3 I don't know how hard that'll be for anyone but yeah ok so now I get to do the demo what I wanted to focus on what the demo was to try and utilize this kind of request context idea in a play like demo project and try to solve some common use cases like authentication requests may be tracing requests between servers so one common pattern we'll use for that is where we where I work we use correlation ID so we log everything with this ID and then in Splunk we can kind of click on that ID and get all the requests that relate to that that request and actually we can trace between servers as if they pass that correlation ID to other servers via headers and we can also use it for logging so maybe we want to have consistent Logs with like everything from the request that we care about maybe the user ID or the correlation ID and other things all logged consistently and we want to know like at compile time that this this is safe so and I also want to retain some the familiar patterns that you've seen in play and i lastly i'll show that you know you can take this context idea and and safely break it up into separate libraries if again alright so there we go might be cranking my neck a little bit here move this up to the other side alright so we'll start with let's start with just a simple example here returning 200 so action builder if you're familiar with play this is you know you're gonna provide a function that returns a response either a sink or sink turns out with implicit functions you don't need to really have two separate methods anymore instead what I do is I just have a responder of our sudden return type and a responder of future of response is easy to implement just as a responder of response is easy to implement and because this is an implicit function if you want you can declare that you need the request here explicitly or not you can just leave it out either way it works fine all right so let's run this make sure it works okay so let's go to our main function where we're running it all I'm doing right now is constructing the app so if I can there we go so I'm going to demo let's build this request here and we're gonna import from our app okay now controllers gives us access to our example controller and then we can return okay I'm gonna call handle with our request I've also implemented this like little dummy thing where I can force something to execute synchronously with this dollar sign don't do that but just for example purposes so we'll just say right now when we run it we should see we're getting a 200 Joule alright so next I want to kind of show okay well where did this give us so inside here we have access to the requests in context so we can say request out here dot path and say and we can do this because requests out here it takes you know sorry this this is a pattern that I used in a lot of companion objects so what I did is I made it a trait where you can sort of extend this trade in your companion object and it'll give you two things for free the ability to say I want this context to something and that's an implicit function from the context to X and that also I want the context that's at this location here so you'll see me use here here a lot and that's what that means so when I say request here it's just pulling it out of implicit scope and getting the path pretty straightforward all right so next we can show we can see the path down here can everyone see this all right okay so next we can show all right well let's say we want to add authentication to this so we'll have an authenticated request and what we'll do is we'll actually refine our action to things that are authenticated you know actually sorry first things first let's let me show this part here so you know you probably won't just want a request by default you might want something that is your own custom flavor of it so for example created this request context which is extends this play request context which gives you just the request by itself they could all show that here there so it's just a trait that takes a single val request that's another feature in Scala 3 you can have tree parameters in the sort of where the constructor would be in addition we want to extend this trace context so here you can have you know the correlation ID as I mentioned before you want to log this with everything and maybe we find a user ID in it or not so this will this is something else we can define here so in order to create this thing we need to be able to refine this request context from our request so here's an example of how we do that we this context refiner dot sync all it's doing is saying okay well I'm gonna take some function implicit function of A to B and I'll define a way to refine it the context refiner also show this here it's essentially like a function from your original context to a future of either an early response or your refined context and so here what we're doing is we're just it's a convenient method for building it in which it's never going to be an early response and it's never going to be in the future it's just going to future not successful wrap this thing and a right and then you know it's something you can refine every time so it's gonna just build our base request context and this gives us our correlation ID from the headers and our user ID from a header as well this is not a good way to get the user ID but we'll do that for now okay so so now we've changed this dependency here obviously we can print the request path but we should also be able to print request context that here dot correlation ID of course we have to change how we build this controller so let me go to my controller module and now instead of passing a regular play-action builder I'm gonna pass this requests context action builder and you notice the way I did that is I just called refined on this play action builder and refined it's just something that says okay give me a refiner and I'll give you a new action builder with that new refined type and the refiner will either get the next thing that it needs in this case the base context and then call the next refiner with that to then get a future either of response or the new context C and of course if the first refiner fails and we get a response an early response we just returned that and we carry everything else along so pretty easy to implement that and then you get your new action builder okay so let's try running this now I should see both yeah so we see both the path and the correlation ID it's just randomly generated but if we printed it twice we would it's the same value all right so let me go in back to the example I was gonna do here so let's say we want to refine the request even further let me say we want an authenticated request so you can have requests that are unauthenticated and they still have a correlation ID but you might also want to have authenticated requests that have everything that an unauthenticated request has but also an authenticated user ID I'm going to find our handle and yeah so pretty easy definition we're just going to return response 200 and see oh wait we're gonna call this thing so let's go to main and we'll take this here call it result - and call up it edge it them authenticated here okay we got oh wait we got a 401 was that well it looks like our authenticated context the way that that is refined from the request context is by looking at the maybe user ID and maybe user ID from the request context is received from the headers and our requests didn't have any user ID header in it so we go back over to main and let's add that header so we just need a map we're gonna add a user ID and we'll say these are ideas one for now alright so now let's try it okay cool we got a 200 pretty cool all right so we've refined our context and we can also print our off context here authenticated user ID nice I'm gonna skip ahead though because I want to get to the end of this it's likewise we can define our own our own responders for stuff so some you know maybe you find that you just want to return a string so we'll just have this example here where maybe we just want to say if we return a string it should just be a 200 with this in the body right now you'll see that just as you would expect tells you that hey there's no implicit argument of responder for string so let's go ahead and define one so what we do is we'll just go we'll just put it right here okay same thing as before sink just assumes that we're gonna return a response synchronously and that there's not really going to be an error we need to handle there we go so now we see that the error is gone away and we should be able to call this from me so we'll take this here and I'm down to one minute so I'm actually going to skip ahead and show one last example that I want to get to which is demoing how we can do the logging so let's say we have a logger we'd take this in our arguments list up here didn't really go over this but essentially an app logger is something that will take this context you know if it's laga Balazs a map then it will convert it to a map and put it into the log message in a formatted way and of course now when we call logger info it's going to look for the request context or some context that it can log and we'll put that into the log message it's nice about this as you know at compile time that this is safe assuming since the type checks will just go ahead and say that that's that's good let me go add it to main so we can see the long messenger what that looks like oh yeah oops we got to create a new controller Oh No there we call it from me it's yeah we'll run it okay so you can see the log message down there has the path the user ID the correlation ID all logged all right so yeah you can find this on github context and dottie he's gonna go over some of the advantages of thread locals I'm sure you guys are probably aware of why thread locals are bad fit for this I mean take away if you're going to add context you know it might grow horizontally and trimming down the size of what you want to pass in that context is easy to do if you if you have to like control over it whereas thread locals you're you're gonna pull in maybe a lot more stuff that then you need and then essentially if you want to break it into smaller components you know you could have your blue library or yellow library your red library and you know things that only depend on yellow library stuff can take a yellow context and so on and so forth at the end of the world you're constructing that top one and it is an everything below it but it doesn't actually the things below it that you know in libraries that are published separately don't have to know about that and that's pretty much it thanks I assume