FP Scala Meat & Potatoes: HTTP, JSON...

Scale By The Bay: Paul Snively, FP Scala Meat & Potatoes...

Recording: Scale By The Bay: Paul Snively, FP Scala Meat & Potatoes...

you so so it occurred to me this morning that and only this morning which you know is not a great not the most auspicious beginning that I'm gonna talk about a couple of libraries whose lead developers are here so if this is awesome it's because they're brilliant and if this still makes no sense by the time I'm done it's my fault so so I want to talk about as the description says on the website a lot of us still have to write basic crud rest api's we take JSON we store it in a database we look it up in a database a sequel database not even some fancy-schmancy new thing and we need to get it back out and does purely functional programming have anything to say about helping us accomplish that goal and and for that matter what is the goal what is sort of the point of purely functional programming in the first place and can we discuss that without going too deeply into some of the more abstract theoretical terminology around these subjects if you're interested in that sort of thing there have been and continue to be great presentations about those right here especially today and I'll mention some of them a bit later but first of all let's just start off by talking about good old HTTP and first of all I have to apologize I intended to do this live and unfortunately we ran into some issues with the tool set that I was using so throughout the entire presentation please keep in mind that you can type all of this into a repple and that's that's very much part of the point is that I want I want to emphasize and I want everyone to keep in mind that you can be at your keyboard typing this in interactively you do not have to have this sort of compiled deploy cycle that's actually one of the one of the advantages that I want to sort of call out but but the first thing we need to talk about is what is an HTTP service and the most fundamental intuition the simplest intuition the one that I want you will all to keep in mind every time you think HTTP service is as simple as this it's a function if you're a Scala programmer that's a perfectly fine function signature write an HTTP service takes a request and returns a response at the end of the day somehow there are a lot of details that need to be explained about the request and there are a lot of details that need to be explained about the response but ultimately that is what an HTTP service is it's a function that takes a request to a response so keep that firmly in mind because we're never going to deviate very far from that intuition as we as we develop a service as we develop a arrest service but let me quote my good friend and former colleague ruin RB Arneson don't lie with your types okay if you're a Scala developer you might already have some ideas about how this signature isn't exactly honest right it isn't exactly it is lying if you're actually developing an HTTP service for example first of all there can be some problem with what your service does in terms of talking to some other service on the network it can fail and this thing can throw an exception or something like that or you could try to parse the JSON you're expecting JSON in the request there might not be JSON in the request there might not be anything in the request so you can fail to parse what's in the request and have to return hopefully you're still going to return a response you're going to return an appropriate response but that's something that you have to keep in mind so there are some there are some constraints there are some respects in which this signature lies also the signature doesn't say anything about effects right it's just this signature claims that you just take a request and transform it to a response do some bit manipulation and and return the response and that's almost never true especially if you have to talk to a database somewhere in the middle so this is naive this is this is a little bit of a naive model and we need to elaborate on it ever so slightly so what do I have here the world's simplest service right forgive me I'm visually impaired I have to sort of lean over and stare at my own slides so using the HTTP 4's library we can very quickly and very easily develop a complete HTTP service and in fact there's a type called HTTP route so that actually does this elaboration that I just described around that function signature it is just a function right in this case it's a case expression reflecting the fact that we might want to do pattern matching on the request but it takes the request and it returns in this case a response that the DSL we've imported lets us build from just a status so no content is a status I forget what it is 204 something like that but this is in fact a complete HTTP service boom done right it's it's simple to the point of uselessness but it is in fact a complete HTTP HTTP service and I want to emphasize that because here we have the manifestation in HTTP for s of the slightly elaborated version of the signature that we just saw it's just a function it takes a request it transforms it to a response and we're done ok what what's what's one of the nice things about just having a function one of the nice things about just having a function is just plain old functions are easy to test and here we have an example where I say you know given given an incoming request actually I build a yeah given given an incoming request let me run this function over it and here I'm actually taking advantage of again a little bit of the elaboration that HTTP for us gives us for free with this or not found and I'll explain more why that's necessary and actually cool in just a few minutes but anyway here's just a little bit of a machinery here's a little bit of syntax you know for a comprehension around this elaborated function that is our service and then we get a response back from our function and then because we're actually returning this response in okay I'm gonna use the word right monad right effect monad we need to actually run the monad in order to get the result in and we would get the result which is you know a response that says not found not very interesting but it does underscore the fact that this is just a function and we can invoke the function with a little bit of syntactic overhead but actually to me it's just a little a little bit too much syntactic overhead so one of the things that HTTP 4s gives us out of the box is also a client you have both an HTTP service and an HTTP client library so one of the cool things you can do especially given the HTTP services say it again just a function is you can construct the client from the HTTP service you can if it's basically function composition under the hood so you can say hey client construct one of yourself from this HTTP HTTP service and then you can build a request and submit it just as you would to a real service somewhere on the network and that makes testing even even less noisy and I'm still not happy with that so still too much work so here is I think the most compact straightforward simple leveraging of the fact that you can build a client from a service submit a request again we're taking advantage of some some dsls that HDTV 4s provides and this is just a dirt simple way to simply write a function that is an HTTP service and then test it by constructing a client from it and submitting a request to it and seeing what you get okay but a no content response is not very interesting and ignoring the request is not very interesting so here we have what I call the lamest echo service ever right it actually does some pattern matching on the request to say you know is it a post is it opposed to the user endpoint and if it is opposed to the user endpoint I want to I want to parse the incoming JSON into a case class and in this case all I'm doing is I'm just returning what I get like I said the world's lamest echo service but there are some interesting things that hopefully warrant the claim that it's easy to use purely functional programming in HD HTTP for s and C RC to do JSON rest services first of all as you can see I have some imports here that basically say I want to automatically derive the encoder and decoder for my case class to and from JSON that's a little controversial you know some people feel very strongly about having some sort of other descriptor for the things that you're going to be serializing and deserializing and you shouldn't use automatic derivation that's a conversation for another day you can do it I would argue if you're only doing it for internal types between your micro services for example that's a justifiable use for automatic derivation I did have to say JSON encoder or sorry JSON of here in order to make the the encoder visible to HTTP for us so I have a I have a JSON encoder and decoder which knows nothing about HTTP and then I need the JSON of to say hey you know HTTP here's what's called an entity encoder so this can be a body of an HTTP request or response that's what that's for now there's already some interesting things going on here and just in just a few lines of code that I kind of want to call out one of them is this as you know this as a business where we take the requests and we say it's it should be a user right and the operative phrase there is should be because this operation can fail right how can it fail there are quite a few failure modes here some of the obvious ones are the the content type might not even be application slash JSON right you might not have valid JSON I might not even parse the json parser might choke on it it might be valid JSON but it might not actually parse into this user case class so the encoder could actually fail all of this is subsumed by this as operator what you get from the as operator is once again a monad sorry there's you know that's kind of what we're talking about the monad in question here is the cat's i/o effect monad and one of the things that I think we don't talk about enough when we talk about monads especially effect monads is the fact that most of them are failure monads they model failure as well as having some effect like talking to the network or you know talking to the disk or what have you they also model failure so that's also why I map over in the last line I map over this result before returning and hopefully with an okay status because it's just like mapping over an option you know when you have a value that might or might not exist same thing except here it's a more general notion of you may have succeeded or you may not it's also worth mentioning that HTTP for us here if the i/o has failed that failure will propagate out and you won't we return in okay this this looks like you're always going to return okay that's not true if any of the aforementioned things fail then HDTV for us will happily automatically return a status 500 for you instead of the okay so okay here is a little aspirational this this can fail you know it's it's a naive echo service but it can fail in multiple ways and HTTP 4s will do the right thing with with this simple code so one of the things I really want to emphasize throughout this discussion is just code the happy path and and when we talk about monads and error monads in particular one of the things that's really worth emphasizing is their fail fast so the first thing that fails here is what the response is actually going to reflect so for example if the request header content type header is not application JSON it's not even gonna bother trying to parse the body right it's it already knows it's not JSON sorry so that that error will propagate all the way out to the response automatically okay well parsing JSON converted it into a case class returning it I mean okay great lovely but what if we actually have to do something you know with the data like store it in a database or quería from a database this is some sim code using the wonderful Dubey library for talking to JDBC databases thanks to Rob Norris one of the two lead developers uh Ross Baker is the lead developer of HTTP for us so both of those gentlemen are here anyway here's just some plain old sequel right one of the things I really really appreciate about Dubey is that it doesn't hide the sequel from you far from it and in fact here here we write for better or for worse right depends upon how you feel about sequel but here we have some just plain old sequel queries against the database for our user and again some of the things I want to point out are there's this really nice sequel interpolate so that you can just write what looks like a string but it's actually not this actually gets compiled down to a I forget what the JDBC term is but sorry yes prepared statement thank you and it is actually typed safe if you screw up the types of these interpolated values here with respect to the types in the case class that they refer to this won't compile so there's a certain amount of protecting you from yourself that is also one of the things that I like about typed functional programming you know something I really appreciate the other thing I need to talk about here is describing what you expect in your result so for example when you get all users you'll notice that I asked for a stream and I think the other one is yeah looking up a specific user by ID and that were potential that returns an option that user might or might not exist right so we need to call that out also this all ultimately happens against this thing called a trans actor the trans actor just mediates the connection to the database it owns the connection to the database and it's what you actually do a transaction with which is reflected in these in these lines down at the bottom here transact against the trans actor and then do something hopefully helpful with the data so for example when you insert something you want to kind of want to get the ID back so we copy the the user and make sure that the copy has the ID that was generated when we inserted it or what's the other thing that we that we do with create user yeah we specify that we're going to want the ID back right when we we're going to generate the ID and we're gonna get that back from the query okay and I think another important thing to call out here is that notice that there's nothing about JSON or HTTP in here this is just the sequel stuff it's completely standalone it's completely isolated you can test it in isolation you can play with it in the repple you can do all that stuff completely apart from from HTTP great separation of concerns okay so now I apologize for the lack of white space here but this all does fit in slide so here what I've done is I've actually integrated imagine that this DB here is an instance of that class we just saw right so now we've actually integrated the interaction with the database into a couple of paths in our HTTP service doing exactly the same JSON manipulation that we did before but now we're actually inserting into the database and we're returning results in in the response and and we have the create and the the lookup of all the users and so on and so forth and this is actually a complete service obviously right it's it's taking JSON it's inserting it in the database or it's taking and it's it's taking a request to get all the users and it's looking all the users up in the database and it's returning them to the client in a very very very small amount of code that fits on one slide and again notice what I haven't written I haven't said a word about content types I haven't said a word about how many results there might or might not be when I ask for all the users it could be zero it could be a million who knows right by the way that is probably a little scary right when I have shown code to this to colleagues at previous employers the immediate feedback has been that's just dangerously naive right just ridiculously dangerously naive it's like it doesn't it doesn't handle it doesn't you know what happens when you have ten million users and all that kind of sort of thing it what happens when the connection to the database fails what happened you know all that sort of stuff and the answer is it does the right thing in particular I want to call out that when we asked for all the users if there are a million of them that's fine because we're actually getting an FS to stream you might have seen the scholars dead stream reboot presentation earlier you might have heard of FS two of previous previous events we actually get an FS to stream of users from the database and HTTP four s automatically turns an FS to stream into chunked and coded response we don't have to do that ourselves so this actually works in constant memory and it doesn't matter how many rows there are in the user table in the database those all get chunked out to the to the client by this service this is strictly speaking this is all we have to do to write a JSON REST API the talks to a sequel database there's really not much more to add now that might lead you to believe that HTTP 4s is very opinionated right there's only one way to do things and that's actually not true because it is built on top of cats and FS too so for example if you wanted to hop in the middle and do something different you know with any of these io monads that could fail for example you might want to say handle error with right there in the middle in case you needed to do some custom error handling you totally can do that if you wanted to oh I should I should talk a little bit about what's going on with this with this transformation on the on the requests all the users thing because we actually get back a stream of users from the database we need to jump through a couple more hoops to return that as a JSON array so that's that's what we do what's a JSON array we omit the square bracket and then we iterate over the stream of users we convert those to JSON we convert that to ASCII and stick a comma between them that's that intersperse operator comes from fs2 and it's nice it just says put this between every element of the stream as they come from the stream and then we omit the closing square brackets so that generates a legal JSON array from a stream of whatever in this case a stream of users and because we are constructing the text sort of manually then we do have to call out that yeah this is still application JSON please when you send the response back make sure the content type is is application Jason but this actually works and it works no matter how many users there are in the database and you know it just it does the right thing by default if your definition of the right thing for example is return of status 500 if the network goes for fluey okay hmm I want to talk about another really cool thing about the fact that HTTP for us models services as functions there's a dual relationship with client if you might have heard Greg file talk about duality in an earlier presentation this is one of my favorite things to do with this not only can we construct an HTTP client from a service but given an HDTV first client we can also turn it into a service they have a dual relationship it can go either way and that's awesome because if I am working in a micro service environment and some other teams somewhere else in the organization is not quite done with some service that I depend on and I need to mock out some functionality of that other team service this is actually all it takes I if I'm using the HTTP first client to talk to that service over there then I can turn that client into a service I can stick a little mock service in front of it with this tie fighter operator and that's another one of those funky functional programming things because what that means is I can do that because an HTTP service is a semigroup Kay and that's the that's addition ok let's just call that addition even with the TIE fighter thing you can add two HTTP services together it means exactly what you think it means let the first one take a crack at the request if it doesn't handle it let the next one take a crack at it so if I've got another team's incomplete service out there and that I'm I've got my client to I can turn it into a service I can put my mock service in front of it and then turn it back into a client right so now request to that client go to my mock service first if that mock service doesn't handle that they go off at and hit the network and go off to this other service and then as that other team implements more and more functionality I can take more and more functionality out of my mock service until I can remove my mock service completely and not do this client transformation at all but that two lines there is all it takes to do that okay actually serving stuff yeah again I'm really disappointed that I wasn't actually able to do all this in the repple as I had originally intended because it would have again underscored the fact that I have not actually run in a server right these are all just functions the HTTP service is a function the client is a function everything is a function but at some point at the end of the day you actually do have to presume you want to actually run a server nice thing about HTTP for us something that I know Ross is very proud of especially is that yeah you can run it in a servlet container if you're in an environment where if that's still necessary corporate environment for example or that's still necessary there's the blaze n io to implementation that a lot of people like to use when they're able to be in a completely self-contained environment very high performance ridiculously high performance and specific Tomcat and jetty implementations as well the ecosystem is pretty amazing Searcy is not the only supported JSON library practically all the popular JSON libraries are supported you have a couple of different metrics packages with drop wizard and Prometheus let me see what else oh yeah XML support because that actually is still a thing I believe it or not Scala tags and twirl templates for in case you don't just want to rest service but you actually have to generate some HTML and and ship that back all sorts of great resources about both HTTP for us and do be first of all Ross's presentation is at the top of the list and and Rob Norris has has done a couple of really awesome presentations about doobie and especially about how types work and in that context you know doing straight straight sequel but in a strongly typed way with with Evie and I wanted to leave plenty of time which I think I have for Q&A so thank you for coming probably hang on for the money back to resources yep absolutely [Music] yeah so so the question the question is sort of my take on some of the other more popular possibly frameworks in the Scala ecosystem like Skeletor ax and play these are these are great tools that are that were developed along a different set of of principles you know play is obviously the standard offering from light Bend and has evolved quite a bit since its inception so for example it started off as a sort of a full-stack application platform very strongly oriented around the traditional sort of templating and you know end-to-end generation and handling of the whole you know before the advent of rest api is really and single page applications and so on and so forth it's evolved to support all that much much much better and of course it's also evolved as the underlying infrastructure such as akka and akka HTTP have also so it's gotten much more modular and it's gotten in my mind much more appropriate to tackle some of the same things that you would use HTTP for us and Searcy for and and do be it's not purely functional and that is without going off on a very very large tangent there I'm committed to doing purely functional programming whether it's in Scala or Haskell or whatever I feel less strongly about that but the reason is that the the ability to do local algebraic reasoning about my code is the best way I know to write high quality code one of the things that I've tried to emphasize throughout this is that I've only coded up the happy path and that's all I need to code up by default that's extraordinarily unusual you didn't see any try-catch blocks you didn't see any sort of if this than that then you know you didn't see any sort of weird sort of control flow you saw some you saw some maps and you saw some four comprehensions right you saw as give me a user from this JSON that comes in from this request done that can fail that is captured by the i/o monad and then I just map over the i/o monad or you know whatever or oricon's try to construct a response using by just saying okay but I just I say okay around an i/o of something or a stream of something an FS to stream of something and that just works it does the right thing I don't have to sweat whether that stream is in a failed state or that IO is in a failed state did the json.parse correctly as the header correct I don't I don't have to care and all of these concerns are covered by the standard machinery of purely functional programming in Scala with the cats ecosystem that's not true of the other platforms how much of them is an artifact of just like nice frameworks that have air handling better in them versus like what functional programming range in team yeah what's the what's the difference between carefully designing api's in a non purely functional context right versus being in a purely functional context great question yeah you if you look at some of these other tools if you look at some of these other platforms you can see results that at least syntactically are very similar to this the difference I would say is that in this context you can continue to expand this use of failure monads and effect monads out and integrate more and more larger and larger increasingly complex components of your application in exactly the same way that the rest of this works so the way that I like to characterize this is I like to call it fractal understanding I can zoom in to any expression in any of these slides a single expression and say okay I understand completely how this works and when I say completely I do in fact mean I can reason about this using the Monad laws or the applicative laws or all that sort of stuff I didn't want to talk about that I'm trying to avoid talking about that in this presentation but I do in fact day to day rely on my ability to zoom into any level of detail in the code and say with at least four and preferably five nines assurance I know what this code is going to do before I run it and I can zoom out to any greater level of granularity that I want including the entire program and say exactly the same thing using the exact same reasoning tools that's also not true of the other platforms do you think it's possible or so how to build something like this over the apartment context when you are collaborating with a bunch of junior developers who don't have that fact letters it's it's certainly harder and and here's you know if I'm going to get company political at all I guess this is where I do it I I think I think as an industry I'm gonna say the technical term is we suck at internal education we we insist on either hiring cheap labor who we then expect to basically sit at their desk and grind out code and go home at the end of the day and if the end of the day is eight nine o'clock Soviet and the seniors are the architects who at some some by some magical unicorn process have achieved mastery of these kinds of things and they're responsible for you know this sort of stuff I've been very fortunate in my career I have to say to have worked for organizations like Intel media like Verizon labs like formation where the commitment and hurtle education has been manifest but intellectual honesty requires me to to admit that there is a learning curve around all of this and the the great thing now is my gosh there are you know we had the red book signing here earlier that's awesome Sam Halliday has written functional programming for mortals we have the architecture astronauts are sorry that the type astronauts guide to shapeless so there's less and less excuse from a resources perspective not to learn this sort of stuff and frankly my major goal here is to pull back from talking about things like monads and semigroup K even though I kind of had to mention both of those but to sort of pull back from the type theory pulled back from the Attic theory just emphasize that this is just a function the service is just a function you can sit there and type it into your ripple and pass an argument to it and get a result back that's what a function is we we and I'm as guilty as the next guy we've mystified functional procure functional programming so much and right yeah good question well first of all to two answers to that one you're absolutely right JDBC is blocking right so one of the things I didn't talk about here and should have is how you would actually shift execution of the database interaction code to another thread pool and there's there is an answer to that in the in the cat's effect context and in fact it's called context shift I didn't I didn't show that here it would add a few lines to the code wouldn't fit on a slide but it is important to actually do in practice the additional comment I'll make is please don't use the global execution context it's a it's a fixed thread pool with however many CPUs your box that has and you will starve threads if you use that to talk to JDBC so check out another library by another friend and former colleague of mine Chris Davenport there's a library called line backer that helps you manage your thread pools construct context shifts for them and send the work where it's appropriate to send the work [Music] today thank you [Applause]