SF Scala at Twitter: Vladimir Kostyukov, A history of Finch

SF Scala at Twitter: Vladimir Kostyukov, A history of Finch

Recording: SF Scala at Twitter: Vladimir Kostyukov, A history of Finch

hello everybody um it's great to see so many people interested in finagle and projects around it um my name is Vladimir and this is the first time I'm speaking at a SE Scala but not the first time I'm talking about Finch so hopefully it will go well um you're probably wondering who the hell this guy well I'm the creator of the finch library and a full-time scull developer here t on a finagle team and to prevent your questions no I don't work on Finch full-time and now we don't use Finch internally at T Finch is just one of those pet projects we developers do in Spare Time for fun so if Finch fails for you don't blame me I don't make any money of it all right what this talk is about um I'm going to give you a brief overview of the history of Finch I'll show you its Evolution starting from the 40 lines gist file to field futured library that is already used in production by several companies and I'll mostly focus on the notable design decisions I and other contributors made and who am I kidding I'm going to talk about shitty things we did before shapeless so the current version is 0.6 um we are currently kind of in the middle of the journey to the first table version but it feels right now that we found a great puth that we make that will make it to 1.0 which will hopefully happen this summer so my talk is about where is it where is it from and where is it now and Travis Brown in the next talk will probably show you where is it going so and before we start let me ask you a few questions how many of you folks have heard about Finch before well not too much right all right um so and you de started on GitHub right probably you know what to do um so um almost an year ago I was working on rest API ban for a small startup and in the very beginning I wanted to use finagle for this since I just go to Twitter drop offer and was trying to get familiar with this Tech but I quickly realized that um bare finagle HTP abstractions are quite low level and rarely composable in the wild and I also didn't really want to use finat for since I was trying to stay close to the finagle bar metal API as much as possible I was mostly thinking about something immutable High composable simple and small I was trying I was trying sorry I was I was kind of trying to apply functional programming to finagle HTTP and build something actually useable it I something that an average scolar programmer who basically doesn't know the category Theory can efficiently use and more importantly have fun and before we start talking about Finch let's see how we can use finagle HTTP and the rest API setting and how many of you folks are currently running finagle HTP in production not too much all right interesting um so it's going to be like a new thing for you but I'll hope you you'll catch up so here's the typical finagle service or micros service if you will that emulates fetching a user by its identity and first of all we have to we have to fet a query string pram name which is an option so we have to provide a default value for this and this line looks okay for Java programmers maybe as scolar programmers we might want something better here something like a composable function that takes a Bram name and returns its value so we can reuse it and compose and not write this line every time so um we have to always we we also have to build response adjacent response in this case and so I love fagle a lot but I can say the same thing about HTV package so we wrote three lines of a mutable Java code to build a simple Json response um which is quite Rose and in Scala we will likely want something a polymorphic function that that builds an HP response dep depending on the argument type finally that's how we do rent nle HTTP we have this fancy roting DS cell available out of the box in finagle and overall this looks pretty good and pretty solid we extensively use scar features here like pattern matching to extract Road details but roting service is just a service that you basically can do nothing about it and we'll probably want something more flexible here something that we might we can compose two things together to be able to reuse them and like do nested things if needed so the bottom line is while it's totally possible to use finagle HTP in the rest API setting it doesn't really look like an idiomatic Scola style which promotes immutable and composable abstractions that easy to reason about it that's why I started working on Finch which is a scal tic version of the finagle HTTP Finch is aimed to make a life easier with finagle HTTP so if you're currently happy with fatra you probably don't need F Finch is for those who unhappy with finagle http so and the nice thing about Finch is it's not a framework but a library so you can incrementally integrate its Primitives which are just function and types into any existing finagle service Finch doesn't force you to follow any concrete Style on organizing your code base like placing everything in the controllers uh Frameworks usually do it this fin is totally up to you how to write your services and how to organize the code around them and fits this fits pretty nicely into any existing service so this gist file published an year ago is the first ever version of Finch and as for today Finch is still very young project but it is rapidly approaching the first stable version and this doesn't actually mean that you can't use Finch right now in fact most of the finch users has started to use it from its early versions Finch is doing really great job on making its a Pi changes smooth and most of the time invisible so don't be scared of trying the current version which is 0.6 updating to 1.0 won't break your code much so you're probably wondering why Finch um yes we love birds at Twitter but Finch is an outside project so I could pick any non bird name and in fact the birth name happened here accidentally I was thinking about a name that says both it's about finagle and it's about about something really small I was kind of hoping to build small number of abstractions on top of finagle HTP something like a tiny teeny finagle clone for a particular problem and I found this combination which is finagle plus inch and where is inch is unit of length PR small thinging actually and it turn out it's a bird so we have this beautiful B bird T logo right now um yeah by the way I have three stickers so yeah we are not as rich as VRA we have only three three stickers I'm going to exchange for questions in the end of the talk so be ready if you want one so um who is using Finch right now as for today four companies are officially using Finch in production and also I have some information from trusted sources that at least for other companies are currently developing their products and Finch but haven't launched it yet so if you somehow currently currently using Finch in production please make it to the adopter list we keep the contributors motivated so we can continue making Finch more awesome well um the official part is over right now and we are going to start looking at the code and I have to warn you I have like 40 slides with bloody scolar code we have to discuss tonight and probably the best way to survive a stalk is grab some beer and get a little bit drunk and don't try to catch all the detailes but focus on the like big picture sorry all right um the first abstraction that made it to Finch is an end point which is a composable rotor where the roting logic is encoded in partial function from HP request to service so um it exposed the only one method or else that takes two in points and composes them in terms of underlying partial functions so we can use it like this we GR similar end points together and then merge them into a single end point like root end point and this abstraction is pretty straightforward sit completely uses um the built-in finagle Ro in DSL although it's very powerful thing actually given there is an implicit conversion between between an end point and finagle services so we can do sort of nting points here and these things survive till these days uh while there is already a replacement and Finch for this cold Road combinators API some people still use Vis approach since it's just super simple and safe the syntax is very intuitive even for scull and newcomers and some time ago I got a feedback from a person who managed to write rest API server with Finch having zero just zero scull experience and this is pretty nice um another early obstruction is a monatic request which is reader mon over the HP request um it's basically a function from from an HP request to a future of something and the reasonable question here is why's future not try on option um well in the very beginning we wanted to make it look similar to regular finagle services so that we can sort of unify the API and this work great since the since the finagle Futures are quite lightweight given that we only deal with them with satisfied State and the overhead of this future is pretty low but it simplifies other things a lot so the first API version was quite verbos there was a pair of two readers required an optional for every possible output type and every possible request request part like query string param heer body and so on so we had like 40 readers in total and but now it's just like 10 and I'll show you later how we managed to reduce the API surface while maintaining the functionality um that's how we build the request here of type user using the Mantic API via VIA for comprehension so this request read two query string prams required pram ID and optional pram name and the basic usage pattern looks exactly the same we use finagle Services since the request returns future we simply chain them with flat map so it should be clear now that if the if the request here fails we will have a future exception in response just like from regular final service there also was a special case of request here called validation rule that didn't really lead anything useful uh but check the given PR bullan predicate um is true it returns an EMP to Satisfied Future if it's true or throw validation fail exception otherwise and this looks stupid but it's actually useful and here is how um we can use it in for for comprehension to compose it with other request readers and validate their output values and there is no restriction here we can chain any number of validation rules in the same for apprehension and they will be applied sequentially but bad news this thing is not reusable at all you will have to write a new validation rule every time you need it since the predicate is is a value not a function usually it's not reusable until it takes a function as an argument so but bad news there is a bit abstraction for this in the current version of inch that you'll discover later so remember how we build ajacent response in the three lines of java code well response Builder makes it much less verbos by providing a bunch of polymorphic methods that build HTP respon responses depending on the argument type so it supports three response types a Tex plan response Json response and and the response with no content and usually you don't speak to response Builder directly but to its concrete implement ation backed by HTP status code and and it looks pretty simple uh so we can treat responsibility as just regular function so we can just call it um but the question here is how can we support a new Json Library like Heron out should we overwrite an apply method on the response Builder to to support new Json type well actually there is a better solution for this a type class based encoding so we have a type class in code response that knows everything about how to convert um an arbitrary type A into a string and how to send it over the response body so uh we can replace all the polymorphic methods we had on a response Builder with just a single method for arbitrary type A for which there is also an instance of the cond response available in the scope so it will has all the information needed to to serialize any arbitrary type properly and that that's how we do Json and Finch for now it supports two popular Json libraries like argonut which is preferate for scalar projects like pure scalar projects and Jackson targeted to better Java compatibility but there is no limitation here you can with this approach in just a few lines of code you can define a support for a new calization library no matter if it's XML or Json or whatever so there is also opposite situation when we have to be able to read a content of HTTP request from the HTTP body and for this purpose we need some sort of string based body readers like required body and optional body and in order to convert a string into some meaningful type in your domain like user you would have to apply parsing function to to to to required reader required body reader which sounds okay but we can do better with type class based request decoding so we have a type class the code request that knows how to convert a string into arbitrary type A so and it also returns try so we can handle all the errors properly um so we can pass an instance sorry we can pass an instance of decode request type class into a body readers so they would know how to decode how to decode an HTTP body directly to required type and no surprise here that's how we do Json decord and Finch uh these two different libraries and once again there is no limitation here you can decode anything you want as long as there is an instance of decode request available in the scope so um sorry in that moment we realized that we can also apply the same approach to other things not just HTP bodies like query string Pam heers and so on so we can use the same API pattern for everything which is quite nice so and to do so we did two two simple things first of all we provided implicit instances for all the output type Target type we wanted to support by default and second exposed this magic method s on a request reader on string based request reader so it can decode the underlying string into an arbitrary type and this dramatically simplified the API of request erors since we removed all the instances of request erors which have coded out put type and this how it looks before type class based decoding and this how it looks after and this how it looks today so we move towards simple and short names in the API which is kind of popular story for the functional libraries and the nice thing about type class based decoding is that it's very pluggable you can easily extend the functionality of all the request readers to support new output type by providing a corresponding instance of the decode request type class and in this example we Define a new decode request instance for for a type date time from J time library in just a few lines of code so we can now Define a request Z that fetches A J time interval from the two qu string Pam start and end and as we can see type classes fit very nicely into the problem of type save decoding and coding and we use them quite extensively in Finch so nowadays the Mantic API of the request real look like this and we've already discussed everything from here except for this default method that actually defined for optional readers to provide the default value for the underlying option and this looks good but the question is do we always need a mon for that kind of things well we actually no yeah uh while the API Power by language support looks nice we lack the ability to properly handle the errors from the request reest here so the nature of monets is fail fast so the request here compos with monatic API will return the only first appeared error which is a nightmare for front end developers but luckily we have a better thing than Finch right now which is which called applicative API so um it consists of both the composite type tier that wraps two values and the new compositor on the request Z till takes two request Z and composes them into a single reader that basically read two values in parallel and the API looks pretty nice due to scalar language support of the infix types but this PN marching here looks kind of ugly and the question if is can we do better here well what if I say that we can replace it just like this which is pretty short well and I have a good slide that explains how how it's possible yeah uh so it's not that simple actually but still an arrow combinator is allow us to treat a function of a type A till the B till the whatever uh as as a regular scalar function so um so what we do here is Define an implicit class like Arrow 26 on for every possible composite type we have and this class we Define a new method tilled that that is actually implemented in terms of map and which does P Marching for us and this looks pretty solid but we are now trying to move this Bo plate code even from Finch uh something like to deep down to the St to the libraries like shapeless and cats and that's basically what we're trying to do right now so the validation rule has also been Revisited in the recent releases we replaced that ugly version of the validation rule with backed by request of unit uh with this fancy obstruction that is basically a bullan predicate that is composable in in terms of logical or and logical and so we also exposed two new methods on the request re like should and should not so you can compose the request with validation role and by default Finch provide provides a bunch of reusable validation rules for both numbers and strings um and as you probably noticed the requestor API now looks like a tiny DSL request processing um which fits pretty nicely into the concept of functional libraries functional programming and here we Define two request readers that will fail with future of exception in case of unsuccessful validation speaking of the domain specific languages uh there is a road combinators API available on the current version of Finch and the concept of vdsl is somehow similar to the concept of Library scalar combinators or escod given a plenty of predefined rers that know how to handle only a single part of the road and you compose them with special functions called combinators to build complex things that solves your problem so and rotors always extract something from the road so it's basically a function from from an abstract road to an option of both remaining Road and the extracted value and all the built-in rotors might be divided into two groups um matching rotors that extracts nothing and extracting rotors that extract something for example rotors get and rotor star it's m are matching rotors they doesn't extract anything but much uh the given part of the road and two extrating rotors like int and string they extract int and string values correspondingly from the abtract road so the road combinator API is very intuitive and lightweight I don't think I have to commment this but there is one thing I want to talk about um in this example we compos two two rers with or or else combinator that basically says if the first rouer fails use the second one and this looks very powerful but it's actually not um it only works if the types of two rotors stand the same which doesn't really fit on the actual story we usually don't serve only users or tickets or something on the same API server we usually serve everything and in order to overcome this for now we force our users to unify their rotors to use the same output type which is which is finagle service so every router extract an HP service that actually does a job and this works pretty great uh and more importantly fits nicely into the concept of how finagle does it well in finagle HTP you usually deal with raw HTTP types directly VI finagle services from HTP request to HTP response and no big deal but these types mean nothing in your domain um when I say I want a I want a service that Returns the user by giving ID I'm thinking about service or function if you will from integer to user not about function from HTP request to HTP response um and so we lack all the type information behind this useless HTTP types which is kind of terrible because in Scala types actually matter and I think you would agree with me that types help us to better understand what's going on in the code and just compare those two functions from in to user and function from HTP request to HTP response it's clear that the first function Maps an integer value to to some user type but what does the second function do well who knows maybe reboots your server I don't know so what we that's basically what we're working right now uh we want to make Finch mode type safe by getting rid of those useless HTTP types and so our users our users can finally see actual types behind them and this should be possible with heterogeneous rotors bed by shapeless and I'm I hope service will give you some more details on this direction next talk but the idea basically is to allow users compose routers of different types so they will have a router for shapeless H list so also we want to allow our users to compose request serers and browers together so they will have this fancy DSL that looks like more like URL not Scala code which is very nice and finally we'll probably merge the functionality of both requester and RoR to to have sort of some sort of single abstraction for everything which is kind of like simplification so the bottom line is um f is where is the right tool for you if you wish to have uh highly composable functional Primitives aligned with finagle scalability and performance and I really like with twet from Travis it somehow explains um where is Finch right now it lied between with awesome scolar libraries and working really hard to to combine them in a nice way so here's the link to GitHub page uh feel free to join file an issue or even send a pool request and last thing I wanted to say is uh just a quick reminder I'm neither an expert in functional programming nor in rest apis I just made a yak shaving tool that I missed and then other people found it useful for their problems enjoyed the project so we kind of started working together to build something actually useful and nice and as for today a lot of th contributors made a real difference in the project and contributed the mission critical parts of the API so if you kind of want to make the same difference try it and leave a feedback and we'll work together to address it before 1.0 so thank you that's all I have do you have any questions do you have any strategies that you follow when or I mean maybe not you but the users of finished so far follow to build very complex interrelated systems using microservices so thinking in terms of batching of Collections and things like that for transmission on The Wire kind of thing yeah um so I would say that um you can do with Finch anything you want as long as you can do the same thing with finagle it's basically like doesn't um doesn't hide anything from finagle so it's it fits very nicely into the concept of how finagle does it so if you can do this batching thing with finagle you can do it with Finch it just like makes some things more nicely for you can I have this um here the first question ah yes sure whoa just two questions left two stickers I mean you have a really nice abstraction with the request reader um what about for HTTP servers using finagle that are talking to other HTTP servers so um like a a response reader or like a a reader writer state yes that's a nice that's a nice question thank you uh we have a ticket in in I believe on the next Milestone this that says it's it's your ticket right you have a sticker that's nice question yeah it's about consuming HTP services so we kind of hoping to build something like response read something like this something really nice something really composable that you can use yeah just one just one sticker so you you mentioned hello Yeah you mentioned that you differentiated Fringe from it's not a framework but a composable thing right uh do you restrict yourself to a subset of scholar features that you plan to use or um um yeah the problem is uh given the wide variety of features which is available in Scala to build composable uh yes so say do you restrict yourself in terms of code code quality of the features set of Scala to use it but not use that feature um um yeah we basically use I believe almost everything I hope yeah and now um we're kind of like we're kind of trying to integrate more with shapeless which which does every magic for us so basically everything that shapeless use we will use kind of the same pattern maybe trus can say yeah if I can hijack the question a little bit so we are using a scolar liting tool called W remover which does restrict um the kinds of things it it basically gives you either warnings or errors if you're using lots of different scholar language features so it's even it goes even further than um the language feature Flags in Scola 2 Tin or than the Scola lter itself so yes we are restricting the language quite a lot through use of right thank you here's your stier oh all right how simple are you planning to leave Finch cuz it seems like it's just really like some nice extractions on top of HTP but you also have integration with like a 2 for example um we does we don't really have it right now there is a ticket about this but there is a library which is called fagel off too I I brought it yeah and it's should be pretty easy to integrate it with off too if you're talking about this is it in your PL to um it bigger make it bigger no I don't think so we are not going to make it bigger it should be like super simple super like core obstruction that allows you to do basic things it's a library it's not a framework um there is one question there um what was the monospace font that you used in the presentation the monospace font yeah um it's it's a presentation engine called shower if you Google it shower jsjs it's kind of like default team I don't know I don't know exactly how I'm not I'm not giving a sticker just one sticker left guys other questions um right I probably give it to Ben thank you oh