Scale By The Bay 2021 : Ramnivas Laddad, Developing Declarative Backends

Scale By The Bay 2021 : Ramnivas Laddad, Developing Declarative Backends

Recording: Scale By The Bay 2021 : Ramnivas Laddad, Developing Declarative Backends

thanks sandeep for the introduction uh i'm gonna today talk about clayton it's a way to create back-ends in minutes and still keep the application flexible performance and production training so be this this talk has essentially three sections the why what and how so in what why we will talk about the need uh in what the exactly what we offer and how is we're going to talk about how do we build it how did we build it so this is a project that will be open source at some point so just so we'll talk about why part so why do we need a backend that can be created in minutes so if you look at current state of the world we have clients such as services tools but also mobile application desktop applications and so on and but we have the information in some backend some databases such as postgres as well as maybe rest api available or graphql will be available to you and a millions of cloud services um search engines mail gun things like that so what we do is we create something called backend and the backend offers some api maybe graphql api or rest api and the clients talk to these through this api to the backend and back in in turn talks to the data sources the way currently however when implement the backend is we typically write some sort of data access layer which knows how to talk to the rest services or postgres and other s3 and so on it then we introduce something called service layer and service layer usually would uh take care of things like um security taking the data manipulating it probably talking to multiple services and bringing that data together and so on and finally we have api layer and api layers job is to expose that data through appropriate apis so for graphql we'll write resolvers for rest api we will write controllers here now the details may vary but essentially you have to take care of all these things accessing data combining them together applying security and exposing it the way we right now do is we end up writing lots and lots of code for doing the same thing and even if you look at from one back into other back end there's a whole lot of commonality between them so what we are proposing instead is to have a declarative backend playtip and i will talk about what exactly it is in a minute and that way we don't have to write all that much code so what is clayton well clear tip is a declarative language so it allows you to express domain model access control services business logic and system word concerns in a in a special language that makes it very easy for anybody with some programming background to be able to do that it's an interpreter and compiler so that language uh during development mode you can interpret it and then you can get your back end but when you are finally ready to ship it you will compile it and you will get an optimized binary that you can deploy to any environment and there's also suite of production tools which allows you to for example deal with database database schema migration and so on so next five slides i'm going to swap between showing a slide and giving a demo so first thing we need to when we want to create a backend is to deal with domain model so in case this domain model we're going to come with very simple domain model so we want to do concept management so concepts are held at a venue and when you can host multiple concepts so let's let's start a demo so what i'm going to do is i have essentially blank slate this auth code that i will use it later to save time but right now we are going to create a file called index dot play and now i can write my model so i need to express my domain entities so concept is the one and it's a primary key and i want it to auto increment then i want it to have title and then i will say is it a published concept or not so that allows me to work on the concept details and not have to publish it that's it so i wrote file and off code and let's see what what can i do with this uh declarative backend so i just start the server called clay and say serve so basically serve the back end and it started the back end in about a millisecond and then if i go to this thing i get graphql api so let's observe what is it so you have query let me probably pump font so we get to basically get a concept given an id or a bunch of concepts given a filter and ordering and so on we also get bunch of mutations creating deleting updating concept but what we are interested in we are interested in further connection between the concept and menu so let's continue with the domain model and we're going to say model menu id same thing and it's a primary key auto increment and we're going to have name of the menu and also published here so we have now model and if i were to go to graphql now let me actually start in watch mode so we'll pick up the changes as we do and if i go here i will get here queries for venues and similarly mutations for concept and mutations all right so what i'm going to do now is i want to create a database based on these parameters so what i'm doing so it basically gives me oh sorry actually i need to do one more thing i need to create here a connection between the two let's say when you column and basically concert has a venue and when you can be hosting multiple concepts all right so let's do this again so it now gives me uh tables to create and constraints to follow so let me just copy this go to another terminal and create a database and paste it all right so let's go back and start our placer so let's reload this nothing should change but now what we're going to do is we're going to start using those queries so first thing let's try to get what are the winners in the system id name published and we execute it and we get back nothing because obviously we just create a database and we didn't really add any venues so there are no venues in the system so what i'm going to do is i'm going to start invoking some mutations so let's create first venue when you you can give data so this is just a graph automatically created graphql api so let's call it v1 when you want publish let's make it publish true and get back its id so i created a venue let's create couple more venues so v2 and v3 but v3 we are going to make it published false so similarly let's create a few concepts and connect them to [Music] database connect them to a venue so i am going to create here a concept i can set next title let's call it c1 publish sure and which winning we want to host at the ide one so we're going to create some symmetric data so when you constant one is hosted inside we can do it in one shot let's do that at the same time so i'm going to create concept one concept two in concert three that says this is c two c [Music] three make it win two venue three we make it false and we have to do different names to each of these so two and then three it's just the graphql allows you to run multiple mutations in one shot so we are going to just use that and so we created three concepts so now if i go back to my query so in fact let's just write so we i want to know all the concepts in the system i want to know their ids i want to make a title i want to know their publish i want to know what we use there to stay at id and name and are they published and if i want i can further there's no problem you can ask deeply nest as much as you want so if i execute it now i get back all the data so i get c1 being hosted at v1 c2 being hosted at v2 and c3 which is not a published concept it is uh also is hosted at v3 so what we have so far is let's see how many line of code we wrote we wrote like 13 line of code and we have full-fledged graphql api available that allows you to do all the creation deletion update etc and still deal with the relation between objects and so on but as you can see um this is not the api you will directly offer because you don't have any access control anyone can add venue anyone can delete menu etc so what we want is we want some way to secure your model so securing this model uh what we need is we need to first describe what does it mean that somebody is accessing the system so that concept is a context so basically context is anything that you can create it from request so this is part of the flexibility so you can create it from header request anything so we are going to create it from jwt token so i want here id which is an int and i want to say it comes from jwt uh subject field similarly i want to say it has a role and it comes from getting close last thing i want to do is i want to say who can access these objects so if you want to query you can access it as long as if the concept is published or you are in admin role or sorry if you want who can mutate it it's almost got right okay so you have this access control that says anyone can query it if the object is published or the rule is admin mutation however requires that you must be in admin row so i'm going to apply the same rule to benny and now let's go back to our graphql and let's run the same query now instead of getting three concerts back you got only two concerts back that's because our rule is that if i'm an anonymous user or a non-admin user i should not get back published concerts so this portion so what i'm going to do is i am going to add uh i'm going to add an authorization and i already have created agility token prior to save time and this token basically is uh has an admin role now if i execute the same command i get back everything similarly if i go back to my history and say okay i created venue here so if i did not specify this it would say you are not authorized to create it but if i specify this token which is admin privilege i can go ahead and create the venue so now we have our secured our model and we can specify complex rule if you want the next thing is a data model will get you uh quite far uh that's really the major part of your domain modeling is like data and its relationship between the objects but what if you want to go and you will have to go to for example strive to buy a concert or send an email or even to authenticate with google you have to go back and verify the token all these things require that you access external services so we're going to start writing that external service so i am going to create a simple example not quite strike but almost giving the same flavor so i'm going to create a service and let's call it actually what i'm going to do is so there is this service service called json placeholder so it's basically you can go to url and it will give you a to-do item so we want to basically make access to such service possible from within clinton so i am going to say here are to do services and i'm going to have a query and that query will be who gets an id given an id it's going to return but if i save this um so it says like by the way this is the whole language enclave is all type safe so you cannot make mistake and it will not catch you so for example if you were to say you published x it will say oh i don't know this field or if you said here title for example or name so it will tell you that ux this should have been a boolean but you gave me a string anyway going back to our thing published so tudo is not a known to the system so we're going to let it know that true is what we want is basically this payload representing this payload so this almost looks fine to me except i think this should be tighter there is a user id but we don't have to get every data this code compression is by co-pilot github co-pilot it gets almost ready but not always so now it's saying next is all right this query but what does this even mean to execute this query so we have to supply here an external annotation to say okay the tool dot typescript right now we are supporting typescript but uh and javascript but we will support web assembly and other other languages so that you can write your services in all right so let's create this to do the rts what i need to do is i need to write a function to do exactly what the query my specified so it takes a parameter number okay this should be an async function but i am going to do a fetch promise i'm not going to as nice but so what i'm going to do however is i'm going to say let her equal to fetch and i'm going to copy this url and then return yeah r.json all right and let's start the service so now if i go back to my service and reload it and see what are the queries available i get an extra query called tool let's invoke this query id and what can i get from it i can get all the things i described that guru may have tried to complete and i got back data so similarly you can extend it to any other service you can fetch it from stripe you can post it to any other service get blob from s3 or whatever else you want to support so the next thing we want to talk about is okay so external services are actually let me show you one external and additional external service and that's that uh using auth so i'm just to save time i'm going to actually just uh copy paste an existing set of code and should be no surprise to you except for one small thing which i will explain so i want to add authentication to my system and simple authentication is email username password that sort of thing so what i have is i introduce a model called user which has as you expect a pretty standard stuff in a realistic model you will have verified is it locked out account and so on but roughly it matches what you would typically do and the query and sign up look very much like the query here except one is query other is mutation but there is also this new thing called inject so sometimes services need to access back the data itself for example in case of auth if you want to do a sign up you want to go and execute query to add this user to the database or if you wanted to log in then you want to get the password and then do a comparison of the password and then return a jwt token so all these things need uh accessing data and that's available through this creative object so if you look at the api the injector parameter will not be seen in the api so login you will see email password but not created because is supplied by the system you also have so okay so you have business logic where you want to say rather than invoking directly login sign up get me to you may also have situations where when certain things happen in your system you want to take some additional action so for example when a receipt is created as a result of strike checkout you want to invoke mailchimp to send an email saying you have bought ticket for concert and see you there stuff like that different flavor of this is you want to apply similar logic to a part of your application uh code for example you may want to say okay this stripe checkout receipt and sending email is a critical part of your system this is where money is made so you might want to say let me audit that system so let me apply some logic when you invoke a stripe call or send an email or receipt is created let me put that information into some auditable format or you may want to say that i want to check observability of this system so if exception happening too many too often or if time is much longer for each query then i want some observability so that i can get other reading and things like that or i might want to apply rate limiting to the whole system so that if i get the uh the website get abused then i can pause those users and not serve them for a while so we are going to create this observability uh [Music] concern through a concept of interceptor so interrupt sectors look very much like a service in fact they start as a service so let's say timing service so we are going to instead of writing a query or mutation we are going to say i want to write an interceptor and then you can see ah interceptor gets access to this special variables called operation and operation basically contains information about the name of the operation that has happened and so on so if i save it now it says okay i need to have an external annotation let's call it and this time just to show a different thing i'm going to do a javascript and it says you are missing before after round annotation for interceptor time because interceptor you have to decide when do you want to apply do you want to do it before operation or after the operation has completed or in our case we want to do it around it so that we can get time before time after and report that time so let's do that around i'm going to go around any query in the system let's create now a new file called time.js export and that's not what i want i want to say const start equal performance now operation i want to invoke after that so i can the way to do it is proceed and say and then i want to get correct that is also correct almost name is a function and then return present all right so let me start this server and let's do query so we are going to go and say get me all the concepts and it says okay concerts invocation took 49 milliseconds if i invoke again it will be much smaller because first time the query postgres uh will optimize it the next time one of us i will get much faster result so by just adding like seven eight line of code uh in typescript javascript [Music] a few more line of code in clay you got performance consideration also so the rest of the presentation is i'm going to talk about how we got about five minutes i'll i think i'll be okay so how does this whole thing happens right so we start with the model that you supplied so model is supplied has bunch of clay files and those clay files we pass it through a compiler and when i was doing clay serve i actually was passing through an interpreter but they do the same thing it first parses it then type checks it that's why you saw like when i said uh self.title or it said okay that's not type checking because it needs to be boolean and then it builds the model model basically has all the types all the queries mutations interceptor information everything is now figured out and then finally you have execution where you have query mutation plan like when the query comes what should be when a graphql query comes what should be sql query what should be access control rules applied based on the roles and user accessing it uh any external services or interceptor that needs to be involved and then this thing just runs so you don't there's no code generation as i mentioned and now you have access to graphql api and soon to the rest api all of this thing is written in trust so as a result uh we get pretty good performance so if you look at for example let's look at uh am i a little bit yeah so clay see it's taking 9.3 mb of uh memory so this is all this thing graphql api and everything available in just like about 10 mb of ram so this really really um shines rust shines in these things though resources extremely fast startup time so right now if i were to do i'll show the startup time point bit later and it's safe because in rust you cannot have memory uh dangling memory or crashes that are unexplainable so we are using rust and it's very happy with it so when it comes to deployment you can deploy to your local machine as you have been i have been doing or you can put it in a docker and deploy to any of the cloud services or you can do optimized serverless optimized build let me show you that part so if i do here play server it took about 76 milliseconds to start and most of the time was actually spent in is bundling these files but if you are doing serverless you do not want to spend this is you want as little as possible so how do you do that so we have this compiler clay build it basically creates this clay pot file and now you can just run it as a clay server and now it started in 0.77 milliseconds sometimes for the first time it takes a while but 0.77 if less than a millisecond our server started and this is really important for serverless because serverless the time to respond to user is the time to process but also time to start your server so now we have reduced it to below second millisecond and then you deploy to any of the functions so for tooling i already showed you graphql tooling but we also offer a bunch of other commands so the clay i showed you the build to build the optimized binary but and also show you schema create to take your model and create a database schema but there is a reverse possibility you can take existing schema and create a starter model and migrate to migrate database as your claim click model evolves a little bit more about clay build you can actually lock it down to say okay i want to only not support introspection i don't basically you reduce all the thing that you do not want in a final production build and it will create an extremely optimized binary and that's clearly right on time uh if you have any questions i will be out there in a special chat you