funconf 2013, Tony Tam: Rapid REST API development with Swagger + Scala
Recording: funconf 2013, Tony Tam: Rapid REST API development with Swagger + Scala
now we can start okay my name is tony i work for reverb and i'm going to talk about swagger which is a description format for rest apis i'm going to talk three different ways that you can develop faster with it and take advantage of scala in the meantime so let me first give an explanation of what a swagger is for um this is specifically why like i used to have to maintain our apis the book that each guys in the documentation themselves and i had to use wordpress to mark them up which is just totally lame and the process of doing that meant things like this edit edit edit edit edit edit right so what inevitably happens and we've all done this is that you code faster than your document and that would mean that the documentation would fall behind the actual api and then people get unhappy when they couldn't figure out what the heck it was trying to do so we created swagger and what it solves is the following things here that the problem of documentation of rest apis that the server code is always ahead of the docs and that the community was typically the ones correcting us which is kind of embarrassing that people to understand what rest apis expose quickly do so via trial and error so you would make a request you'd prefer the response you'd assume that certain attributes in the in the json response were always going to be there then that they actually aren't sometimes right so there's the whole understanding of how to communicate with the nkis kind of get it open the next thing is that um apac clients so we would have to hand craft every client that we wanted to talk to an api so let's say we had a a special api we make for uh partner and i'd have to go create a new scholar client for them our new objectives new javascript client every time i add something new and sometimes those clients across different languages would be feature mismatched so we would update the python library but not the ruby one and then the ruby guys would be all upset they're always upset but they're upset about that driving test driving our api was hard so you'd have to write code to try it out and then test coverage so knowing what to actually test against it was difficult all right so those are the reasons i won't go into too much more um and does anyone here use swagger has anyone tried swagger some people good so what it is is a specification which is just a json representation of the rest api not to be confused for the fact that there's a framework that helps generate that automatically and it's all open source and powers a lot of different apis including some that i'll demonstrate in just a minute okay so just to make it super clear and this is a confusion because people think swagger they think of the user interface they think of the server implementation but it is simply a json spec for api and that's it and if you produce the json spec then you can be smarter compliant then you can take advantage of the tool set that has evolved around these so it looks very simple like this i'll go into more details in a minute if this monitor continues to work and once you have that specification produced whether it's on your server that actually serves the apis or not then you can take advantage of things like uh sandbox user interface for trying out the api a code generator which i'll walk through which is extremely powerful for creating clients and doing other magical things and then a javascript client which i'm not counting as a code generator for this dynamic but i'll demonstrate that as well and in the process of creating the framework for swagger we've our goal of making a server integration is it like i said the specification that json actually follows the code everything is in sync and if it's integrated you can take advantage of access-based filtering which means if there's super magical apis that only your admins should see with their api keys or their authentication then they'll be documented and if they don't have permission then they'll see maybe and um currently swagger on the server side is is supported in nine languages very much we support two of those um but the other point that i want to make is that because it's just json it doesn't really matter where it's hosted so for instance there's a company mulesoft which has a community called api hub and they just go right swagger specs for all kinds of you know big famous public api so this is the box api there's a facebook group that sort of thing and it doesn't really matter that they don't post it they're just going and describing it now that of course it's tedious to go and reverse engineer or read the documentation that they're trapped in that same cycle of how do you know what to call and what are the responses and that sort of thing but it doesn't get in the way of the deployment of your of your actual production system which is something that's helped us get a lot of enterprise adoption people don't want to touch the actual api service right now you know big company a so they actually host the documentation offline servers and then they can make calls and once you've produced the um the json to describe an api then you can take advantage of this this code generator that we've built that uses mustache templates and it will simply read the server's description and run it through a scholar-based code generator and then write out a different language so at the top right this is the same code the same api description is jerry scholar case classes below is java below that.net so if you produce an api and you don't want to have the hassle of learning all of the different languages that your clients need to talk to then you can use something like the code generator to produce these in in the native dialect and because it's it's mustache pretty much anyone can write templates and modify this to suit what your business actually needs and that's a powerful tool that i'll show in a second okay so that's it for my quasi powerpoint here what i want to do is walk you through what this actually looks like to use so some people were in here earlier ivan carrero gave a demonstration of skolakra and scolatra is a micro framework for rest or this microweb framework and at reverb we use it pretty much exclusively now and what i'm going to do is show you what that integration looks like for scotch and swagger so in this case here this is a simple api that is producing a couple of different uh operations um the the if you're not familiar this is the scala dsl for for representing a get request and uses ruby top style syntax for saying that pet id is an actual path parameter that's being passed in and api operation is is a is a mechanism that scholarship uses to describe what the actual api is going to produce and this is effectively all that is needed to produce a swagger spec so here this tells me that it's going to return a object type pet and it's going to use a nickname which i'll show more what that means a minute i'll get pep by id it's going to produce in the jsonis summary value for this operation which is called find pet by id and there are some notes in addition this takes a path param of type string called pet id which we see crashes here parameter has a description called id of the pet that needs to be fetched all right an indian here is going to then operate and receive a request it's going to extract the pet id and it's going to call this code and return or it's going to say email supply okay so what does that look like actually is it in operation so should start this first so now what's happening is melato is firing up embedded compile that code done it's going to start jenny onboard okay so what happened is i get the same called a resource listing which is the top level and the resource listing has links to um api declarations if i go to pet i now have this json representation of that api itself so scalotra has gone and built this out based on this dsl obviously if i go and change this and reload it looks like a strange mix of yamaha and jason the possessions of elements in the um in the json here yeah oh that's just my json plugin so that is just a dynamic update so the nice thing about now that i have i have this being represented dynamically and it's loaded from my actual server code um i can use the swagger u live which i have to click here and point to this url notice i'm running this on the file system and now i have a full interactive console for calling that api okay let's look at this a little bit more it's so small okay that's fine so what you see here is this is that that method that we just we were just looking at in squadron source code find pet bot id it's giving me only one option for a response content type to explain that in a minute it says that the response class is going to look something like this that there's a name which is a string tags is an array of objects of id and string there's a name inside that and if i execute this one then i should get the response back from the api in the console so that's super powerful the idea now that i can just go and write code i can get really clever about how these api operations parameters are built in the scholarly here's a post operation right the same thing happens and um and put operation they all get described directly in this user interface does that make sense any questions so far that work the same with the player root it's very similar to play there's a play integration as well okay so it is it would play yeah the play routes is a little bit um more decoupled like right here in the in the squad where you edit one file it kind of takes care of everything play has the actual api controller and then a route file that you have to do a little bit of wiring up i can display in a minute any other questions on this i going to know the response content good question okay so that was actually declared here as an object of type and the swagger code will introspect that object and let's see what that actually looks like this is what a pet looks like it just goes and introspects all of the fields inside here and for instance you see this one has a complex type of tags which is a list of listed that's right so if i look at the actual json which is a better way of looking at all this down and i see pet has these properties it has tags which is a type array with items of wrap and tab this is a json schema style response and then tag is an expanded right above here so it keeps one level deep hierarchy for expanding type of audio on the ui you can see the same thing if you look at the model itself so we can do this for lots of different apis people have found some interesting ways of describing one other thing i think that's really important to mention is that slicers while these are gets puts post delete it's trying to take that away from the developer and that you just kind of don't need to know a whole bunch about what the http method is and what's a path param what's a header heterogram of grams because in here this here is a body right that's going to put a body of type json which i can make really easy by tapping on it but sometimes there are options such as this site where some objects or some parameters are being sent that are headers right so this is an actual header we could set skip and limit you shouldn't necessarily need to know that you have to pass something as a header as a path primer free frame they're just options that get sent into the http request so one of the goals of swagger is to say like you can have a very restful syntax you can have the not very restful syntax however you build it as long as it can be described with these parameters and change and people understand it then they should be able to operate against it you shouldn't get in trouble because you have a post that you're operating on as a delete in your in your actual api code you shouldn't get your hands like that yeah the user just says oh i want to delete a pet for instance and whether it's put or delete or whatever it is http wise it should just work but it actually doesn't expose the header cell language or something like that that's right so you can you can you could for instance say that like i need to pass a um api key as a as a request header and then it will just pass through because as long as it's described it doesn't really matter so we look at some of these other people integrating with it what's interesting oh us government right so the nice thing here is that like i just don't need to know a whole bunch about how this thing works right i can just try out that guy i don't know what this is going to return contacts they haven't been slowing down i don't know that they have integrated with their server i look at the headers let's see what the response headers i think they strip out the response center so i don't know what kind of server this is but here we go no info right so you know they might not be integrating with this this might be running php it might be node.js so let's talk about um like what you can do once you know once you know what the description looks like um is there any functionality would like to integrate this with like buzzing or test suite to basically i mean i guess you're it's generating documentation based on your code i guess it kind of is self-enforcing yeah because you can um you can generate um test suites if you want because you know all the methods and all the operations you can the values and try passing in something that's not required is required or vice versa so uh is this always to be used with scala as the server no in fact that is the last one's insane here but most people don't use scholars so the biggest audience is java but there's a big audience on rails and node.js do they have to write scala code to define the schema no so the schema can be described a bunch of different ways let me actually walk through that and i'll show you i'll show you what i'm talking about so remembering that the whole idea here is that you just have to describe your api as json right i can go and do whatever i want in json and describe it and as long as some web server is serving this up then i should be able to access it with the swagger ui so let's do that here this is a hand crafted the swagger spec i'm going to just launch jenny to serve those up oh so in this case here all i need is for these things to be served up so if i look now on my local jettyweb server so now it's just serving up that json again you know you can change do whatever you want in here right like this this is the flat file this is the resource listing and this is the api configuration here so it doesn't matter what generates this json if it's scala code and that's great everyone likes that if it's node or ruby or some person coding this stuff in my hand then that's another option right so if to answer your question you don't need a scholar-based server to produce json the nice thing though is that once i have something represented in the api i can now generate clients against it so in here i have a uh i'm going to use a scala code again so i'm going to use a code generator which is going to read it's just going to read this json and it's going to generate me a scalar now this is simple because all that has to happen is something has to understand the description of your api just like just like an interface in in the java world right so once i understand that api then i should be able to encode a client so let's look at what i just did so it just wrote this code and the reason why it knows how to write this code is it just being perspective and read the json you put it in a structure that that's a mustache template is going to express and from that i wrote this all out so here it's reading this base path which is where the heat guide lives you can see from the actually this is a direct mapping here and it's going to go over each of the api operations such as add a meetup here this is going to go and write the client flip and do that so think of how much work this is when you have a really complicated you know this is a simple one there's only a few lines in it but once you have the ability to generate compliance then you should be able to very easily support a large number of languages without having teams so that simple json described enough for me to generate this ui and likewise to generate this is my favorite here because no one likes writing injectors we have support for scala in an async fashion so using like everything is in features and we have more traditional lobby type stall implementations java python php ruby javascript objective c actionscript dot net people you have people people still using it's called swaggers it's called uh so now here if i look at what this is generated from the objective c code okay all that garbage difference and all the models this is actually a lot hassle for you right this is not a new concept like we did not invent this concept like if you used wizzle or whistle or java and that stuff a long time ago it's very similar but there's one yeah but what i think the most important differences is this right i don't have to write the code to generate that right this is just a mustache and it's using scholarly as a templating engine um i know people have ripped out scallops i think it's too slow and replaced with j mustache and that's all fine but the idea is anyone can pretty much write a client in their own dialect their own style someone wrote a closure closure code generator i don't know any closure pascal even because if there's enough uh richness and understanding of the cause to write objective c then you can write it so for basically you need to respect white space so i assume the status respects why it's based on independent yeah it's kind of representation and everything it's really weird i don't actually know all the people for how it deals with indents and that sort of thing is there's a little bit of art that look good and you see some of it doesn't look in our generated yes a question about generating this with uh so i love the api out there our front-end guys are loving this but it uh the code you have to add to scalocho seems a little intrusive yes their way to hide that literally that scholarship just ran out of here yeah and so you don't take it yeah i gotta take this one well there's a couple of different ways so let me talk through some patterns in here okay so looking at um looking at this server it's actually something that he did that's very clever right so if you think back to anyone here use jersey jack's rs in jersey like everyone's used that right where the methods uh arguments in the method turn into arguments for your arrest requests you annotate them as a path frags a heterogram or something else body parents special sculpture is not as pink it kind of takes in it consumes the servlet and the serpent requests in it and it gets to do something with it so in this case here this operation like if i take this piece of code out i mean that's all that scotch really needs you don't define necessarily it's not bound to those parameters so we have to explicitly tell them we do the same thing with play as well like two integration so what we've done to make this easier is is usually on the parameter side we will check we will describe a parameter that's really common like it's uh skip and limit right skip and limit is used across like every one of our operations so we'll just we'll define something called um it's a query around so you basically define the whole skip parameter and then add it to this list of params in the in this in the the psychological definition so i just say here skip all right so that that's the pattern that we followed is if you have sometimes standard parameters and you just kind of i don't want to say mix them in but you add them in this builder style fashion too to the operation itself that's been the easiest way um i'll show you what it looks like in jack's rs which is a little tighter one there's some tighter binding too to how those um the parameters match to this wider spin actually i'll come back to that in a minute any other questions so now now that you've sort of extracted out the api do you think that maybe rest isn't the right over the wire protocol for this i think that there's a lot of ways that you can transition to better protocols by having a description of the api okay i guess it's also like do you ever think maybe maybe we should just use soap from the beginning can we say soap mic no i think soap is just if you actually go back if like i'm pretty old so i know some of those i remember some things like forba and stuff like that for communicating amongst servers that is offend one of the problems is that a spec like this like if you try to solve all problems then you get something like right you get something that doesn't really make anyone happy and so swagger is trying to keep a very tight and simple way to describe the interfaces right now soap got just bloated right and most people well this could be bloated in two years it could be right and so that's why it's being promoted by someone who has no commercial interest in it so we don't have to answer to um big company y wants us to add actually a very big database company that's up here asked us to have so asked us to add soap support for this i'm like we're not gonna like no we're not gonna add soap support it's gonna just turn it into a piece of junk right so i think as long as it's independent and fairly practical then it won't turn into so three or four it's it there's always that risk right so we have this interesting thing we basically first wrote json by fans put it in the engines and then let frodan guys play with this and we decided to be a contract front-end guys we'll look at the back-end guys right we'll have to start json right not produced from a server and going from there we'll develop uh the endpoint and yet what happened they took it and they keep their redundancy in model definition and they don't want random jigs they aren't baron so they actually redesign redefine the the javascript class and they factor out all the models into their own little javascript thing and then they include it and it produces the model so so how crucial is it to produce the final json with all of the models repeated do we really need to is that the definition can we accept like a sub javascript a format that will generate all the json is isn't generating javascript descriptions a drag i mean that's what you're getting at right json i'm sorry json descriptions of drag and i think it is i think that machines do a better job of generating json people right and so there's a bunch of different folks who have just been writing little markdown dsls to convert to write this and then who cares if it's redundant right so write it in markdown there's a spec called blueprint which is called apiary which is a pretty nice markdown syntax for describing apis so just turn it into json anyone can turn but if you include if you have to basically repeat the models in the current spec your traditional error and for instance yaml can include the yellow so if you would use the almost adjacent you have a standard way to include juanitas at the audio yeah right so that's like can you kind of include models in some ways yeah so i i don't uh i don't what you're getting at is if i have another api like let's say it's let's look at the pet store in the other model right that's right so if i look at the pip layer now there's a tag right there's all this stuff and it repeats there's a pet here and if i go to another description i think that it might be in here as well so to me i i would prefer that it's that there's more duplicity in across these different json descriptions then compactness that can cause confusion so i i don't mind that a repeat like order might be in here and in another api it's just tedious if you manually generate it which i think that there's a lot of work okay so so let's look at some something else that um you can do is and alexis painted so if you if you sit down and let's say that you've got a new product that you want to put out and you've got to describe this api and you have a consumer that's going to be objective c and you know you've got to write the server then my my suggestion is you should start by writing specs instead of writing code and by writing specs you can sit down with the person who's going to consume it and agree on what that contract looks like so if i know that these are the methods i'm going to get and we do this actually at reverb where we look at the models and the operations the front end guide looks at it before the back-end guy goes encodes it right this is a great way to kind of look at an overview of your api now using the the code gen and notice how it's not called client generator it's code generator i can actually just make a different set of templates that read that json and write out a server so the idea is if there's enough information and this kind of makes you start this is a little chicken in english but if there's enough information to to describe your api in json then there's also enough information to write your name on so what i just did is i ran another um another set of templates over and read that's stacked and it just generated right looks a lot like the other one right but just because we have all the information about the api i could just go and generate a stub server and i think it actually works now you could say well okay so you're going to do this once and then someone's going to make a change and it's going to blast over your code and everyone's going to be pissed off that could happen right the the right thing to do uh in my my opinion is to have the template actually um use a partial or extended trade or something like that that's easy stuff right saying that saying that this is my api class i can have it also right out um calling something called a meetup service right it's my job to implement right code generator can't generate your business logic not yet but it can do this and the nice thing about calling another tier like this is a service or mixing it in as a trade is that if the signature changes on the ink is going to break here it's going to give me a compiler if i add another another operation or another parameter here to to fetching meetups i want this thing to break i want the compiler to tell me hey there's an extra parameter on here someone's adding is in the future or some other parameter right this is all generated automatically from the project and the server runs this so this all got generated from that json what that means then is that the server guy can now go off and do this thing the client guide we can generate the client for the client uh consumers maybe you'll often do their the server guide and they'll do the implementation and stuff actually can be meshed together and that's what you get from an interface does that make sense some other things you can do um and currently no here's a java one um and this is just using java and jax rs is there an easy way to add uh multiple versions for your api so if you have like api v1 and then you move on to api two we still want to support your api v1 is there an easy way to do that i feel like overlapping upgrade generating oh virginia is hard okay i wish i had to imagine all that uh i think that there's um there's just different techniques and i don't think anyone is really settled on what the best one is um this is the job say the indentation is a little better yeah i i what we do is we we have separate servers actually support v1 and v2 and then our load balancer will route based on what the path is does that make sense so okay so that's that's um version and then using node.js version it's not hard right this is just running mustache templates and if you have a certain type of server that's not supported to write your own templates this different style that you use this node this server works and npr uses this same module that we wrote make sense so far okay and there is an initial problem yes um i'll show one last thing and then and then okay so as i mentioned on one of the slides javascript is a different it's different from our code generator for business dynamic let me show you what that looks like now we have a javascript library that's runs both in node or in the browser and it's the exact same library it uses a http library called shred instead of java jquery um you've got the right bar whenever you work in javascript which always feels weird ah okay so what i just did is i just started i started at a node raffle and i required this client library that made me introduced and then i pointed it to an api description and told it to build and what it does is it reads the description and reflects upon itself in those of all the functions that needs to so now if i need to call that let me call one of the locals is oh i swear so now by just reading that description i have all of the functions that were available from that api so i see find meetups was one of the methods that i described let's say i don't know what it actually i don't know how to call find meters so i can just say dot help and it just goes and tells me uh in a sort of title the title to meet up to search for tag the tag active should distract us only so now i can just call this uh but regardless i can call and call any of these apis directly without having to build a client and recompile it and just basically dynamically figure out what's running there and and from that you can now have there's a scholar server having javascript clients connecting to it or not net for that sort of thing from that single description um yes what about an api that has you know most apis have some sort of credential system yeah good question so um for this particular use case we're talking about uh well in general okay let's talk about um like various ways of signing the request yeah stuff like that so this this is an example here how in the description for the api it tells you how to make the oauth 2 request so it says what the grant types are in this case it does off code and implicit error token it tells you what what you need to call an order to make the request and then what the token is named so typically people hard code like if facebook then use this if twitter then use this so the idea is that you can describe the authorization scheme here and then from that this also has a api key type that is required so from that then there's enough description to programmatically code your client how to talk and once you get to oauth one and the five million different variants of it then it's a lot trickier because some of them require signing some of we don't there's there's no real standard lot too is pretty easy but that's the mechanism that's that's built there for people to express the authentication scheme and then like we're planning to uh to client support but other people are going to have to the community will have to write so you can write like custom ones that's right and even from the like the javascript console like i can just other questions yes or so you showed the dynamic uh not really code generation but whatever you want to call it discovery thing javascript is there any other languages that support that it should but we don't have one the rest of the clients client generators are all scattered this is the repo and all the different templates oh yeah so some of our ruby client generator just recently there's one other thing that you can do when you have all of this description which is pretty handy not everyone wants the dynamic try it now documentation so probably the simplest use case for for the swagger code generator is to write um static static documentation stuff so this is this is um generated from the code gym as well right so the simplest the simplest use case is actually to say okay well let's generate a set of static documentation so you don't need to just have a try it now interactive ui for for other questions have you done anything with uh testing your client-side or server-side testing uh we have done some things but we're getting a little lazy so i know that people have i think alexa has talked to us about it's our internal testing framework for scholarly it's basically csv based testing where you provide adjacent input to an endpoint and expected output and errors so essentially we will so we think we should we can add something like this to swagger the question is essential swagger defines the shape of the api it doesn't care about semantics uh the assumption is that if you have existing state it's very hard to recreate it but let's say in our test context we start with pristine database we know what we're sticking into the state and what we're getting out so for that case we probably should be able to define a sequence of json inputs and outputs so we structured the csv uh once we have some cycles i was thinking of basically somehow merging with swagger as a kind of amendment to the standard saying you know here's how you can give inputs to these endpoints here's how you can get out so maybe you know we can do it like it is an open source project so check out fire altera that's not what you think other questions so as i mentioned also there's a play to module that that is pretty popular there's samples in if you go to swagger or is it in github and then all of the actual artifacts live in uh made in central but there's the swagger plate two and then a whole host of samples about powder so different uh jax rs implementation brailles actually cxf a lot of people use scale south that's it for me unless there's other questions yeah a general question i can seem to be impressed that you know both john smith and scully the only person knows that i i know uh can you teach us a script basically i'm looking for a course jealous people scholar developers it's basically because you know it's hard to it's kind of a skull folks the series that they just do so but it's obviously useful um so i'm just wondering yeah here's my problem is i'm not good at either yeah i don't know of any good way to learn i'm actually working to get the documentation as good as possible so there's a bunch of stuff on here as far as how you integrate what the spec actually looks like and then the reasons why decisions have been made so like there's a there's a newer version of the spec coming out right now and i'm trying to make sure it's clear as to why you understand it's not just like a pissing match between two people and so we're uh typically in irc.net or swagger is the uh thanks you