SF Scala, Jason Swartz: Build quickly, Fail Faster & Deploy Automatically

Recording: SF Scala, Jason Swartz: Build quickly, Fail Faster & Deploy Automatically

all right so we're all set so um this talk is partly about apis and partly about scholar but I want to know something who uh who here is using schola at work all right good show of hands and who's writing apis as part of their word cool so who's writing Enterprise apis API is meant for internal customer customers awesome then this talk should be really well applicable because you folks are a great audience for this I want to talk about a kind of a case study of how we're able our team at Netflix was able to build some Enterprise apis and astonishing velocity and some of the best practices for for doing so I think Enterprise apis are great by this I mean API used by mostly internal folks maybe for some tools with uh specific parties but not really part of your your main site so um I think some of this is pretty applicable and what really struggling with that shouldn't have had that beer uh it's pretty applicable to different kinds of development but I want to talk about really how you can be really effective at this particular area so um this is another title for the slide this is how in 5 months two Netflix Engineers were able to build 150 well tested in well performing apis with again with 5 months and really no downtime so hi everyone I'm Jason Schwartz one of the organizers of this group and you can find me on all these places and I want to uh uh tell you a bit about what we did with this so I work at Netflix and if you guys have the site and uh have the service and I hope you do uh you've probably seen the main front page that we have where we have uh clients that run on all these different devices and talk or Main apis and that's all that's a that's really great I wish I could talk to that but I don't actually know how that works uh actually help the folks who put Netflix on their client devices so we have an internal tool called partner tools and we help these device manufacturers mostly in Asia some in America get Netflix onto their device and we certify it and test it we figure out exactly what it is and we put in unique identifiers and make sure that everyone who has Netflix is going to have a great experience even though it's running embedded on a crazy variety of systems so what we actually have is internal tools like this for example so our partners come to it they can do different parts of the process to get their devices ready we have tools for internal folks as well we can analyze we can say man this batch of Blu-ray players is having some quality issues or this TV is not able to show Ultra HD or 4K at this time so this is the kind of thing I'm talking about internal site used by some external folks some internal folks and backed by apis so really the setup's pretty simple we have an API server in databases who works on a on a system like this all right it's it's not actually that complicated we have a single sign on we have our API server and Scala we're actually importing stuff from kind of a more Legacy service built in Grails so we're kind of sharing a lot of the same systems sharing the long data but we're trying to say look we want to write things very clean very well accessible and just have everything available in Scola which will be at pretty soon so the core of this talk is given you're going to build this how do you build quickly and fail faster and deploy this stuff automatically so that you can do Enterprise apis with ease so these are the six areas I want to talk about today uh framework rest layers and tests deployments and docks I didn't rhyme at the last one but docs no it doesn't rhyme all right that's all right so let's let's talk about Frameworks so when you're choosing a framework to do this kind of API development there's a number of things you want to choose right and there actually are there's a good choice right now Scola has a pretty good ecosystem in this area so you're going to want to find something that you can download and get start out really quickly you want something that's not going to be a big hassle when you have to do a lot of fast development you're going to want something that's well I'll use the word integrated but something that works with your setup you know you need to make sure you know can this work with our ID can this work with our tools does it have to be on its own can we get it to work with everything we're currently using you're going to want something that's going to make sure it fits into your deployment scheme there are some servers which uh some build as War file so if you already have a war file deployment scheme that's great there's some which only run command line just another criteria for figuring out the right framework last one is well Enterprise apis usually don't need to be super fast or very scalable I am talking about Enterprise so something that's fast enough and fortunately most of the scholar Frameworks are do meet this criteria um we happen to choose scalatra I think this works out really well it's uh you know one we love uh you know standards and easy listening it's not really easy listening uh this is based on Sinatra which is a ruby um API server that has a very simple DSL the guys called it scalatra they keep it pretty well up to date and the folks that are uh continuing to add to this one nice thing is that it comes pre-integrated with some of the tools we already want to use so it has its own built-in features for scholar test so you can do use that to test your services has great oh doesn't actually say anything it has great support for Swagger which is actually developed by some of the same scalatra folks so this is built-in API documentation you add a few extra things to your method you run it and boom you have instant API documentation which is huge having this actually built in is a great reason to choose a framework so this is the most this is the very simplest scalatra um API call or endpoint that you can have you extend their serlent here's a post this here's the endo and then we're turning created any questions about this well again it's very simple of course you might wonder what's created you know well this is actually and I might test you guys on this this is an HTTP code uh 2011 you know if you have a post you say hey I created it and you return it back nice part is they have a wrapper for this you don't have to remember you know is this 2011 or 400 or 409 or formula 404 and if those are taken care of you can say say look hey I've created it I'm good I don't have to remember the code another example of what you may want to look for convenience in a framework um this is something a little bit more complicated but really not very much here's an authentication endpoint we expect to get your username and password we want to you know figure out if this is going to if you're actually authenticated and if you're not we'll return you unauthorized otherwise we'll say okay which in this is uh their shortcut for 200 sure here you go here's your token do you guys see any problems with the code here which is that hint that yeah there's a problem in here so the the problem I inserted is well what if you have bad uh input well you want to make sure that if you don't specify something correctly you can handle it so you typically should make sure that when you write again this is for enterprise apis we WR Enterprise apis check the input make sure that you return good error codes make sure that you have some kind of basic handling and that you have really a uh an API that follows the standards right what he says next I want to talk about rest is there anyone who doesn't know what rest apis are yeah you do really serious thinking about oh well it's HTTP codes is the big thing and action so if you can think get put post delete option and you could think in terms of returning the right action it's huge yeah all right so I'm talking about breast API specifically um there are newer there's you know there's more efficient formats but rest is pretty good enough um I get I clean this up a little bit here uh this is really the quotes that you have if you're writing an Enterprise app though I think you really only need these uh for the most part you should have something to say you know whoa your input is bad and you should have something to say okay things are fine you know if you're doing redirects or stuff there's and this is Enterprise there's probably some problem and you should probably make sure that you're not returning 500 you should have a pretty good idea so if you're getting errors you can clean it up I think 400 and 200 are pretty much the okay response codes for Enterprise apps so uh okay so I said HD put and post and delete and such so the actions the action that you do should follow what you're actually uh you have create and read an update and delete for database actions and you have the similar kind of thing for apis so when you do get you mean you're actually just trying to read and when you put put put you're just trying to update something that's already in the system so it's uh oh this is challenge went itm potent did I did I say that right I only say that at conferences but okay so this you can call it over and over again has the same effect you can create new items with post and what does this do yeah okay so that delete stuff okay so if you respect the codes if you follow the standards things work really well not just because it's not that hard to do this but people have a good idea of what's actually there um here's just a here's just a standard way a suggested way you could lay these out there's all kinds of way you could lay out your end points to say look how do you get access to these where do I put my items there but um most the experts agree if you put things in terms of resources and then even if you're doing actions put those actions in terms of resources so you can have a you can say look I want to publish these models or I want to approve this one uh model by ID for example or I want to search you know then it's very simple the key thing is when figuring out these Enterprise apis is you want to put very obvious endpoints because you want your user to find your you want your users to find them if you're writing services it's really no good you want to make sure that any service you write is going to get found I mean really this is your audience if they don't find your toy what was the reason in writing it pretty awesome you have a question about this slide oh okay it's UHS 80s oh is that how pronounce it I have I don't uh how do okay so right see I'm not going to try either but there are some folks who say look in your response when you link to other things when you give out ideas IDs you should include links so that other folks know where to get it you can do it it seems like a lot of work we don't do it because I'll show you the docs that we have which make it super easy to find the biggest thing is if you don't have good docs and you don't make it easy to find your apis you should include links to them so people get back a response and say hey here's an ID oh and here's a link I can now plug in to get it so I think that is a good fallback if you don't have a docs because you need some way for folks to find the the loot if you made so it's very good point sir um it's just that uh this stuff works out really well it becomes more obvious to folks and becomes more findable I highly recommend keeping to the rest not exactly but I do follow recommend following the rest standards yes sir so seems there's a between lastes it's a it seems like the standard around HP is to use the uh the method as sort of the ver and then the resource is it's true it's not a perfect example it could have been more of a publishing or more of a a resources uh it's hard that that you only have four actions though so if you want to change something you have put in post so I guess my question is do you find what's how do you see the tradeoff between does right it's not it's not reusable yeah and how would I for example publish model would I it's a little bit that's why I think you should yeah you should have docks that should make it clear exactly what's in there like you should have an inventory of what's available but yeah it should also be somewhat guessable so if you if you'd have to guess approve and think oh okay is it supposed to be approved or what should I look for that's a hard question this does actually do the approve and this does do the published but you do have to look it up to figure out is this like a push or a put or post for example uh there so there are folks who say look even this isn't very good it's you know kind of a hack Make This in terms of resources make it a a publishing and make this an approval and then you're getting in Pudding which I left as an exercise for the reader so you win all right that's a good catch have you have you tried both approaches and because as you point out build uh this seems to be clear enough but it is something the nice part of Enterprise apis is you could change things as long as you work it out with the folks using it it could make sense back to the puppy so I want no not the puppy uh so I want to talk about layers this part's super quick and this is more a matter of taste uh it's just about organizing the code I think it's really useful to make sure that you have your services cleanly set up this is how we happen to do it how I'll often do it where I have core service code that just deals with validation making sure you have the right information back that you're giving the right Response Code getting the output out and that's it and that you have some kind of manager layer that talks the database takes care of the business code and keeps it separate again you don't have to but this works out really well so if you want to have multiple services that reuse the same manager code it works out really well you're keeping the service part the endpoint uh completely uncoupled from The Core Business logic that you have again just a recommend Commendation if you want to go a little bit further it's also nice to say look why should I mix all my business logic with persistence code just having some this stuff kind of separated cleanly out there just means you're going to be able to iterate a lot quicker and make uh more reuse of the code that you have but uh you know this is It's Your you know new stuff has come to light this is your opinion uh okay number four is tests who likes to write tests all right you guys are awesome I like to write it I liked it because it makes a lot easier later on if I have tests and we mess something up or the data goes better or whatever happens we find out immediately we get big emails and frowny faces from other people we have tests that fail quickly that so that we can find out rather than having to find out down the line in production so the great thing is with test is uh you can trap a lot of issues that come up before anyone else knows you know how how little you're paying attention when you Cod it in the first place so there's a lot you can test with Services you know I do recommend testing the different levels out testing their interaction you should make sure that you're testing directly to your services but also to your managers for reusability you should do some kind of basic thing to say look when I send this to the service is it actually passing the right thing to the manager and when I'm calling the browser is it actually UNP parsing this Jason thing out here I'm just saying break this down into different areas test these individually you could have everything else working but it actually fails when you try to parse the Json for example and you know the opposite is actually true let's say I expected something to then be returned and I'm using options to wrap my Skol code and I didn't actually use an explicit type and I return it and your callers are saying why am I getting someone's name wrapped in some am I am I the only one who's done that okay so when you're throwing options out into your result and you you forgot that you're actually doing that it's good to actually verify what you're actually really getting back I think you guys are just kind of silent but like yeah I've thrown options into my data too okay um I don't want to labor that belabor that point too much uh this is just a nice example of why it's nice to have a test framework built into your API framework because you could then make it very easy to to do something like this so here's scalatra Suite this is Scala test plus plus scalatra I can send simply say look I'm going to be testing this serlet this is not the real path this is just a temporary path I want to say look okay I should be able to log in here so uh oh crap I put my password in there uh so when I do a post you know inside here it's actually calling it and getting the response back I should get a 200 from this and the actual uh text that's returned should be more than a 10 character token again it's very simple but you are then proving that your that your uh whole service works any questions about testing by the way thanks man and short you know it's nice just to be able to look at this and figure out one am I do I have my stuff right and two what is it actually supposed to do which is great for communication so it provides that ad seret part and it provides this post part so I'm saying post I'm saying do a post to this URL if it's po Json I can say here it's a Json header and here's the content but it has a very simple like DSL or function so I can post it and then inside I'm passing a function of course where I can get the status and the body return from the invocation does it do that's a good question I don't know I mean it registers it it calls it I don't know if it actually goes over HTTP but I am marshalling it so I can like Marshall Jason to text and then pass it in here and it'll unmarshal it back all right so um again I like to I like to make sure that I'm also testing stuff that should fail so make sure so the top one is what we call the happy path it's good to get that part done once you've got the happy path and you expect everything to work properly make sure that you got the air conditions too again this is just a nice way to make sure that I'm verifying yeah not only is it catching it but I have the correct error code the nice thing is if you have users who know Scala they can almost just look at the test and say okay I know what to expect back from your API all right second and last is deployment I find this part really great I get into deployment I'm kind of a build kind of guy um uh it's hard not Everyone likes to do that or it looks hard but if you do the right steps I think it's still pretty easy uh first thing I have to do is lose 100 PB and then I should be able to do that all right but no that's a lot of work so there's a lot of different deployment systems right now I recommend something that makes it very easy for you to deploy so you don't have to spend time doing it you don't want to have to work a lot on deploying because then you're not going to have time to build out the awesome number of apis and give a silly talk like this so uh this is just something we're using uh we we already have a system which takes the uh which takes the Deb in that Netflix so what we're doing from the start is we're using spt to manage our project we use the spbt package War thanks man uh we're building out a war file with uh James's plug-in we're taking that with Gradle and creating a dbn archive that says look I want you to put this on system with Tomcat here's the root. warf fall here's where everything goes we're then using Ami to then build on Amazon machine image and have this all go out there you know there's a lot oh yes sir sorry I we use both uh we switched to a BTU uh BTU dbn image so we could use pick up some newer tools uh but really either one works I was using Gradle uh creating RPMs but I'm just using their nebula plugin exactly so it's you can use both it's very easy yeah as long as we have something that can you can automatically build your machine out it works great next step is to make sure that you have your tests actually used a lot of folks write tests but don't actually run them we have a nice system set up here where after we push to our Dev branch and get we have Jenkins come in here you folks use Jenkins right so Jenkins says hey you checked stuff into Dev I'm going to start this up I'm going to run 100% of the tests if even one fails it sends it back and says sorry I'm not going to merge anything you have to fix your tests if everything passes from Jenkins we do a merge to master we Kickstart a new job which says okay I'm G to build it only takes a few minutes and then actually goes out and deploys it I recommend something like this force yourself to get all of the tests running and then it's easy so at this point all I really have to do from intellig when I have something done is I do a get push uh I go out and get coffee and read Twitter and 20 minutes later it's up and running in our staging servers I don't really or you know theoretically I could be working during that time or you know sword fighting the point is you shouldn't have to spend a lot of time deploying if you're manually deploying you're kind of wasting time you should be very productive during your work hours not have to do any kind of manual deployment process um this is what we have at Netflix this is a wrapper around AWS it's open source it's been out for a few years it's pretty good we have a better system coming out soon called Asgard 2 which will do include things like the auto deployment feature of saying hey you published your artifacts out to artifactory or wherever I'm going to grab them and then deploy this out to a new cluster and then uh turn off the old cluster we don't have that quite there but there's scripts and other solutions to do that but having some ways you could quickly see what's out there is great because then you could go to your clusters in AWS and say hey awesome here's my current version here's the disabled next version and what I want my build script to do is to say hey I have a new Ami that's ready to deploy great I'll shut down and kill the old one I'll deploy to the new one if everything's awesome I'm going to automatically switch traffic to the new one and disable the uh the previous one so we have something as guard 2 will be doing this I know that there's a lot of automated deployment systems like that but you should have a really nice way to do this so you don't have to deal with it at all set it up and let it deploy for you the other nice thing of course I like is when you have a system that can take do a build and then build out your dban archive and then build out the actual Ami and then use that to bake the machine is at your deployment are now immutable which we all like so we have so then you're working with servers that never change when you need something else do an automatic deploy and have it start up and then shut down the old servers one of the biggest problems is a lot lot of people have had issues with you know deploying property files or other kinds of configuration files out there and you don't get everything updated at the same time use something like uh Docker use uh use items which will let you build out your machine images and deploy them as the entire machine and then never touch them and you'll save yourself a lot of hassle so you can you know drink coffee read Twitter sort fight or build apis all right last one I have here oh wait I forget if this works oh yes okay so this is just an example of what I talked about we have the system saying hey we're going to go ahead and deploy traffic out here and then you have the nice ability that if you have a problem which your test didn't catch you can always roll back to a previous version it's just a single click and Asgard to say disable this to enable that I recommend coming up with a system that makes this very easy so this you shouldn't have to spend any time neither should anyone else all your time can be focused on wonderful app development yes sir well I set up alerts yeah sometimes I set up alerts if any if I ever call like a log forj error I'm trying to set up so it'll just email us in production uh I have I use like U log 4J formatting and have a just an error log set up so I can just look for it but you know new relic's pretty good there's anything which will tell you that there's an error I think is really useful all right thanks so last one docks wo we love docs right uh who writes docs for their apis here well you're wasting your time oh sorry what's that oh so you're wasting your time because because there's actually a lot of things I'll do this for you but I'm glad you're doing it cuz docs are really important so I think I have a little bit of time here uh this kind of important story here um don't speak of if you heard this one so there's this guy going around on all fours on the street corner underneath his Street Lamp and a police officer is walking his bead and walks up and sees this guy and says hey buddy get up and the guy says I can't I'm looking for my room key I can't find it I'm looking all over and the officer says okay I'll I'll help you an officer gets down on his hands and knees and he's trying to help the guy find it after 5 minutes he says look buddy we've looked around this corner I can't find it are you sure you left your keys here the guy says no I dropped them up the block off says what are you doing here and the guy says oh well the light's much better here all right it's you know it's like 50 years old but I I like that joke anyways I think that's very illustrative because when you have an API the only apis that you're users are going to see are going to use are the ones they see if you have apis that are not documented if you don't have apis that are easy to access people are not going to use them so you've kind of done a lot of work and created a lot of awesome tests and really put doing a lot of deployments for nothing so if you write an API that you don't want folks to use don't document it but then why write it in the first place make sure you have really good docs and people can find it because if they can't find and use it they're not they're not going to uh make use of it at all all right see it was a necessary joke I had to I didn't have a choice so um yeah the big thing is if you show up and make it easy for people to find much better chance that they'll actually use it so this is an example of swagger this is the built-in tool I was talking about you have built-in support with scalatra it's pretty easy the nice thing is is when you have this done you have this beautiful rainbow view organized by by the uh actions that you have and you have't make it very easy to say Okay I want click and look into it and it's awesome they create HTML fields for all of your input forms you could paste in ad Json you have documentation about what this actually does you can add notes it'll take your case class I know it's by the bottom here which is in the great idea but this is a case class that it'll then automatically convert into a documentation for you it's awesome it actually requires very little work to get this to to happen so here's our authentication item again the only thing we have to do to get this into Swagger is we have to say hey by the way this is a Swagger operation and when people want to look at it I'll say oh well this returns a token response so make sure to take this case class and turn it into documentation and here's the name of it and here's the description and here's the two parameters and Swagger will take that it'll build out all the documentation it'll build out a form and the nice part is this then becomes kind of your dashboard not only is it your documentation for people to find it and not only is it in inventory and catalog but it's very interactive there's buttons you can actually try out your whole API here uh it's not just SC scalatra I think that has swagger you can generate the Json yourself but if you find a framework that does it for you you're going to be a lot more productive all right I think that's six of six so let's say we take this we say look here's the title of it here's the two parameters if we then Swagger IE it this is what we get back we actually get Jason back this is their front end that takes the Json that describes all these fields and turns it into an interactive one interactive form you can do it yourself if you want but the great part is we could try it out put in user and password test it out and actually see the response interactive documentation I think it saves a lot of time yes sir yeah description ofation po swag is actually got nothing to do with implementation itself so there's nothing really um to prevent them from getting up sick yeah I mean I I could change that to user two or something and that wouldn't change that so it's not like you have a way no I mean you could really screw with people it's true uh the good thing is is that you can test it out here so that I I would recommend testing it and of course if you change it the docks are wrong it might be okay but you won't be able to use it so testing it manually is always a good idea good point all right what's it manually is never IDE that's what this gives you you can actually see it you can see how it works try some things make sure it works doesn't take very long all right so that's six of six that's a big thing I just highly recommend if you're doing Enterprise apis or you're doing something similar pick really great framework stick to the standards do good tests and docs and make sure that you can spend your time you know surfing Twitter or whatever it is you do instead of having to do all the drudgery I meant at API development all right that's everything thanks folks any uh any other questions yes sir uh what's your recommendation dealing with API changes like versions of API how do you this this way well for the enterprise we usually figure out try to keep track of who's actually calling our API so if no one's accessed it in a month we usually just go ahead and change it if they have exist we talk with them to see hey we don't want to have two versions of our API we work in the same company let's work together and roll out stuff out the same time if you do some people use a standard like SLR slv1 V2 and that could work but that's a pain cuz then you have multiple versions the great part about Enterprise is it's used internally you should know who's calling it I recommend fixing it I don't like disabled code I delete code all the time I don't like to disable stuff you should keep things I think very well maintained and you know you have G or something similar you can always roll it back if you need to all right any other questions yes sir yeah so you describ stu how like you deal with people that different views right uh I mean like those API so you have like the right view of things I think oh but in the in a deep company like you promote with apis oh well that's a good question uh we have a few different users of our stuff we have other teams we have uh front-end developers we have folks accessing it from groovy and Java so it's a mix of used for websites and use for other systems um I don't know I haven't really had problems with it the F the fact is they know where to find everything we have they can interact with it they can test it if they need extra things We'll add it we usually don't remove you know just to keep things compatible but it usually does make sense to iterate pretty quickly if we do change something it breaks something we won't be able to even get our code in every test has to pass for us ever to merge to master all right any other questions so you gu sorry how you gue oh okay that's a good question sorry if I kind of sped past that um so we're using we're using Asgard and we have an automatic deployment system so what you have here are this is clusters it's kind of an idea of an autoscaling group but with the idea that you have a single Jenkins build here which is numbered for each one so the code on here is different from the code on there we make sure that we always have at least one cluster up and running so that if this cluster is fine we'll shut down this cluster but we never make we make sure we don't have no clusters available so if you do that then you're automatically rolling out to a new cluster you verify it's good you shut down the old cluster people are switched over and you never actually have a downtime at all and by using immutable deployments uh where's immutable immutable Bing immutable deployments you're making sure that you don't need to shut things down that you don't have to like copy things over you have everything ready on a new image that you need you can deploy that out to pools and when it's all ready you you make sure it's good and you shut down the old one and you're all set so your deployment system can help make sure that it'll maintain and move stuff out for you so you never actually have zero things running at a time that make sense all right great well this has been uh this has been fun thanks everyone for coming thanks for PR health for uh hosting this place appreciate it Matt