SBTB 2014, James Earl Douglas: Scala devops: collaborative development and continuous deployment
all right thanks Jason uh so I'm going to talk about the decidedly unsexy uh topic of uh development operations or devops um just a quick show of hands who here is actively involved in devops uh in such a way that it's not another team over which you throw objects over a wall okay a decent number but for the rest of you I hope this will be uh kind of exciting and enlightening uh in spite of the the lack of category Theory and monads and things and if you want to follow along pretty much everything in this talk is available in kind of a tutorial form and if you go to this URL Wi-Fi withstanding um you can sort of see a step-by-step guide to all the stuff that I'm going to kind of gloss over it will also be in much more detail and you can see configurations and how to set it up for yourself and and so forth but in a nutshell what I want to cover is as scholar developers you know we we can all bang on the keyboard and make characters appear on the screen but getting getting from code the compiles to uh projects that are reliable in production is there's actually quite a lot of interesting stuff in between those two and I think setting all of that up in a way that is automatable and reliable is actually very important um and so the I guess sort of a philosophy of behind all this that I stole from a book which stole from another book uh if if you look at the book lean software development uh implementing lean software development I believe uh anyway it's on my last slide uh you'll see this quote which basically asks a couple of really important and maybe kind of uh scary questions uh how long would it take for you to change a single line of code in your code base and get that into production and I think for a lot of us that's actually that requires a lot of manual steps building packages putting them somewhere going through a QA process deciding it's okay getting it into in front of a small subset of users and then a larger subset of users and so forth and then a follow-up to that question is assuming that the answer to the first question is something small do you do this often is it repeatable is it automated and so forth and so what I want to kind of walk through is an an actual example of how you can answer both of these questions favorably and you know put some processes into place that kind of take a lot of headache off of your uh your plate um and I think before we dive into this um the the last thing is is sort of um motivating this is uh you know development operations uh is is sort of like pants uh you know you don't want to have to think that hard about them they're not that exciting but when you get out into the real world you pretty much want them on so anyway Let's uh uh let's go ahead um so these examples uh as I said are um are real world examples that you can use they're backing some open source projects uh that many of you are probably using um quick show of hands uh who here uses the xsbt web plugin for uh servlet-based web apps a few that's right uh cool so that's one of the projects that's backed by this process and there are a few others um okay so before we go any further let's Define what I mean by devops for the purpose of this talk it will be fairly specific uh we want to look at the the overlap between sort of three areas um software engineering writing code and and all the processes underneath that quality assurance so making sure code is doing what it's meant to do and even understanding what that means in the first place and then the underlying it operation so getting things out into the real world in a predictable way um and so that kind of shared uh creamy center of that uh I'm calling devops for this talk okay so the first part of that software engineering uh we all know fairly well um we bang on the keys uh there's a picture of me that my colleague sent me uh and we we make software happen uh so that involves a few things you know we might start with a requirement specification that says okay this is what the thing should do when it exists then we make it happen and then there's various testing and and verification and peer review and so forth and all that kind of falls into the bucket of software engineering and then it operations uh is making sure that new code gets deployed failures are detected and maybe rolled back in certain cases servers are scaled up and down as low changes all these sort of infrastructure kind of physical uh user-facing aspects fall into this bucket and then finally quality assurance is making sure that the users of your software whomever you're developing features for you want them to be happy so you don't want them to get the fail well you want them to have good quality of service access to your features and you know at whatever timely manner is feasible and and so forth okay so I uh for the sake for the sake of presentation I try to break this up into four kind of phases um but it's actually kind of difficult uh to do this because they're all sort of intimately tied together but let's just pretend that it's possible and I'll start with the process of writing software in a way that will support continuous integration and continuous deployment and kind of all of these ideas uh and that will kind of be isolated and then we'll see how that can be then dovetailed with collaborative development so bringing in more than a single person on your software engineering team and then moving those processes into something that's repeatable and then finally something that actually gets code from the developer to the end user and so yeah these are this is really kind of a bowl of spaghetti because these things are all kind of uh touching each other but uh I'll do my best to try to split them up like this um and so the first thing that uh comes to my mind when I think of software engineering as a phase is is really uh encapsulated well by this tweet uh that was actually just made a few days ago that says when you're trying to write software there's really two phases to it first you need to decide what the heck you even want to do and then you do it and it's very easy to overlook that first part um especially you know as coders it's really fun to just start writing code especially on a new feature and there's you know a lot of times we have a pretty good idea of what we want to build I know I need to make a new web service that performs some function given some input and has maybe some reasonable quality metrics and so forth but it's actually really easy to get that wrong different people look at a problem in different ways and might see different aspects of it so with test driven development and again insert the Dr Evil quotes this is a kind of my my definition of this we're really focusing on first figuring out what we want to build and having that like very clearly identified and then we can actually make it happen and so this gets further broken down and we We Begin by first making a specification and so that might be something informal like hey make sure that you can handle floats in this already existing numeric uh function or might be something more specific like hey we need this very specific web service that will be used by these customers in this way uh has to have this latency and this availability and these types of parsing rules and so on and so forth so it can be as detailed or uh or coarse grained is uh is appropriate um but regardless it means defining the problem and then since this is tdd the next thing we do is write the test and the idea is the test sort of solidifies that specification as code as something that you can run once we have the test we can write the actual code the implementation that will make the test pass and then we can look at our test coverage and and derive some interesting understanding from the information that we get from that um so with specification we have in this case I am sort of simulating a customer adding a new request to a project and who here uses GitHub sweet so you all know this view this is a new issue so imagine that a user is going to your project and they open a new issue in this case it's a feature request and they're saying uh golly it sure would be swell if there were a way to find the sum of a bunch of numbers and this is a really good user because they have put in a sample request and the sample response and kind of fairly well boxed the uh the problem that they're trying to solve so what what they want is the ability to call this slash add endpoint pass in a query string that has some comma separated numbers and then the get a response from that endpoint that just includes the sum of those numbers so there's our specification The Next Step would then be maybe a development team member or or someone who has some kind of more technical ties to the code would then maybe make a comment on this issue and say okay I think I understand your problem I think you want an endpoint that looks like this here are a couple of tests that I think cover your your use case and so we have one test that puts together request sends it off and then checks that the answer is correct the the numbers were summed correctly and then there's kind of an edge case what happens if we call the endpoint and we have no numbers to add and in this case we expect the the we expect it not to fail but it just doesn't return any result um and so this is kind of an implicit contract where the developers saying okay if here are some tests if these tests pass are we good to go is the service implemented and presumably the user would then come back and say yep that's cool uh or it's possible that they would say well no you kind of forgot a case what if we want to add non-integer values or what if I want to add integers that are written as English words instead of numeric values what happens when I send a thousand requests at once so there might be all of these different you know use cases or edge cases that we want to cover um for the sake of Simplicity this is all we need to care about okay so now the developer has a reasonably well defined contract they know what they need to do they've actually even inspect out the test code that they need to write which is kind of Handy and so they're ready to start developing so the first thing they do using git is make a new feature Branch so it's a new Sandbox that they can start writing code in locally and they put in there two tests that they already specified on the GitHub issue and so I just quickly run through these so the first test says oh by the way these are Scala test tests in case you're unfamiliar in the first test we set up a list of numbers we set up a HTTP request using it as yet unspecified get function we send that off and then we expect the response to be 42 and then in the second test we send an empty request we just called slash add and we expect the response to be just an empty string okay so now the tests are written uh using SBT we run these the both of the tests and we see that they both fail not too surprising if both the tests were passing they'd be a little bit suspicious but it would also mean that we're done and we can go home for the day but we expect these tests to fail because we haven't actually implemented this endpoint yet so this is good and so now it's time to implement them and as I uh mentioned before the xspt web plugin lets us write servlet-based web applications and for the sake of this demo that's what we'll be using so to make these two tests pass we'll write a servlet that implements this endpoint as the tests describe it and there's our simple servlet that just fits on the screen um thank you Java and so this is a very very simple server implementation all we do is check the query string of the request which might be a null if there was nothing provided so we wrap that up an option and uh and then iterate so if there was a query string provided we extract that split it up by commas turn each string into an INT and then take the sum and write them out to the response and so you can probably see that there's a lot of room for failure here we're making a lot of assumptions about the input that it is a sequence of strings Each of which that can be converted into an integer uh that they are indeed integers and not Floats or doubles or big ends or something else but again the remember that our tests uh should only pass if we're we've completely satisfied the requirements so this is a good start we should see if this passes the tests uh and Adida does so even though we've kind of identified some potential problems it looks like we're done and so at this point it's kind of a judgment call in the real world uh since we've so quickly identified a bunch of problems with this thing we would probably go back to the customer and say hey by the way you might not have realized that uh there could be things other than integers or uh there might be values that are not parsable as integers or there might be you know requirements a b and c in addition to that should we worry about those things and how do we handle them and then the answer might be no we don't care yet let's get version one out and then we can add these as kind of you know new features or something or the answer might be yeah you're right we really don't want something that's this brittle in production and so it can kind of depend on your particular situation but for this presentation uh the customer is totally happy with this um so we're good to go so now uh the next step is to look at the test coverage and test coverage is really uh just a way of looking it so we want to know some information about how the tests relate to the code and with test coverage we can run all of our tests in such a way that our code has been instrumented to collect metrics about how the code actually executed under test and in Scala with SBT we have this coverage plug-in that does this very well for us so instead of running SBT tests as we did before we can just run SBT coverage colon tests and it will do this instrumentation and then run the tests and generate some interesting reports for us oh and pretend that says total two past two and so here's a I don't want to get too much into the coverage yet because that will come later with the continuous integration and deployment but here's kind of a little sneak preview of what the one of the reports might look like and so it'll give us some summary information about how many lines of code were run and how of those lines of code how many were executed during the test and how many were not and so this gives us a good idea of how much of our servlet code actually was exercised by the tests and two reasons that this are useful are we make sure that we captured all of our cases so imagine if some of the servlet code did not run we need to ask whether there are some tests missing maybe we need a different type of test to exercise that code or maybe we wrote code that we actually don't need based on the requirements it might be just super close code that we don't want to maintain and then on the flip side if if all of our code ran you know seven times if we looked at and unfortunately that's not in this slide but if we drove a dove down into this ad servlet class we could see line by line breakdowns of how many uh calls to each line of code there were and if it's some very high number we might start to wonder well maybe our tests are too rigorous or you know uh will end up causing us you know maintenance cost in the long run and maybe we can simplify those tests but for now we decide that our coverage is good our test is good uh as a developer I'm ready to have somebody else actually take a look at this code and decide what we do next with it so as the developer I um will push my Branch up to my GitHub project and open a new pull request and I think most of us are familiar with this so I opened a request and I say hey here's this new branch called issue five uh the the title and the description link back to the original issue so that we know we have some kind of traceability and I'm ready for someone to take a look at this and so I'll go and you know knock on the door of my co-worker or uh you know there might be if if there are multiple people on this project someone will get an email when I assign it to them or so forth but one way or the other someone else is going to look at this pull request now so the first thing they do is a little code review they look at the tests and they say uh yep those tests do indeed cover the specification that we want and then they'll look at the implementation and go through all of the same questions about do we really want code this brittle etc etc but so let's say that they they like this code the next step is would normally be upon acceptance of the code to merge the pull request and put that code into Master but actually before I get there I want to take a little quick side step into the sort of continuous integration phase of devops because we actually were not quite ready to merge this code we want information from our build system which is hosted outside of our local environment uh and so the the motivation behind continuous integration is that we want to take repetitive tasks and take them off of the plates of developers we don't want people to have to worry about things that are automatable uh especially when they're complicated and involve a lot of steps not only is that kind of mental noise and it distracts from um their creative output but it's also error-prone and often difficult to repeat so with continuous integration we'll take our GitHub project and we'll point it to a build system of some sort in this case we're using Travis uh who's familiar with Travis okay how about Jenkins or Hudson okay so pretty much everyone has used Jenkins Travis is like a hosted uh Jenkins and it has some very nice integration with the GitHub apis so it makes for a very uh and it's free for open source projects so it makes a nice uh integration point for our process here uh so let me go back one side so we have our GitHub project we have our pull request and this project has been set up through a config file that you can look at in the example code to tell Travis and GitHub how to communicate with each other about things like pull requests so when the new pull request is open Travis detects that and it will pull the code just like Jenkins do a build in this case it's using SBT with these coverage test command uh so it runs all the tests it looks at the output to see if the tests pass and then it also takes the coverage information and we'll send that off to another service called coveralls and coveralls is sort of like Travis in that it's a hosted service it's free for open source projects and its purpose is to give you kind of a nice view of the coverage of your project project in a nice historical way and so coveralls gives you that view that we saw earlier but in um with a lot more tooling around it and it too uh will integrate with the GitHub API and tell you very interesting things so let's say I I submit a pull request and I forget to write complete tests or or whatever so my coverage goes down uh coveralls will then leave a comment on that pull request saying hey by the way the coverage dropped from 100 to 90 you may or may not care about that okay so now we go back to uh the peer reviewer looking at the pull request and we can see here at the bottom is this nice green box that says all is well the Travis build passed the code can be merged go for it and so we do so the peer reviewer clicks the button and now that code is in master and this actually will trigger another build in Travis because it detects the change to the master branch and so we'll go back through the testing process and and so forth okay so what have we looked at so far uh it's kind of a whirlwind Whirlwind tour of these tools um but mainly this is the flow we have a developer who's writing code they're pushing it to GitHub GitHub is then talking to Travis uh saying hey there's a new PR or there's been a commit or there's been a merge or whatever um and Travis will say hmm okay I'm going to take the new code and I'm going to build it and and tell you something about it and by the way I will also tell coverall something about it and that's pretty much it for continuous integration uh the next step to sit on top of that is taking the output of what happens here and then actually putting it into the real real world in some way and so continuous deployment is basically everything we've seen plus this extra step of getting packages out there for other people to use and the two tools that we use for for this phase are a tool to host kind of arbitrary binary packages and that's bintry bin tray is is especially useful for hosting big jars that you produce from SBT uh it's uh it's basically like a maven repository or an IV repository if you've ever gotten anything from sonotype or a biblio or any of these other things bintry is similar except it's way easier to use and it I think the SBT Community is maybe standardizing on it it's it's part of the spt docs as well and so this is this slide is a little bit cheating because I'm not using this web project example for this bin Trace slide and that's just because for the web project we don't actually publish any jars there's no point we will actually publish the the war file that contains the web application so for this slide I used a screenshot of the xsbt web plugin project which is a jar that's put up on the web and when you run SBT locally and uh you're you've told SBT about this plugin spt knows how to go to ventray and pull that jar down but for our web application we'll use Heroku and so Heroku most of you are probably familiar is a web hosting platform and they have a convenient way to host War files which are servlet web applications using I think a using a tomcat Runner that they've set up you know you could replace this with Amazon or with Google app engine or um elastic Beanstalk are you there there are many Cloud providers but this one's a very simple uh example to show off okay so now we've added these two extra tools and they integrate at the point of our continuous integration server which is Travis okay so now that we have been Trey and Heroku wired into Travis there's an extra deployment step that Travis knows how to do so we've built our code we've submitted a pull request Travis has pulled the code from the pull request built it checked the test results and now it's pushing information to GitHub it's pushing coverage information to coveralls and it's also pushing packages assuming they successfully built to bintry or Heroku and so when you push a package in this case a war file to Heroku the your web application will get hosted automatically and so you can start throwing requests at it and so now we're getting back into our our QA facet of devops and so now we are start to verify that this feature is actually complete and so we send a curl request with a you know very simple list of integers and sure enough we get a result so that's cool and then we can also start to look at edge cases and and other things that we might be concerned about and so now we're kind of uh off the script of the specification because it said nothing about what to do when we can't parse the input or how many requests we need to handle in a given amount of time but it's still things it's still aspects of our system that we could look at and so as part of QA we might throw these edge cases at our server and decide well these actually are fairly important let's submit them as new issues and so handling unpersonable input could be a new feature that we then describe and go back through this exact same process okay so this has been like a kind of a cut down example of what it looks like for one developer to write code build it test it look at the coverage get it into a source code repository that other people can look at and then have reviewers look at the code look at the test results look at the coverage results and then approve or not approve and then finally have somebody maybe one of those two individuals or even somebody else do testing of uh of the final output and of course in in the real world you're rarely the only developer on a project so you know you can imagine a slightly more complicated example where we have uh you know two developers or ten developers and they're all working on the same project and so we have lots of different pull requests coming in and feedback on those pull requests and uh you know different people committing to different branches and so forth but luckily all of that scales with this process and you actually really don't have to change anything that we've seen so uh in conclusion uh what what we really uh what our goal in kind of all of this is is to answer both of these questions favorably um how long would it take to deploy a single line change of code uh we want it to be fast we wanted to be well instantaneous but at least minutes uh is a good start and we want it to be so easily repeatable that we don't even have to think about doing it and so these tools just handle that for us the the way that we were able to get there through these tools is uh is through automation so the I guess the kind of con the the thread that ties all of this together is that all of these processes are are simple enough that they can be automated easily but there are enough of them and they're tedious enough that they're difficult to do manually so we want to automate as much as possible so that we just don't have to think about it and then we sort of get for free constant builds and constant deployments but of course there's a lot of stuff that I didn't cover uh things like you know with Heroku we have the ability to automatically scale up and down as load changes uh we also when we deploy our war in this example when we deploy our web application to uh Heroku there will be some down time while it brings down the old Dyno and brings up the new one uh so those are actually easily addressable uh just slightly outside the scope of this presentation uh similarly a b testing where we want some users to get one experience and other users to get another experience these would be test these would be achievable through things like load balancing based on certain conditions or uh you know having a multiple node system that you only partially deploy to and then switch over things like this of course none of this is available in the free versions of these tools but they're totally doable with these processes uh oh and similarly non-public bills so Travis coveralls Heroku GitHub and all these tools work great for open source projects but you know most of us work on non-open Source projects as well and and so they all offer paid versions that let you do the same kind of thing but you don't have to expose your code to the world and then I also didn't really cover the configuration of all these things which is interesting but not really appropriate for slides but most of it is the Travis configuration so most of it is telling Travis hey here's a project by the way it's a Scala project here's how you build it here's how you generate coverage information here's what to do with it um here's how to deploy it into production through Heroku and so on and so that stuff is really interesting uh once but then it's kind of repetitive and so forth so I recommend looking into that but um again it was outside the scope of this talk uh and then so finally uh there are two books that I would recommend especially continuous delivery that's pretty much the textbook version of everything that I've covered here and they um there are so many quotes from that book that I I could have just like read the book to you guys uh and uh hopefully achieve the same effect and then there's this other book implementing lean software development which I couldn't remember the title of earlier it's it's not quite as good I mean the content is great but it's a little bit wordy so if you had to pick between these two I would definitely pick continuous delivery and then there are also a couple of tutorials that you can go through so the code behind this one the web application and all the the new endpoint that we implemented and the tests are all in this first this scholar CD project this continuous deployment for Scala and then that is sort of an extension of Scala CI which is continuous integration in Scala so both of these are step-by-step tutorials that you can fork in GitHub and then apply to your own processes encode any questions so your question is when you're merging into Master it's not always so simple and there might be merge conflicts or or there it might merge cleanly but there's some other problem that you didn't foresee and you know this happens all the time in production code especially with bigger teams and so for this example um you know it's very direct we open the pull request the pull request is against master and it gets deployed right away assuming the build is green if you didn't want if you wanted a little bit more control over that you would just change how Travis is configured so in this case we tell Travis if the build passes deploy it and so the you know we're if if the build is green it's in production and that's basically our only Gateway it doesn't have to be that automated or it doesn't have to be that uh fragile I guess uh it's totally up to the way you configure it so you you could say if the build is green push it to staging or if the build is green push it to a testing server and send QA an email you know and then they would manually push a button to actually put it into production they're you know an infinite range of possibilities thank you oh one word what was that kind of accidents so you're asking about in practice is it really this fast mm-hmm um so the the plug-in the SBT plugin that I showed you is actually this fast and it's an example uh and then there are a couple of other open source projects that really aren't very exciting um that that I'm a maintainer of that also use the process um but in practice uh things that can slow it down you know to be slower than a matter of minutes can be uh you have a lot of tests in your testing process just takes a lot of time or you might have stricter policies about when software gets put into production and that could be you know for safety like you want to be very absolutely sure that your feature works or it might be more marketing driven you know we don't want this feature out until it's announced but uh you know all of those are sort of details that you kind of work out and then decide how how that kind of deployment chain looks but but in practice this this framework um as a basis is is pretty reliable yep okay so you were talking about the when we were in our QA phase and we're checking does the system work and how much load can it take these are these are reads these are not affecting our system for other users uh what if what if our tests involved creating some data and then manipulating it some way and checking it but it's really test data and we don't want that in production you can there are different approaches I think they're kind of outside the scope of this presentation they're because they're sort of orthogonal to it I think the intersection might be if you want a QA environment that's totally safe to mess with and you can throw whatever data in there and it will never touch production then the the only difference is rather than pushing to directly to the production server from Travis you would push to a QA server or you know some kind of staging environment or something like that uh but either way you could also architect your system in a way that test data and real data can live side by side in such a way that it doesn't affect users because of the semantics of the way you've designed your data models um you also have a test cases in the same or specifications slide in the next slide also you have a separate slide oh yeah it's a little bit confusing yep the one before which passed the API document the other one this was when the the requester submitted the issue originally was there a different one that you were thinking it seems like sorry I'm not quite sure which slide you're referring to um are you talking about the the code review I'm really not sure it's in the beginning foreign maybe you can grab me offline and we can find it yeah cool well I don't want to sand between you guys and lunch so thanks