SBTB 2023: Alex Klibisz, Pragmatic, scalable, and ergonomic CI/CD using Github Actions and K8s.
uh today I'll be presenting about CI and CD using GitHub actions and kubernetes and just quickly before we start just want to emphasize that these are my own opinions and experiences I'm not speaking on behalf of my employer in any official capacity so to set some context for this talk I just want to quickly highlight some of the things our team is building we focus primarily on building a lot of the cloud software and infrastructure behind all of Tesla's Energy Products so I've included some posts here highlighting a few of the recent projects where we played a role we've worked a lot on Virtual power plants adding non- Tesla vehicles to the supercharging network and residential features like stormat and we've also worked on a ton of other apps uh unfortunately there's not enough space for me to even include most of them um but if you're interested in learning more you can check out the links on this slide and also incl include them at the end as well and if you're interested in learning more about the technical details of the software that we're building I also won't be talking a lot about that today I'll be focusing on CI and CD but I would highly recommend checking out a couple of excellent talks from my colleagues Natalie and Hector on the left and Percy and Colin on the right and I'll also include these Links at the end so let's start with some quick definitions to make sure that we're all on the same page here so first what is continuous integration or CI so to me CI is simply the process of automatically validating the changes to your software before merging them and especially before releasing them so typically this amounts to running some sort of a build Pipeline with compilation linting uh and testing steps on every pull request for example and then continuous delivery or CD uh is the process of releasing automatically or if not automatically then very easily releasing frequently and ideally in small lowrisk increments and continuous integration and continuous delivery get grouped together often because they rely on a lot of the same underlying tools one of those tools is GitHub actions GitHub actions is github's platform for CI CD and just general programmatic automation so I'd like to cover some details on our engineering workflow uh is that influences a lot of our choices in CI and CD so as said before we're primarily focused on building Cloud software we work in a few different languages I happen to work mostly in Scala but there's also a good amount of python typescript uh goang and some others we deploy our applications primarily into kubernetes and we use Helm charts to bundle up all the amals and do some templating and we use a tool called Argo CD to handle diffing uh and synchronizing these resources into the kubernetes Clusters we have a handful of Library repositories and then we have many more application specific repos which depend on the artifacts published from those Library repositories as Engineers we write and maintain all of our own tests and we upate all the software including on call rotations to make sure that it's running 247 all of our changes go through code review and automated testing before we merge and we make many releases to production every day uh we have no fixed release schedule so given all of that why does good CI and CD matter to us so we found that it enables us to have fast and understandable iteration on our software that lets us make frequent lowrisk releases which lets us build uh reliable and robust systems and that ultimately leads to an excellent customer experience so another way to say this is that it lets us as Engineers stay focused on building the product and allows us to keep moving quickly so let's jump into GitHub actions starting with The Primitives so a trigger in GitHub actions is an event that happens in GitHub that starts a workflow so I think there are about 30 of these triggers at this point but the most common ones that we use are when someone opens a poll request when a pull request is merged uh on a KRON schedule or just running manually a workflow is the outermost runnable unit that gets Tri Tried by a trigger or an event in GitHub so we might have one workflow for CI and another one for CD and they would typically have different triggers all the workflows are specified as yaml files and they live in the repository and there's very little point-and-click configuration involved here which is nice a job is the runnable unit within a workflow so each workflow has one or more jobs and you typically split a workflow into jobs just to divide responsibilities and jobs can also run in parallel which which is nice within a job you have several sequential steps um a step is ultimately just executing a command and if that command succeeds so if it exits with Code Zero then the step is succeeded and if not then the step fails and the job fails the so it's ultimately a shell command but there are some nice abstractions on top of it so you're not always literally running shell commands yourself and then finally a runner is the computer where a job runs so we have exactly one job at a time on a r R uh when you get started with GitHub you typically use github's provided Runners so you don't really have to think about it all that much uh but you can also self-host your own Runners which I'll get into more later so this slide is sort of the hello world of GitHub actions we have a yl file called hello world. yo on line one we're giving it a name on lines 2 through 5 we're specifying the triggers for this workflow so we want it to run when somebody opens a pull request on line six we start specifying the jobs and in this case we only have one job which is called hello world we use this runs on key to indicate that we want it to run on a runner with the auntu operating system and then we get into our steps which is a sequential list and in this case we just have one step where we run the command Echo hello world so when you run this type of workflow and open up a pull request uh you'll see the workflow running and it's basically just showing that hey I've started the workflow and it's got a job that it's running you can click for more details so then you click into the workflow View and it shows each of the jobs in this case we just have one and we click into that job and we can see that it executed our step which just echoed hello world so that's sort of where you get started with GitHub actions so when we started with GitHub actions we decided to do a proof of concept for some of our scholar projects that use the spt build tool so we started with a workflow like this for CI we want to trigger it whenever somebody opens up a pull request to the main branch and then we have a single job with a few steps so the first two steps are checking out the source code and then installing Scala and the spt build tool then we run spt compile to compile The Source Code spt test to run our unit tests we use Docker compose to boot up some containers that we're going to use for integration testing and then we run our integration tests which uh in some cases we'll use those containers that are now running and if each of these steps succeeds then we get a green check mark on our PR and we know that our code is working similarly for CD we have a separate workflow in this case we use this workflow dispatch trigger which basically means that we can go through the UI or through the API to start this workflow we check out the source code and we install Scala and spt this time we're going to log into a Docker registry because ultimately we want to push up a Docker image then we assemble a an application jar that contains our source code we build a Docker image that contains that application jar and then we push up that image to our Docker registry at which point it's ready to be deployed into kubernetes so that was a good start and it showed that GI up actions can work but we're still missing a few details so for example we need to give SBT some appropriate memory settings we need to configure spt to bootstrap itself uh and its dependencies from an internal artifact repo and then we need to do like at least six other things and by the time we do all of this we end up with a ton of yaml and if we were to just take that yaml and copy paste it into all of our repositories it would work for a little bit but it would be a ton of duplication and then inevitably it would drift cause confusion and just be really difficult to maintain so we need some form of abstraction here and one form is called the reusable workflow which the idea here is that you essentially start with a regular workflow you add some input parameters to it then you commit that to a repository and then you can reuse it in all of your other repos so we quickly reached for these reusable workflows and the first one we started with is called reusable spt C so like the name indicates it's a reusable way to run CI on any project that's using spt as long as it adheres to some uh common structure so you can see on the left here how we're using this reusable workflow on a pull request and on the right you can see that it runs as a job that's executing about 35 or 40 steps that set up the workflow uh build our code compile it run test and so on and I've zoomed in here on these steps uh we particularly like splitting them up as granularly as possible so that we can tell you know when something fails we know exactly what failed so to jump in a little bit deeper on the left we're actually using the reusable workflow and the way we specify that is with this uses keyword so we point at it essentially and it lives in another repository and then we pass in some custom input parameters so these are parameters that we've defined in the reusable workflow and it essentially tells the workflow what to do so for example on line n we're saying that we want it to run s coverage checks to measure our test coverage then we say we want to use Scola style and Scola format to lint the code then we tell it that we'd like to run integration tests and that we have a Docker composed file that it should start before the integration tests run and we say that it should also run assembly to make sure that we can still assemble our jar and here we're saying that it should inherit the secrets that are available to this outermost workflow so these secrets are used for things like authenticating to our Docker registry or to our artifact repository and by saying that we can inherit them we're essentially just giving the reusable workflow permission to access them so overall the reusable workflow implementation ends up being about 500 lines of yl which would be a lot to copy paste but using it in a in a individual repo is only about 10 lines of yemo which is great because we use it in virtually all of our scholar reposit stories so then we also have a reusable workflow for CD um this case it's a little bit more interesting I'll zoom in on these code segments in a minute but the high level is that it's using actually three separate reusable workflows and running them sequentially so the first reusable workflow is responsible for building and releasing our Docker image as input we tell it where the docker file is and what the image name should be and it knows how to do the rest essentially build the the code build the image and push it up to a registry in the second reusable workflow we're uh deploying the new image to one of our lower environments so we use Argo CD to deploy and the inputs here are the Argo CD app name and then the image tag which we've pulled in uh as an output from the previous reusable workflow so basically pushed up the image tag and then it tells us hey this is the the new image tag the third the third re usable workflow uh is essentially taking a list of values files so Helm values files each of these specifies the latest image it's taking that new image tag and overriding the previous image tag with the new image tag and then making a pull request to our repository so the pull request looks like this you can see that we've got a series of Helm values files typically one for each of our environments that we deploy to we had the old image tag and now we've replaced it with a new one and an engineer will then take this pull request approve it merge it and at that point our changes are staged and ready to be deployed and we use Argo CD to synchronize them into our production clusters so overall the implementation is about 700 lines of yaml but using it is only 30 lines and again that's great because we use this in a lot of places so as a recap reusable workflows are basically a way to share a workflow across many repos we use them for standardizing CI standardizing CD for code scanning with different security tools and for various kinds of linting like for our Helm charts or pull request titles but in some cases you don't actually want to reuse the entire workflow you'd actually like to reuse a single step and there's an answer for that too which is called the composite action so with a composite action we start with an existing step and we add some input parameters to it we commit that to a repository and then we can reuse that step in our other reposit atories so if you've ever used action checkout or action setup python or any of these other setup actions these are composite actions it's basically a bundle of code that somebody has packaged up and made available to you so you can run it as a step so here's an example where we're using a composite action to send a message to our internal chat server so online uh seven here we're using this uses keyword again to basically reference this notify chat action and then we're passing in some input parameters like the title message Channel and so on the result is a nicely formatted message that lands in our chat server and you can imagine that this is useful in a lot of different jobs so it's very nice to have it packaged up as a composite action that you can use anywhere you need so we've also used this composite action pattern for configuring our build tools so that they use an internal artifact repo for running the schol Steward tool and for caching and restoring build artifacts and dependencies in our jobs so an interesting question here is you know you've got all these reusable workflows and composite actions how do you actually verify that they're working or who tests all of this yaml so like I said before we test all the code we write so we also found ways to write automated tests for these uh workflows and composite actions so on the left we have an example where we're testing this reusable btci workflow we essentially just want to make sure that it's going to run end to end successfully on a simple project so we've committed a simple SBT project into a repo and then we're referencing this reusable spci workflow to run on that project and then the only input we provide is where it should find that spt project um and if this runs and10 successfully then you know that at least you can build this sample project in reality we have probably like 15 to 20 of these different test cases for different types of projects and different combinations of inputs on the right you can see we're testing the notify chat action that I talked about before this case is a little bit more interesting because we're actually testing that when we give it a bogus input it should fail so it shouldn't silently succeed and then fail to send the message so we check out the source code which includes our composite action we reference the composite action and we pass it a bogus Channel then we provide this continue on error flag which basically says this job should continue running even if this individual step has failed which is important we give it an ID so we can reference its outcome and then the final step we're taking that outcome from the previous step echoing it and just grapping for the string failure and if the contains the string failure uh we know that the step is failed so the only way to get through this workflow successfully is for that second step to fail as we expect it to and then for us to assert that it failed you can think of this as CI for our CI and CD uh we're not able to test all of the scenarios because some things are just stateful or difficult to set up for a test case uh but we're able to test enough to where we can keep moving quickly on these uh reusable workflows and composite actions all right so now we've got these reusable workflows they're tested they're concise to use and so we start integrating them into a bunch of our repositories but we quickly hit a hurdle which looks like this so we've got an awesome testing job except that it started 4 hours ago and it's still waiting for a runner to pick up this job so what's happening here is that we just have too many jobs and not enough Runners to run all of them right now so Engineers are sitting around waiting in a build queue which is not great uh so one solution here might be to just add more static longlived Runners that are just always on and sitting around ready for a job the problem with this is that cicd workloads are very spiky so they tend to quiet down on nights and weekends but you can see otherwise uh if we were to just try to pick a number of Runners to keep around all the time we would either keep Engineers waiting in some cases or we would have to have a ton of over-provision resources that are just wasting money and energy so this kind of a workload is one where some type of autoscaling might be really nice to have so fortunately we already run a small kubernetes cluster that we use to deploy development tools such as Argo CD and it also just so happens that there's an open source project called the actions Runner controller which is a kubernetes controller that lets us deploy GitHub Runners into kubernetes so at a high level we can take this actions Runner controller or Arc for short we can deploy it to our kubernetes cluster and we get autoscaling runners so to dive a little bit more into how this works we have GitHub and our kubernetes cluster we deploy four new resources into the cluster we have a web hook server this thing called a horizontal Runner autoscaler if you ever used a pod autoscaler very similar a controller and then a runner deployment which is similar to a regular deployment when someone starts up a job in GitHub GitHub actually sends a web hook event to our web hook server at that point the web hook server tells the horizontal Runner autoscaler hey we need another Runner to run this job and then the controller takes that and adds a new Runner to the runner deployment so the new Runner comes online it's usually pretty quick like on the order of 5 Seconds maybe it registers itself with GitHub and at that point job number one starts running on Runner pod number one and I say pod number one because this is literally just a kubernetes pod that happens to be running the the runner software we can do the same thing for a second job and then for a third job and so on up to of course a limit of the Max uh replicas that we've defined and then at some point these jobs complete and our runner deployment scales back down to zero so another important detail here is that we actually have multiple Runner deployments and each of them has its own autoscaler this might seem weird but the reason reason for this is that each Runner deployment uses a fixed size runner in terms of the CPU and memory resources that are allocated and you can imagine that resource requirements vary so if you're just linting some yaml that's very different than uh building a large scholar project so each of these Runner deployments has different sizes and we can specify that a job should run on a specific one by using these runs on labels so I've oversimplified it a bit but you could think of it as we essentially have a small medium and large and then when you write a job you can say my job should run on the small one the medium one or the large one so this is how things end up looking over the last month so on the top we have the number of nodes that we've used on the bottom we have the pods that we've used for our Runners um and you can see there's a lot of variability here and again if we had just tried to pick a magic number for the number of Runners that we want to have then we'd either leave Engineers waiting sitting around in a build queue or we'd be wasting Resources by over-provisioning so Auto scaling is cool but you can also take it too far so every job when it starts is making a few API calls to GitHub and if you start too many jobs at once GitHub gets unhappy and is ends up rate limiting you um and we found that this comes up very occasionally um but it has been an issue in some cases we've gotten pretty good at balancing the right amount of autoscaling so that we don't uh you know get in trouble with GitHub another thing we can do that's uh useful within Arc is to have a custom Runner image so it's just a pod and you can specify the image that you're running so we've built a Docker image that contains our most common tooling so things like languages build tools and various clis and the nice thing about this is that our jobs start up faster because we're not wasting time reinstalling the same tool over and over and we also reduce the number of API calls we're making to GitHub a final important topic here is observability so we're running tens to hundreds of these Pods at any point in time and it's tied in very tightly to the engineering workflow um so we want to make sure that it's working the various components of Arc actions Runner controller emit various Prometheus metrics and these are pretty useful in some cases we also rely on structured logs to diagnose very specific issues you can see that a log is tagged by the job ID so it can be useful for fig figuring out what happened with a particular job overall this works pretty well but there are still some tricky cases in particular it's difficult to distinguish these sorts of nuisance failures so when a job didn't run because autoscaling failed for some reason or when a runner got killed because kubernetes decided to get rid of the node that it was running on or when a job failed due to rate limiting a lot of that is still difficult to distinguish within just the metrics but overall we're very happy with Arc we've been running it in you know production as production as our developer tooling gets for over a year now and we'd recommend it especially if you're already using kubernetes so the final section of the talk uh I'd like to cover how we optimize GitHub workflows and jobs for Scala and spt projects the first thing that we can do witht is use this Dash client flag which is going to start an spt server in the background and minimize spt startup times so anybody that's ever used spt knows that the startup times can be very long so if you run a command like spt version which just prints the projects version you're going to spend about 16 seconds in this case waiting for it to boot up and then you'll spend about half a second for it to print the version um less than half a second to print but point is long boot up times with spt so one thing you can do since SP spt 1.4 is use this- client flag and the first time you run it you're going to also see the same startup time but it's starting up a server in the background and that means the second time you run the same command or any other spt command on that project it'll connect to that spt server and it's effectively A Thin Client that just connects and you don't have to go through that whole setup process of uh loading up spt so you get a run time of like half a second which is really solid the way we use this is we essentially just add this- client flag to each of our commands and again this makes it nice because we can split up the workflow into individual steps so you can tell exactly what failed and we're not paying this penalty of having a very long startup time with spt here are some links if you're interested in uh digging into this the second thing we can do is run a separate parallel job for every one of our sub projects this is particularly useful in SBT projects that have a ton of sub projects it can actually be faster to just um run these all in parallel and if you have the resources available they will all run in parallel which is which has been very helpful in some of our larger repos it's not a silver bullet uh you still have to watch out for rate limiting um but it is an option that you have finally we found it useful to cach our dependencies and our compilation artifacts this seems pretty obvious but I'll go into the details of how we you how we do it so for dependencies in particular we need to cach two directories there's an spt boot directory which holds spt's dependencies and this corer home directory which holds the Project's dependencies we also need to tell SBT about this but you can just pass this in as a configuration parameter in your SP spt opts file or there are other ways to pass this in depending on how you run spt and then for compilation we essentially just need to cache all of these Target directories which is where spt puts the compiled artifacts and when we do this the next time we run a job that's been able to pull down the cache it only needs to recompile the files that changed since the last time we pushed up the cache which can also obviously save a ton of time typically pull requests you know if you're going quickly and making small changes or changing very small increments of the overall project so I pulled some numbers here actually ran this yesterday with and without caching on sort of a medium I siiz project and you can see that overall without caching it took about 213 seconds and with caching 40 seconds so it's about a five times speed up and we happen to use a custom action for caching internally but all this also works with the publicly available uh cache action that's provided by GitHub so there are some other fun topics that I did not have time to include um maybe you can get your hands on the slides later and jump into these if you'd like but at this point I'd like to summarize so what do we like about GitHub actions it's a pretty intuitive ux you never have to leave GitHub so if you like GitHub that's great and it's pretty easy to get started there are some good constructs for reusability these reasonable workflows and composite actions and you can even test them with a little bit of effort everything lives in source which is particularly useful because there's no point-and-click configuration that can drift and the documentation has been super helpful and the community is strong there's a lot of people using GitHub actions so pretty much if you have a question it's probably already been answered some things we like about the actions Runner controller so it's relatively simple to implement autoscaling especially if you're already using kubernetes you can get your own Runners without using kubernetes there are some other options but if you're already on kubernetes uh yeah it's pretty straightforward the runners are ephemeral which is super nice cuz you're not leaving around State for the next job to trip over the documentation and Community have also been uh solid here and then obviously there's a little bit of room for improvement that we'd like to see or we'd like to help contribute to uh one thing that would be nice is improving the observability for distinguishing these sorts of nuisance failures like when a you know when a runner has died compared to sort of nominal failures like a test failing uh right now it's difficult to distinguish these in uh just the metrics it would also be great to see some more flexibility in runner sizing so right now you have to deploy a separate Runner deployment for each size of Runner that you want it would be cool if the workflow could determine that dynamically and it' be great if the runner software made more efficient usage of the API uh especially during startup to minimize the impact of rate limiting so with that I'd like to say thank you to youall for joining uh today if you're interested in learning more about the software and systems that we build check out the two talks linked at the top here uh if you want to know more about our project uh products check out tesla.com energy or the energy and charging accounts on X and then most importantly if you're interested in helping us accelerate the transition to sustainable energy check out tesla.com careers or just come find me at the conference uh today or tomorrow thank you