Scale By The Bay 2019: Joe Beda, Kubernetes is a Platform Platform

Scale By The Bay 2019: Joe Beda, Kubernetes is a Platform Platform

Recording: Scale By The Bay 2019: Joe Beda, Kubernetes is a Platform Platform

[Music] all right thank you so much let me grab this I like to wander when I'm talking so before I get started can you all hear me is this working well yeah awesome so I'm what my friend Brian likes to call an over-represented person in technology his head white dude old if as we go through this you're not over-represented and some of this stuff isn't clicking for you and you want to go deeper please reach out to me I'm happy to make the time I just want to put that offer out there before we actually get started all right so this talk I'm titled kubernetes is a platform platform I think another alternate title might be why I'm still excited about kubernetes so we're like five years into kubernetes that's a long time to do anything I'm kind of exhausted by it I know you are too but hopefully this is actually will help you know bring some depth and actually look at it in an interesting new light all right so a little bit about me I'm a platform builder I started my career Microsoft I've actually been up in Seattle my entire career and working on the Internet Explorer project and you can boo me if you want and the fascinating thing starting your career at Microsoft and wherever you start your career oftentimes that's sort of like you imprint on a lot of the the values of that company and and say what you will about Microsoft they are through and through a platform company I mean I can get up here and bounce around and say developers developers developers but you know Steve was right right it really is about developers and Microsoft feels that in their bones and that really imprinted on me early so a lot of what I've done through my career is really about platforms and so for me a good platform is something that enables people to be more productive it raises the the toolkit that they have to work with and it allows them to do things with your platform that you couldn't imagine when you actually created it and I think that's one of the most sort of satisfying and excite things about working on infrastructure things like this so I was at Microsoft for about seven 1/2 years and then doing client-side stuff and then I'm like I want to figure out how this stuff works on the server and so I joined this this little company called Google up in Seattle I was you know we had a like four or five people up in Seattle at the time and this was 2004 right around the IPO and I'm like well you know if I'm gonna figure out how stuff works on the server side Microsoft not the place Google probably has this stuff figured out so I joined and immediately started writing client software because Google at the time like you show up and they're just like hey work on this I think they're a little bit better about it now but over time I actually did move towards doing stuff server-side at Google and again that imprinted on me and so Google does not run stuff server-side and manage infrastructure the way that the rest of the world does and so I was sort of twisted in my sort of learning how to do stuff on that side of the the wire and and it only later did I actually figure out sort of how things work in the real world but Google runs this system called Borg you've probably heard about it over the course of 10 years they essentially found a way to use every single ounce of performance and capacity out of the computers in their data center it you know the primary driver for Google for a lot of these systems ends up being one of efficiency when you're running that many machines a single-digit increase in efficiency actually turns into real money and so a lot of the focus was there combining the you know early Google was all about search and search was essentially two processes there was a crawling and indexing process which was largely batch and then there was a serving process which was low latency and so a big part of Borg in the infrastructure that was built was really aimed at sort of bringing those things together into one compute platform to use every single piece of the of the computer along the way though the abstraction that developers at Google used to run stuff in production changed it moved from being managing servers to managing abstract processes that are scheduled by a system there's definitely echoes of HPC systems the stuff goes back a long time time sharing that type of thing but the birth of this you know the the with this really was the birth birth of a lot of the core technologies that are Linux containers today and so Google spent the time putting the C group technology into the Linux kernel which is sort of about like half of the core kernel features that make containers work and and it worked really really well and and so when I was at Google I did a bunch of stuff I did some Google Talk I did some ad stuff and then finally we did cloud things and I started Google compute engine well at Google which was really about how do we actually bring something that's familiar to the rest of the world to Google VMs because Google generally didn't run vm's at that time but then we wanted to bring the developer experience that Googlers had closer to the developer experience that the rest of the world had and so that meant either having everybody at Google start rewriting stuff on raw VMs or bring a borg like experience to the rest of the world and that really was the genesis of kubernetes along the way you know Google doesn't work like the rest of the world so we had to change we had to adapt we had to listen we brought in early partners like Red Hat through openshift and it was you know really really exciting to actually see the the merging of these ideas the stuff that had been proven at Google along with the needs of the real world all right so what does kubernetes you all probably heard about it and sort of the most practical level kubernetes is a way to take a process package up everything that that process needs into a container image and then give it to kubernetes and it decides which computer to run that thing on and once you get that dinah system of running a bunch of things across a bunch of computers that creates some new problems and some of those problems are things like how do you find those things how do you communicate between those things how do you do load balancing how do you handle storage and so kubernetes starts with that dinah system of dynamic workload placement and then does the necessary things to be able to fill in the rest of the gaps and start solving the rest of the problems that that creates okay but what does kubernetes really kubernetes is a database but you didn't see that coming it's really a database with a policy layer in front of it with a bunch of control loops that actually work against that database and one of the fascinating things that emerged out of this is that the patterns that we use to solve that problem around workload placement and communication and replication and the things that that sort of our part and parcel of kubernetes those patterns ended up being useful for so much more and so one of the things that I'm gonna really go in deep here is about how are the core patterns for how kubernetes does what it does what are some of the sort of design philosophies that we have there and then how do we how did we create these extension points so that we can make these usable in more context and so that's what I think is really exciting here all right so here is some beautiful artwork that I drew all right so at the center of kubernetes is really I got a laser pointer check it out is the Etsy D which is a which is a database now this is an interesting database it's actually if you trace the lineage it goes back to this thing at Google called chubby horrible name for a system and and that there was a paper written on it that inspired things like zookeeper so it's in that same class a databases oftentimes called a lock server highly consistent database there whether or not that's the right database for kubernetes to have been to have used I think you could argue that there are some interesting features that actually come out of Etsy d in this class of databases that I'm gonna get to in a little bit that are critical for the way that kubernetes works the only thing that talks to Etsy D is this creatively named server called the API server and so the API server is essentially a policy agent in front of the database the database is a raw database the API server essentially takes different schemas exposes them to the world implements a lot of the sort of API gateway manage many type of things that you want to do on top of that things like permissions things like authentication so that's primarily what that API server does there is some sort of kubernetes in the API server you know the these these abstractions are not as sort of clean as we'd like them to be that's one of the things that the community is going to be working on in the future is to actually sort of take this part here and turn this into sort of what some folks have been calling sort of a universal control plane that is not specific to kubernetes and so I think that's an exciting future direction that we're going to see happen in here so then there's these other processes that we call the scheduler and the controller manager so the scheduler is the thing that actually does the bin packing of which container is going to run on which machine it's it's interesting that it's called scheduler because we usually think about scheduler it's like a time thing but this is actually a space scheduler so it's the placement engine and then there's this sort of kitchen sink thing called the controller manager which does a lot of this sort of you know background processes to be able to make kubernetes do its container thing and then finally on each of the worker nodes we have this server called the cubelet it's essentially the agent and it's sort of fronts docker or the container runtime whatever one you're actually talking to so it's the thing that actually runs the the actual containers for you um naming here cubelet is is a google ism in borg we had the the Borg master in the Borg tlit the successor system to Borg was called Omega so there was the Omega master and then there was the omelette it's a really bad joke it's really bad alright so I use the term controller here and I think this is really fascinating because this is essentially the core distributed system distributed system primitive in kubernetes and it may be a little bit different from some of the stuff that you might have dealt with in other contexts the whole goal of this is essentially reconciliation over time kubernetes by and large when you look at sort of the overarching control patterns does not have state machines instead what you have is you have these component I systems where you actually tell kubernetes here's my desired state here's what I want you to make true and then it's up to a controller to look at the real world look at the desired state and try and make some forward progress so it's sort of a greedy algorithm in terms of relentless forward progress how do I actually say here's what here's what the user wants and it could be the user or it could be some other system because we layer these things on top of each other here's what the user wants here's what the real world is let me try and converge these things over time now this is a really really resilient distributed systems pattern because if the controller crashes and loses all its state it can wake up and actually discover all that it can start over and that question of what does the user want what is the real world let me try and converge these things that has actually really resilient if things crash if you have these things fighting against each other if you have bugs this thing ends up being very stable the other thing that's really nice about this is that it's it's really good at dealing unexpected situations when you build these big complicated state machines oftentimes your state machines really don't take into account some of the failure modes that you really should be thinking about and so a lot of you know over time a lot of the sort of debugging and hardening of these types of things is understanding more and more of the different failure modes adding states to be able to deal with those with recovery all that type of thing with this controller model if you find something that you didn't expect well that's just a reality that you have to deal with and you still make that forward progress towards where you want to go all right so this is a picture I mean my family after hiking out of the Grand Canyon and the Grand Canyon if you've ever hiked it is fascinating because they say that down is optional and up is mandatory and so you know you take a couple of kids down to the bottom and then you sort of push him back up it was fun and this is at the top you know but like along the way your job is there's very much this one foot in front of the other relentless forward progress you know the kids are sitting here going my desired state is to be at the top with ice cream my current state is I'm not at the top what do I do to actually get myself closer than that and so they're very much at this sort of primitive sort of you know primals sort of like okay I just want to actually get from here to there and that's a very much that's my example of a controller there in real life so here's a sequence diagram of how these things interact and layer on top of each other and so sequence diagrams are objectively the best diagrams for engineers so I love this these types of things so the way that this stuff works is that the user here creates a pod and for all intents and purposes replace pod with container writes that into the API server and the HPI server writes into NCD so this is essentially writing in here's my desired state of what I want and then the scheduler actually gets notified hey there's a new pod that hasn't been assigned to a machine it then goes through figures out which machine it wants to run that on binds that pod to the to the worker and then writes that back into a CD and then returns and so you can see that at no point does the user actually talked to the scheduler directly right the API server doesn't actually really reach out to the schedule I'll describe what I mean here by watch and then finally there's the the cubelet the actual node it actually says oh I have a new pod that's been assigned to me it says it should be running it's not running let me go ahead and fix that it then goes through and essentially calls docker run my machine went to sleep or something here what happened me trying plug and unplug see if we can get this going maybe okay there we go sorry about that right so this thing essentially calls docker run and then updates the status as sort of like because this could take a while and there might be intermediate states like pulling images and stuff like that so the fascinating thing is that these things layer on top of each other they don't communicate directly all of the different signals that they get are through this sort of controller model intermediated through the API server and the fundamental database all right so so that watch term that I used there and so like in distributed systems that's sort of like how does one component find out about something else often becomes a really really hard problem I mean this is you know Kafka right so the way that this works is that there's a quality of data by databases like Etsy D like zookeeper like chubby where you can subscribe in a lightweight way to say hey when any of these particular items in the database change let me know right away and so that's called a watch it's different than a subscription in like a pub/sub system because the watch is not stateful it's actually based on the underlying connection if your connection drops and the server is like oh well and so the next time you reboot or you reconnect you have to essentially do a recent go and like I don't know what happened in the immediate time all I care about is what is the current state of the database not the events that happened in between and so this is a big place where watch systems differ from event systems in event systems oftentimes you'll get all the intermediate events you may care about the end state but you'll actually get the blow-by-blow as it goes with kubernetes you could have a controller sleep for 15 minutes come back up it doesn't care what happened in those intervening 15 minutes all it cares about is what is the desired state now what is the reality how can I make relentless forward progress towards converging those things and so this is what we call sort of a levels level triggered system because you're actually based on sort of what is the state versus an egg trigger system which is your based on events and so it's a it's a really it's a bit of a twist from sort of the event based distributed systems that I think a lot of folks are building at the applications all right so a lot of times we talk about kubernetes in the class of systems that we're talking about here as a container orchestration and orchestration to me implies that you have a plan that there's some big score and everybody has their part to play and it's all planned out beforehand state machines are kind of orchestration but with kubernetes with these controllers reacting against each other and working with each other it really is it's it's more like jazz improv than an orchestra so like let's say that you're an orchestra performance and the fire alarm goes off you know people probably exit calmly you know put down their champagne right they leave right and and it very much interrupts that performance now if you're at a jazz improv formance and the fire alarm goes off it's just as likely that the performers will work it into the music they'll actually work with it they'll roll with it they'll actually deal with those unexpected situations and I think that's one of the really interesting qualities of this type of distributed system all right so so that's the core of how kubernetes works one of the things as we built it that comes through is what we like to call sort of a UNIX philosophy kubernetes introduces a lot of concepts these concepts are granular it's a lot for folks to wrap their heads around and oftentimes can actually be a barrier for folks understanding the system but those concepts are are relatively small scoped and reusable and layer above and so just like when you're using when you're learning UNIX and there's a command line and you find that I have all these primitives I can put together to do unexpected things that's how we wanted to build kubernetes we wanted to have a bunch of primitives a toolbox that you can use that you can put together to both do common things but also to do the unexpected things what it means is that sometimes the common things are not as straightforward as easy as you might like but you know through the fullness of time we think that it's really gonna these primitives are going to actually sort of last and be reusable in new ways all right so see today there we go okay so the core primitive I talked about here is a pod so pod is essentially a footprint a resource footprint on a physical machine or you know one of the worker nodes could be a virtual machine I guess it includes a set of containers these containers can actually work together in concert in a sort of tightly coupled way and so this paves the way for things like sidecars and service meshes and a lot of this sort of you know more advanced deployments that kubernetes enables it also includes a network configuration for that set of containers along with storage requirements and how that storage gets mapped into each of those containers so that's the core primitive that the that the cubelet understands and that's the thing that we're talking about as we schedule these things now on top of that we build a controller that we call a replica set and this is where we see that sort of desired state really come into play the replica set is actually fairly simple it has a replica count and it has a template for creating a new pod and what it does is it reaches out and it says well how many pods do I have that actually match this pattern there's this label query system that comes in there also how many pods do I have that match this pattern if it's less than the replica count the desired state then I'm going to create new using the template just stamp out new ones if there's too many then I'm gonna actually go through and find some of those pods that that are in access and actually kill them and delete them and destroy them and so it's a very simple thing to really reason about the the replica set wakes up looks to see how many pods are out there looks to see how many pods that should have and then take some action to be able to make forward progress now what this means is that if you start out and you you have a replica set and you're running sort of you know some sort of like web server application server and you put in 100 as the number of replicas and the system's like okay a hundred I'm on it boss right it goes through it starts creating pods and let's say it gets to like 50 pods right because it may over time maybe there's a limit to how fast it actually moves that we've some dampening in the system to make sure that it doesn't you know it doesn't actually sort of swing back and forth like somebody learning to drive stick and so like let's say that you've gotten to like 50 pods and then you change your mind you're like oh I didn't mean 100 I ten and a lot of sort of state machine like systems you might actually have to go through and complete that first operation to go to 100 before you actually go through and actually go back down to ten in this sort of controller pattern you can be at 50 and then the controller wakes up and it doesn't know that you cancelled or changed your mind or whatever all it knows is that it's like I have 50 you want ten let me go ahead and start killing some of them and so that's the where you know you can be reactive to unexpected changes that's sort of the jazz improv type of feel in play so the replica set is very simple it doesn't understand about versions and so when you want to deploy a new application oftentimes you want to make sure that you are responsible about upgrading versions and you want to kill your old versions and roll out your new versions on top of that we layer this thing called a deployment it does a rolling updates essentially manages a set of replicas sets just like replica set manages pods the deployment manages replica sets it brings up a replica set for the new version it has a replica set for the old version and then it actually changes the number of replicas over time to actually be able to spin up the new version and spin down the old version and in doing so does a rolling update of all your processes kubernetes does not have a built-in way to do something like Bluegreen deployments but using some of the extension mechanisms that I'm going to be talking about you could build that yourself alright so that's all for so like server workloads this sort of like I want to bring something up and I want to keep it running forever we can also do this for batch workload so we have job which actually creates a pod and it actually runs that pod to completion and then on top of that we have cron job which actually launches a job based on a template based on a time schedule so you can see we're reusing some of those pod primitives as we go all right so that's all fine and good for the built-in things now this is where things get really really exciting kubernetes has a bunch of built-in schemas and a built-in controllers several versions ago and these things have just really hit Ga with kubernetes we introduced this concept of what would become custom resource definitions this is essentially taking that database adding new schemas adding new API types so that now if you're like okay I like the deployment idea but I want to do some slightly different now you can create a Bluegreen deployment using these custom resource definitions so you can take the control patterns that work for kubernetes work for the built-in stuff and you can instant extend it into domains that kubernetes didn't think about and so that's really exciting because we recognize that there's going to be these types of control patterns that folks are gonna want to do that we're not going to always we're not gonna be able to do everything and some of the stuff we're gonna do we're gonna get wrong so we wanted to make sure that we created these extensibility mechanisms so that people could take these ideas and run with them so that introduces this term operator that you may or may not have heard of so I'm talking about operator as in a piece of code not operator as in a person when I first heard about this as a little confused but then when it clicked I'm like oh yeah computers used to be people too the idea here is that an operator is a type of controller this is my definition of operator it's a type of controller that has domain-specific knowledge for managing a kind of system and so instead of creating a generic controller for doing Bluegreen deployment let's create a controller that knows how to actually do a H a deployment of say my sequel or Postgres right so now this operator has domain knowledge for how to actually deploy my sequel if something you know dies unexpectedly knows how to take those sort of operational run books renders that into code and actually drives that process automatically something like RDS run on any idea AWS in my mind as an operator right what is Amazon do they took a bunch of code they took a bunch of operational best practices they rendered those into code and they put it behind an API so that it was automatic this democratizes that so that anybody can use kubernetes and the kubernetes control plane concepts to be able to create these self managing systems you know that that are sort of in the vein of something like RDS and we're starting to see an explosion of these things things like confluent has a commercial Kafka operator for example and there's a bunch of community operators right had just released a I believe it was a Postgres operator into the CNC F as a sandbox project so we're starting to see these cases where these COO Nettie's control patterns are being used to do more than just generic application workloads but also create self managing systems and I predict that over time we're going to see applications that have sort of built-in operators they wake up and go oh I'm running on kubernetes I know how to actually use signals from the system and interact with the kubernetes control plane so that I myself can be self managing as an application so there's a there's a fly in the ointment here making these controllers making these operators is a big pain in the butt they're not easy to write and so one of the sort of bleeding-edge things that we're working on right now in the community is exploring new ways to make these things easier to write making this stuff be more accessible to more people and we made a lot of progress we still have a lot of progress to go there's one example it's it sort of been hasn't been driven super hard recently called meta controller from this engineer at Google that essentially lets you write one of those Bluegreen controllers that I mentioned in something like 125 lines of JavaScript and so taking common patterns making these things more accessible so that everybody can start using these distributed system primitives using these patterns to be able to do more and more control and then finally like everything that we're talking about so far actually ends with launching and managing a pod an actual container workload but these control patterns are useful for more than just managing things on kubernetes we've seen folks manage other types of systems like say a networking system or a load balancer system so if you think about like a centrally controlled load balance or something like nginx you have a config file managing that config file is actually kind of a pain in the butt and if you have to manage that config file across a bunch of different teams what often happens is you have like the one or two people whose job it is to manage that config file and you have to file a ticket or send them email every time you want that thing updated as you start scaling this up to larger organizations that really doesn't work well and so one of the things that we see folks doing is using kubernetes such that all of the different snippets of config that may end up in your say nginx config put those in as kubernetes objects and then you have a controller that watches for those changes looks at the state of the actual config for your nginx and then goes ahead and pushes an update it's that so we can start reusing the patterns for configuring more than just pods we can use them for configuring network for configuring storage or for configuring larger business processes I've talked to companies that are using this pattern so that they actually have one of these customer resource definitions that represents a team and an application and they have controllers that say well you know for every application I need a slack channel I need a a github repo I need a CI CD pipeline and they go through and they have run these controllers and these controllers reach out to other api's to automate creation and maintenance of those different systems and so that way if somebody is like you know disgruntled and deletes their slack channel this thing will automatically recognize hi I should have a slack channel there's no slack channel let me go ahead and recreate that for you and actually add all they members and and keep it up to date so it's that reconciliation rib-eye thing you know continuous terraform is another way to think about some of this and so that I think is really for me the exciting thing about kubernetes as we look into the future it's really about taking these things that have these ideas around distributed control some of the the core concepts for being able to run this stuff and then creating that Universal control plan creating that platform that lets you build higher level abstractions that that you know capture a lot of the operational knowledge a lot of those patterns make those things more accessible and so the core stuff that is part of kubernetes right now I predict that over time this will fade to the background right like how many people actually know the latest version of Linux that is like released and stable right like people you know yeah I'm sure I'm sure there's some but you all recognize that like that like that's not where the action is right now Linux is foundational but it's faded to the background right people are really worried about other stuff and I think that's the success of good infrastructure is that it fades to the background it helps shape the way we approach problems and fundamentally it becomes boring and so our goal with kubernetes right now is to get it to the point where it's boring but then it's also something that is boring in a good way in that it's something solid that you can build on we're not there yet but we're still I think making good progress so with that I think I'm out of time I think I hit it pretty good right yeah all right but I'll be hanging out a little bit if folks want to discuss some of the stuff later thank you so much [Applause] [Music]