SBTB 2019: Adelbert Chang, Everything old is new: today's infrastructure as yesterday's Internet
[Music] yeah so my name is adilyn bear Chiang I am a software engineer at Target where I work on inventory positioning and control team supply J our supply chain optimization comes tough and today I want to talk to you guys about something that's been on my mind ever since essentially it took my role at Target where I'm now working more to sort of systems and infrastructure space and and being asked to think about a lot of these core systems that our team is using so infrastructures are very rapidly growing area which makes it simultaneously really exciting but also kind of overwhelming from all these different new technologies from containers to cluster schedulers to servers mesh to service meshes to server lists it's very hard to keep track of sort of what the lay of the landscape is going day to day week to week month to month sort of figuring out what's hot what's new what's trendy what's what are AWS and GCP and all these new cloud all these establish cloud companies sort of trending towards and while we do have a lot of this popular technology that a lot of people are going to be talking about blogging about speaking about we do want to as engineers want to figure out how much of this is marketing and how much of it is sort of technical excellence right and so when we go to some of these websites these can be kind of hard to evaluate and of course when you do go to these websites these splash pages are going to often be targeted towards to draw you and as as much as possible but as engineers we're trying to sort of get at what the difference is between these systems are as opposed to maybe what sounds trendy at the moment and what will be trendy in oil change in like three months six months a year so I pulled some examples from popular open source info technologies so this page is going to be about cluster schedulers so I want your company's website kubernetes starts off with kubernetes is an open source system for automating deployment scaling and management of containerized applications tells us a little bit if we're just looking for say if we're a company that's putting everything containers and we want to deploy it somewhere sound solid so far but doing our due diligence maybe we go to a competitor's website and we see no Matt is an easy-to-use flexible and performant worklet Orchestrator that can deploy a mix of micro-service batch containerize and non containerized applications since we're maybe in the containerized world maybe the non containerized part isn't really relevant to us but like so far between kubernetes and nomad it can be hard to tell what exactly the differences are why we should choose one or the other certainly maybe kubernetes is more hobby marketed but maybe Nomad is a more mature product and then for completeness sake maybe go to mesas website Mesa subtract CPU memory storage and other compute resources away from machines enabling fault tolerant elastic distributed systems to easily be built and run effectively so just looking at these three which are all in sort of same domain they can be very hard to figure out okay what if I'm being asked to choose a cluster scheduler for my team I want to be sure to choose the one that's gonna last beyond however long this kubernetes hype is gonna go on or however long the current state of my orgs and for is gonna go on so so far pretty hard to tell what the difference is this DS example is also applied to other domains right not just cluster schedulers if we look inside this sort of service much space we can see link Rd is an ultra-light service Metro kubernetes gives a little bit of information gives you observability reliability and security without requiring any code changes it sounds awesome right for completeness we go to stos website SDL makes it easy to create a network of deployed services with load balancing services service authentication wandering and more with few or no code changes and service code and so this is to1 isn't even from the front page right I actually dug into the docs a little bit to get this blur because the one on the front page was actually a little bit more vague than this but these two are essentially to this exact same statement right both of them are giving you observability monitoring they're both giving you reliability load balancing and they're both giving you security services service authentication both being both advertising that you can do to all this without reporting any code changes so just look at these descriptions can't really tell what's happening and not to rag on just like these sort of cloud native infrastructure technologies even products I work on are guilty at this too so for those of you who are here last year at scale by debate I gave a talk on this continuous deployment system called Nelson and on a front page of the Nelson's website Nelson fuses years of field experience with rigorous formal methods to prove to provide a continuous delivery system that just works users are empowered to focus on building their products whilst operators came guarantees the applications are meeting to organizational best practices lots of little marketing speak there it doesn't really give us a whole amount of detail we sort of have to dig through the docs to get a bit more information maybe a competing product spinnaker spinnaker is an open source multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence created at Netflix has been battle tested production by hundreds of teams over millions of deployments it combines a powerful and flexible pipeline management system with integrations to the major cloud providers and my point here really is that across schedulers across these service measures across these continuous deployment systems even across like container runtimes there's a lot of effort being put into marketing which is not a bad thing right the landscape is huge marketing push can be hard to dig through and this is not necessarily bad because market forces are certainly a factor that people should consider when adopting technologies for instance the sheer momentum behind kubernetes has led to the adoption of the existence of systems like eks the key key like aks where your organization can just sort of throw money at these public cloud companies and you get a managed cluster for free whereas that same kind of momentum wouldn't behind necessarily other cluster schedulers for instance you don't we don't really have the same level of like EWS GCP adoption as say Nomad right so market force is certainly are a factor to consider my focus on this talk is to not just consider these market forces but also to sort of look not just at the technical differences between these competing products but also to understand their technical capabilities and limitations over the long run beyond sort of the trend right it turns are sort of this illusion that are really being pushed onto us constantly and they're an illusion they come and go technical capabilities tend to stick around organizations organizational challenges tend to stick around right what's hot today is gonna be hunting six months isn't gonna be hot in here so what my focus for this talk is gonna be is are there principles we can use to evaluate systems against each other right even though we can even when we do try to dig through Docs and we can get it we can enumerate a list of maybe what Nelson does well what spinnaker does well what linker D does well it would be nice if you had sort of maybe not a formal set of rules but at least some set of principles we can use to sort of check check our math make sure everything looks okay make sure everything is aligned with what we need for our work and for me especially in my sort of previous stint as a sort of systems researcher several years back I believed in all these principles if we are indeed willing to go back 15 years and actually as I was writing these slides I realized that even though 15 plus years sounds like a long time that's really just like mid 2000s which is kind of surprising to me so I'm gonna take us knock back 15 plot not just 15 plus years but actually take us back 30 or 40 years back to whatever this diagram is who does anyone know what this represents who actually knows what this diagram is one that's the Internet right so this is back then called the ARPANET this is a diagram from around 1969 this was sort of the first I guess live deployment of like a separate geographically separated components trying to talk to each other I am a little proud to say that that third node on the left there UCSB if I'm alma mater but that's all I say about that but so during this moment during this era where we are now we now have this like actually we're moving beyond just having focusing on a single system and focusing on and moving this towards this trend of geographically distributed nodes there were a lot of design questions for distributed systems that might might not be as applicable today or might be taken for granted today but the principles that rose from them I believe still hold today so some of the questions that came out of the design of ARPANET and what is now the internet our questions around how do we do reliable delivery how do we how do I make sure the bits from UCSB arrive at right in the way I want them to how do i do congestion control when all these geographically distributed systems and separately managed systems are all trying to talk to each other at the same time as ARPANET grew into an internet and different organizations or different stage or different providers had their own sort of federated systems how do we route traffic with not just within a system like maybe I'll have one way of routing traffic just within California but also how do I then route traffic in between different say Internet service providers right which are separately managed but still need to be able to talk to each other and just led to a lot of research a lot of designs a lot of systems level thinking a lot of iterations and a whole bunch of papers talking about different trade-offs and different ways to think about these systems came out three of them stand out to me and our sort of foundational papers especially in the networking space display papers are hints for computer system designed published in 1983 into an argument system design published in 1984 and puzzles in cyberspace defining tomorrow's internet published in 2002 as it turns out it is very difficult to cover three landmark papers in 30 minutes now 20 minutes and I made the same mistake two years ago when I try to cover three things in spam twenty minutes it did not end well so what I'm gonna do today is instead of trying to go in-depth and sort of cover them as like go deep into them I'm just gonna try to take on one lesson from each of these papers and apply it to a mono system today just sort of get people thinking hopefully get you guys thinking about how despite these papers being 3040 years old how do can apply to even water systems today I do want to say that these papers contain more than one lesson there's no way I'm going to cover any of these papers even I couldn't I don't think I could do any of these papers justice covering just one of them in the time I have left so I encourage if you are interested to read them come up to me after talk to chat about them by the interest of time I am just gonna take one lesson from each applied to system and then move on I want it before I get into into that I do want to open with a direct quote from the first paper on my cover hands from computer system design so just paper is basically lays out several instead or several sort of principles that the author has sort of experience in iterating across different systems not just networking systems but also does single systems or design programming languages across different fields so he starts off with these are non novel foolproof recipes laws of system design in our operation precisely formulated consistent always appropriate approved by all the leading experts or guaranteed to work and I think the same disclaimer holds to this talk personally everything I'm about to say I believe in these are opinions that I hold but maybe your opinions differ but that's fine I think my goal here is just to get the conversation going and to begin evaluating systems among these different axes which I think sometimes I'm not given the attention they are due so we'll start with the 1983 paper hints for computer system design it starts off with a machoke which I think a lot of people will often espouse keep it simple do one thing at a time do it well kind of like the UNIX philosophy kind of maybe a more slightly more modern version maybe as people to keep it simple stupid kind of vague so we can go a little bit deeper a couple paragraphs down the service must have a fairly predictable cost the interface must not promise more than the implementer knows how to deliver this starts getting a little bit more interesting I think especially if you're an engineer in industry this was particularly speak to you if you've ever had like a product manager suddenly say oh we have this one client who really wants this nice dislike educate featured like no one else really wants but like it'd be really cool if we had this or just trying to like sort of makes you leak the attraction a little bit just so you can support maybe just one or two or three other customers the cool continues on especially it should not promise features need to only buy a few clients like it just said unless the implementer knows how to provide them without penalizing others and finally it ends with make it fast rather than general or powerful I think often times it can be very easy or as especially with in a programming language we're taught to say oh right really generic functions which are great but maybe in a systems world if you really need a sacrifice speed to make it marginally more general think twice about that so there's trouble with slow powerful operations if the client who doesn't want the power pays more for the basic function now this sort of this block these three quotes when I think of this paper and when I rewrite this paper as sort of in the past couple years the first thing I think of is sort of this distinction between hash shakur and other products are talking into so who here is familiar with handshake or products or as used yeah so high scores this is a company probably best known for terraform maybe their second best known product would be console and they also run Nomad which is scheduler and vault which is service secrets security solution and what hash core I think does a really good job of doing is all of the systems they build are completely usable independently they all have a very focused goal and while there are nice integrations between them plenty of people and it's completely possible to use them completely ignoring the other products right a lot of people use terraform and without using console vault or no man there's a completely doable thing it plugged into several backends AWS OpenStack whatever you want for products like vault and console you will see a lot of companies use console as their service discovery solution or as their leader election solution and they have tool chains like console template which are designed which are just a simple binary you pull down you point out your console cluster and you can and it works you don't have to be deployed Nomad in order to use console you don't have to be in a specific environment to use vault everything is completely usable independently and especially to my point about that last part where make it fast as opposed to general powerful one of the things at this wonder products they build I think really speaks to this as nomads so cash check or norma is literally just a scheduler console is just a key value store and ball this just about security and because no because they have very focused goals they're able to make Nomad really good at just scheduling all right Nomad is in the business of helping you do service discovery or helping you mount storage or helping you store secrets it's just given give me a pull of resources tell me what the schedule what I need and I'll do it as fast as possible if you go to their website you can see them actually bragging about it they are comfortable scheduling 1 million containers across 5000 hosts and five minutes this is a number they are comfortable advertising so let's contrast this with another system that I believe is maybe at the opposite end or a little bit towards the powerful end compared to hash core products and that's kubernetes so kubernetes has taken the industry by storm but one of the big issues I have with kubernetes at least philosophically is kubernetes wants to do everything for you it really wants to be your service discovery solution it wants to be your scheduler and wants to do storage for you and wants to do auto scaling for you and wants to do secrets that can take management for you and it does provide a happy path and so long as you sort of fit their happy path you're probably good or if you're within some scale limit then you're probably good but if you want a sort of diverse women if you want to maybe say use kubernetes with an existing console deployment or you want to use kubernetes but have your own ingress solution these things start to get a little bit tricky and I think I don't really have a confess I don't really have hard evidence to back this claim but it would be not surprising if because kubernetes is trying to do so much for you that that's limited how much speed they are willing to advertise so if you go to the kubernetes website you can see that they have a hard limit on they only support 5,000 notes this used is never used to be 2,000 I was actually up for 2,000 I think it should only recently became 5,000 meanwhile Nomad is comfortable just bragging about 5,000 notes I'm sure no merits kelpy on that another limit another hard limit they have that the enterprise is they could do 300,000 containers meanwhile Nomad is bragging about scheduling 1 million containers right no mass is very focused on saying I'm going to schedule and that's all for kubernetes while it it is more powerful people who maybe don't need all that power or pain it's cost that they can now only scale the 5,000 notes or only scale to 300,000 containers and if you need if maybe you don't need all this pensions behind a storage orchestration or service discovery but you want to deploy it like a lot of containers then like you sort of need to figure out how to do how to work your way around that so again I'm not claiming that necessarily that kubernetes cannot reach the same scales as Nomad but I do believe that because not just nomads but just hash score products in general are very focused that lets them optimize for the fast case in the general case much better than a complicated system like kubernetes so second paper down one cover is ending an argument to system design this was published in 1984 this was a networking paper starts off with in a system that becomes apparent that there's a list of functions each of which might be implemented in any of several ways by the communication subsystem by his client as a joint venture or perhaps or done only each doing its own version and to give a little bit of context this paper was primarily I believe inspired by reliable delivery of TCP and so one example you can think of is how do I make sure the bits I want to send from machine a machine B get there correctly and sort of the gist of this is maybe the communication substrate can provide that functionality but really it's correctness at the application level that matters right sort of the maybe more pessimistic view is even if your pits make it over the wire you just don't need those bits to go from your your NIC to your actual application correctly for things to matter right if it makes over the wire but it's it gets corrupted between the NIC and actual application then like there's no you ship to retransmit anyways so that sort of what inspired this paper this paper goes on to say the function question can completely completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints so the actual application level concerns not just the routers or switches in between sometimes an incomplete version the function provided by the communication system may be useful as a performance enhancement and while I think the first sentiment is very important I think the second sentiment bears emphasis so some other examples just sort of give you some bearings about what else they may be talking about it you can think about in to an end and encryption right I could if you imagine internet where maybe say Comcast or some other Internet service provider tells you that we're gonna encrypt traffic flowing through between your cell phone between your router and someone else right but that involves us trusting Comcast or involves us pressing some sort of middleman and to really really if you wanted like absolute certainty that your stuff was encrypted that needs to happen end-to-end and so one of the things one of the sort of trade offs that this paper is talking about is even if your ISP or some communicating a substrate were to implement say encryption for you you would still want to implement your own encryption if you want a sort of the full complete correct version of it and now the downside is you now have to we're not being encrypted twice right even if we because we don't trust the communication substrate we are encrypting once we are paying the cost of encrypting once on the application end and then we're being we're paying the cost again to encrypt it the networking layer so this is so the paper is essentially sort of describing this trade-off here and the modern version of this I think is sort of this pattern of this this recent notion of site carpenters who here's familiar with this notion of sidecar or even just service meshes so this became popular with the advent of cluster schedulers where and containers as well essentially the idea is that you want to deploy a container the hope is that you are able to deploy container that just focuses on the application and emits logs on a standard out and imagine miss metrics on support and there's some other container called a side car that's sitting next to your container whose sole job is to do sort of organizational or administrative tasks like capturing the standard out from your container and forwarding it to some location service or capturing the metrics and forwarding it to some metrics and gesture and pipeline and oftentimes these sidecar pattern or another example that I'm actually gonna get into is reverse proxies so systems like envoy is probably the most trendy one right now nginx and eight your proxy will probably have similar products and oftentimes D sort of site cars often marketed to provide functionality quote unquote for free right we saw earlier in the sort of splash page of like link Rd and sto that they would provide observability security and I forget and monitoring without any coaching miss RIE said this is often how these things are are being advertised and if you sort of think about it this is very similar to the reliable and secure transmission examples they inspired a paper provide some functionality inside the substrate and in theory your application should just be able to emit quote-unquote like dumb messages or do just focus on what action wants to do and trust a substrate to do it but as an attendant paper argues the you can only really implement these features completely and correctly at the application level all right while you can't provide well for some things you might be you might be willing to trust say the substrate to do like log aggregation you probably don't want your application always like every time you write an application to like write a connector to Splunk or something or every time you omit metrics to write a connector to in flux so for some of these cycles I think actually make sense but for stuff like maybe reverse proxies where they often will advertise themselves as oh your application can just omit sort of dumb HTTP requests to this proxy and this indus proxy will this transparently handle retries and exponential back-off and all that good stuff for you that's cool but if you need to retry based on something other than say the HTTP header where you actually know maybe parse something and if something doesn't look right in actual semantics the message that you're getting then the fact that you need to even though the message got there sort of correctly a return of quote unquote 200 according to the reverse proxy you have to retry anyways and this is very similar to the analog of reliable transmission this is basically the exact same transmission and again because this does retry mechanism is now implemented both in the application layer and also the networking substrate you pay this cost twice now does that mean that the reverse all reverse proxies are useless no it just means that when you are evaluating these reverse proxy solutions already service resolutions that are providing a lot of dysfunctionality quote unquote for free or even all these advertiser platform at platform of the service systems you've got make sure that you are ok with this trade-off that a should you need to implement your own version of it that either you are okay to pay the cost or there is some sort of escape hatch where you don't have to pay the cost of the implementation in the substrate and if you are willing to say trust the substrate then be okay with knowing that you for things like maybe retries or for others things I need more richer or semantic information that you just won't be able to do those which is a perfectly fine thing to say I am just trying to the paper and I am just trying to say be aware that this is the trade that's happening and don't bees or blinded by the market talk of oh you get this functionality for free without any coaching there are trade-offs here so testing into an argument argument I am going to move on to our third and last paper this is a paper called tussle in cyberspace to find tomorrow's internet this was published in 2002 the sort of takeaway message from this paper is designed for variation and how come do not design supposed to dictate the outcome rigid designs will be broken designs that permit variation will flex under pressure and survive modularize the design along puzzle boundaries so that one tussle does not spill over and distort unrelated issues designed for a choice to permit the different players to express their preferences and the systems I think that I think of first when I read when I read this paper are this recent trend again in control planes and these continuous deployment solutions and in service measures some control planes will tie themselves to a system right so I don't know if it actually is tight but at least under splash bears link or D which is a service mesh and our go p-- which is a workflow engine or their leader D is a service manager for kubernetes Mario is a service is a workflow solution for kubernetes and that's great if you are willing to put all of your money on kubernetes but personally for me while kubernetes is really hot right now I'm willing to bet that it's not gonna be hot in five years or in 10 years and if your organization is willing to make that bet then that's cool but this is just something to be cognizant of and so the point here is infrastructure is rapidly growing try to design your systems not so that they're tied to whatever is trendy at the time but try to make it make what your system is actually doing separate from what the sort of data plane is going to do and so one example is going to be a system that I work on called Nelson Nelson is a automated deployment system that does sort of version and mutable deployments so real quick just to give some context imagine a service graph with versions attached to them whereas oftentimes when we wanted to play a new version of a service what would happen is the new version just in place replaces the old version right so if you deployed to 2.0 where do I want to eat curry leaves will eventually become 2.0 and the whole system just sees a new version what Nelson does is actually deploys to that new version in a completely separate location so now we have two versions of be running and in order for the rest of the system to sort of catch up or to upgrade they are expected to deploy a new version of themselves and explicitly declare instead of so he used to depend on 2.18 he is going to redeploy a new version itself and then explicitly say oh and this continues on until eventually the graph looks like that and then as the system sort of stabilizes and converges Nelson has a garbage collection mechanism for figuring out what are all deprecated services deletes the old versions and finally the graph is left like this so no sign is very clear goal its focus is on immutable version deployments does not really care what the scheduler is communicating with is doesn't really care what exact secret solution is using doesn't really care what exactly service discovery solution it's using it just wants some way to deploy a versioned thing in want some way of informing different versions of discovering specific versions of other systems and it has some notion of a garbage collection policy and the way Nelson's design very explicitly ratifies this right the mean when you start up Nelson it starts up let's see three four five seven yeah it starts up seven sort of background jobs each we have a very specific task the main ones are sort of the pipeline processor which is a thing that actually deploys it onto a scheduler then we have the routing cron which is responsible for service discovery between a specific versions of deployment and we have the cleanup pipeline and sweeper which are responsible for the garbage collection and what happened is so this is a real thing that happened about two years ago Nelson used to support ass backends specifically hashish core Nomad hash your core console and a hash a core vault then I came along and I said well I want to use this for kubernetes but I was sort of scared that well how much is Nelson tied to sort of being all in on this on this hash accord business and as it turns out that was not really that tight in I literally had to just implement the scheduler portion of where I could teach Nelson this is how you schedule something on a kubernetes and all the other service discovery and garbage collection mechanisms continue to work and if you go find you can go to the Nelson github and find this PR and you'll see that I pretty I mostly just changed the schedule implementation and I didn't really have to touch anything else and we've had other people who say well my organ wants to use a git lab instead of github where my org wants to use heat up us load balancers as opposed to something that's console based or something and they are able to implement those without touching the rest of the system and despite the fact that Nelson was developed several years ago for Verizon due to the way the system is architected we're very comfortable sort of shifting with the flow of whatever is trendy whatever is hot whatever your organization needs and so I think I'm biased because I work on it but I think Nelson sort of espouses are really at least a decent example of what it means to design for tussel and not to design for a specific concrete outcome so my takeaways the tech may be different between the 1980s and now ARPANET certainly was much smaller compared to the Internet and service worlds that we have today but I believe the core principles remain the same and are still applicable systems design principles when a 1980s are still applicable in the era of rapidly evolving and heavily marketed systems we need these principles to guide us to make good decisions and to finish off I hope your systems are simple correct fast and Tussle resilient that's all I have for today I don't think we have time for questions but feel free to come up to me and find me in the halls and yeah chat [Music]