scale.bythebay.io: Tim Perrett, Nelson: Rigorous Deployment for a Functional World
Recording: scale.bythebay.io: Tim Perrett, Nelson: Rigorous Deployment for a Functional World
all right good morning everybody thanks for coming we just got a couple of stragglers coming in so we're just waiting for those guys and yes so it's nice to be here I mean excuse me excuse me so yeah so I've been reading infrastructure engineering Uprising labs for the last five years or so and today I may be talking to you about this thing we built called Nelson so it's an open source open source system now or thankfully now it is and it's a sort of kind of metal Orchestrator like it's a it's a system that orchestrates schedulers and multiple data centers so and specifically the thing wouldn't be talking to you about today is how Nelson is built on a set of theater foundations and she has some principle basis and be covering these topics and how is she applied those just over what is essentially a very unprincipled domain of deployment so when I'm sort of preparing this talk you've got got to this thinking about this concept of rigor and the dictionary defines it as extremely thorough exhaustive or accurate and in general I kind of a sort of reflecting on you know is this a word that we can use in good faith to describe software and you know I think it's sort of fairly like broadly speaking like you know there are been some amazing things right like NASA run a rover on another planet millions of miles away there are doctors that do medical procedures inside people being hearts like you know there's some amazing software and then you know then there's like pretty much everything else and it just you know like I I kind of was looking at least and thinking you know why do we find these funny like we should be ashamed of ourselves for some of the things and and you know the fact is it's funny because we all do it and we all know that this is the case and so you know was preparing this talk you know thanks to Renard's have pointed me in direction of this quote and I think this is really like really very true and pragmatism is the opposition to principle on principle and so yeah anyway so so getting back to rigorous systems if we think about what a rigorous system would be like you know theory is a foundation so category theory come Natur accept theory lots of other different types of theory so things that are based in in math by and large and I think you know we don't sort of typically approach software like this and for whatever reason I'm not gonna go into the reasons why people don't do that but you know we I think we all know is that something for some reasons its familiarity some reasons it's a you know I guess scary names whatever it happens to be I guess yeah we largely don't do it but to build rigor systems we need to do it so I think just broadly speaking and are some interesting trends in industry that sort of this idea of building principled systems and sort of formally verifying systems is becoming more popular even some even some popular systems like Amazon s3 among others they even have like TL A+ specification models now and you know this is kind of promising like you know some to see more of this stuff in in commercial software and I think a lot of the time if we build systems based on on on you know discrete principles and based on of these these are otherwise principled or theoretical foundations you know we end up with more sane software you know which essentially means less to a.m. hey Georgie so I've got some completely unscientific graphs I'm going to use to illustrate my point so I'm you know please sort of bear with me here and ignore the fact that the lines don't go to 0 0 this is which was a question I got last time I use this chart so yes needless to say so so I think there's what these two curves like so principled versus pragmatic curve and I think over time we end up in a situation where you know principal if we have a sort of pragmatic approach to things you know the sort of pain over time is like significant because we end up with these sort of relatively for ball of mud pieces of software that like you know we iterate quickly we got to market quickly things are all good until about halfway along the chart and then suddenly you know it's off our pace slows you know sort of exponentially and so on the other hand I think rigorous systems are like you know you have to take this principled approach so there is kind of a degree of like I mean paying perhaps maybe the in great vertical axis for this but you know yet there's a great deal of thinking that to begin with but you're over your over time maintenance and cost of using them maintaining that system you know it's all good and so I think that these these systems or these approaches are kind of a you know they can both work but only one really leads to kind of you know a more a more consistent view over time and I think sort of on a probability distribution the principle approach is sort of more more likely to enable more people to ship these rigorous systems I think you know the sort of pragmatic approach over time becomes like a sort of paralysis for business so restricting their ability to diversify or change their own software and and you know this is a you know this is a discrete problem so I guess I'd like to propose that if we build software based on principles then you know we have we it's easier to refactor easy to change easy to approve and yeah over time our sort of bear in mind that sort of 80 or 90 percent of our time is spent maintaining software not Ryan green for software so so anyway on to the point in hand so when we talking about deployment today which is or by and large the sort of antithesis of rigor we largely use bash scripts bandages Python you know it's basically a ghetto of software development is the complete opposite of rigor I mean how many people here have a deployed pipeline based on Jenkins okay got some brave souls who actually rave their hands I'm going to assume everybody else is so ashamed of what they have they don't want to raise their hand no matter whatever a question I ask so so so yeah you know we've all been there right funnily enough whenever I talk about this I like to talk about that stuff in the third person like it wasn't me who did that but like I've also done that which is how I know not to do that so so yeah I feel your pain and share your shame so it's it's a yeah so so okay so what is a problem statement so the thing we're trying to solve here is that provision the applications is sort by and large till still too slow whatnot you're in bare metal or cloud the story often tell us five years ago it took us two weeks from Co complete to deploy you know which is you know two weeks is forever right so then I got the be end of my bonnet about four years ago and decided hey I'm gonna rewrite this and and so I rewrote that system and changed a bunch of processes and got it down to about twenty minutes twenty to forty minutes and so which case now and today it's about a minute so you know so this kind of improvement over time and secondly I think sort of run time traffic control systems are a pretty medieval like people will often sort of tell you about their canary deployments and how they take a machine and then then it's two machines out of ten or whatever they're doing or you know ten machines out of a hundred whatever it happens to be you know I don't want my operational decisions to be to impact my like right now every machine we run is like pretty massive like half terabyte of RAM with like you know 64 cores or more you know so it's like I don't want that to be my unit of work my unit of composition for these experiments I want to run so so you know we need we need fine-grained kapha traffic control systems and I don't want to couple our CI NCD systems so I don't want any sort of monolithic operation and whatever I do and you know I want to make sure that the system is not brittle the vast majority of deployment systems are exceedingly brittle and so even if people do implement least these sort of monolithic systems often on jenkins with post job after post job after post job many teams sort of become ignorant of the side effects it's like if you were deploying once once a week and hey now you're deploying 100 times a day it's like well you've got more artifacts suddenly you're producing like terabytes of artifacts a week you're producing like all sorts of bills and all the artifacts meditator this orb associated with all this stuff so it you know you're sort of success becomes a problem and it's a very interesting kind of a you know set of set of side effects that occur so as mentioned like you know with all good systems you need to build these things like multiple times before you get it right so this is my third or fourth time of doing this so I you know I hope it's it's right for some definition of right so and so lessons we had to do that like you need to automate every single part of the system any pieces you don't automate either get encoded into your organizational folklore or um or simply become a crutch for sort of automation avoidance and there's no testing of distributed systems locally yes even if you use docker you simply can't predict all the things that will go wrong scaling issues emergent properties and systems vast majority of large distributed systems have data borne issues so you know you're not going to tech them anyway you need uniformity and whatever you do that will be beauties beautiful snowflakes and an you know we can we can talk a little bit later about how we can handle those but last but not least and this is something meat spending some time talking about later is that you need automated lifecycle management if you are doing continuous delivery then suddenly you know you don't have this issue where like you know you've got one or two applications you have a thousand applications deployed and like or a thousand versions of the same application so it's like you need to do automated cleanup you need smart systems that know what's needed what's not needed how do you gracefully phase things out so I call this automated lifecycle management I think it's it's of an absolute requirement so goals we're building their system what are we what are we aiming for so when you use the minimally powerful components doesn't matter whatever whatever part of the system in the same way we would we make our function sort of just do one thing and you know how to be so minimally powerful I want this in my system as well so build systems just a build system deploy system just with deploy system you know so on and so on and so by and large this is interesting relationship between if you want to reduce the platform complexity then you have to increase responsibility of your users so if you basically don't trust your users you're gonna have a complex platform period so I think there's the sort of interesting thing with with this particular piece of software or this area of software that like there's an interesting relationship between this organization of Sociology and like how the technology is last but not least or so on this slide when the decentralize the process gatekeepers there should be no built-in there should be no tickets for filing deployment no configuration management this is all just garbage process that like makes no actual added value so why one it is going to codify all those things automate it make it self-service so if if you so just for example like make sure that if people are checking things in like yes suddenly you have get blame you know what change when you know what who changed it so you know simple things like that so this is a it sort partly partly process partly technology so yeah as mentioned they want to reduce the deployment time to two minutes or less I want to support multiple DC topologies we're gonna be talking a little bit about what my credential management and secure introduction later and I want to encrypt absolutely everything encryption at rest encryption why our privacy for all the things and there should be no manual access so if you rely on an sssh there's no SSH there's also just as a slight tangent this is funny thing that when you say hey guys we're taking away your SSH and people are sort of like you know what what will I do without SSH and then you say well what will you do when you log in with SSH there bill well I go and look at the logs and and you know and it's just people have these very rudimentary things that they believe they need this very privileged access for and you know these just these aren't things they're it much easier easier and better ways to do this so this is how we kind of arrived at Nelson and so the first question I usually get is where is it go stupid name and and and so and so uh as you can probably tell I'm British and so there's this great quote which is a desperate affairs required desperate remedies so firstly the fact that there's a gigantic monument to him in the middle of a London is is of you know you know he's a relatively prominent figure in British history and secondly like you know this dude is like yeah this dude knows what's up it's ops life all the way down he knows that there's like some serious pages where you go answer anyway so overview what did we build and so so unlike other systems right so people be another question we often guys like a how does Nelson compared to kubernetes and it's like well you know it just doesn't like like it's it's it's not that does like apples and oranges like there are complementary tools so I think of Nelson is like like if you're running keep today or like oh no mad or mess oh so whatever you're running you know you as a developer don't need to make discrete choices like how am i doing logging how am i shipping my logs in a way that when I've got multi-tenancy doesn't thrash the NIC how am i doing like you know costs for all my systems how am i doing like interference management and like if you're not sure what any of those things mean then like yeah that's like really normal so and it turns out that there's this kind of a there's a whole set of things that people don't know to be even think for and and that's fine that's why I want them to be like I wanted to focus on shipping their product which is the best thing for the business and we'll talk a little bit about operational responsibilities in a minute then Nelson is essentially providing set best practices for your organization as supplied by your operations team and all the application engineering teams get that for free completely based on github there was a kind soul in the community huge who's currently making work for gitlab which we're pretty stoked about but currently works for comm or enterprise and you can use whatever build system you want we literally don't care in our case we're using we have some state of the art runtime stuff with envoy we'll talk a little bit about that later and we're doing secure introduction with chrono chose from vault so every single container is completely unique we'll talk about that as well and and we it can target any data center running a scheduler so in short Nomad maslow's and there's some folks in the community who are working on in planning it with kubernetes so that's pretty exciting so workflow wise it kind of looks like this so you put some put you put some code up to PR review and by the way we'll be going to the graphs and stuff in a second sir ed will put some code up to review obviously it's so awesome it gets merged then gets built in our case by Travis Travis then a does the build produce as a container sends it to the staging registry which is C in this case notifies github says hey new release software version 1.1.2 or whatever it happens to be and then github then notifies Nelson simply with the webhook so in this case we don't care what what CI system you're using it doesn't matter Nelson will then notify Oh github notifies Nelson now some figures out is this something we need to do something with and then if it is it does any of the necessary deployment so now some sort of essentially pretty decoupled from any of the other things in the system and it only has this sort of coupling with with the what's of source repository which in this case is github a common enterprise so in the ongoing theme of completely unscientific charts the sort of PI responsibility so I think that there there's this common misconception that you know ops don't care about applications the only dad's do and I think that that's I want to sort of dissuade that would dispel that myth because the operations people do care about your applications it's just that we care about all the other things so like users have concerns like where is my application deployed was it was the health of my application you know these kind of questions which are like application centric operators on the other hand I basically care about everything else like for your given application was the capital the utilization of hardware about security what about logging what about QoS what about sharing interference like these are all the kind of things that operators care about now they make a huge impact on your application and so I wanna I just want to sort of say it's all part of the same high but ops and dev care about care about the application just the way that we view it is different so so from a high level perspective Nelson has a set of features that allow us to do this so we have this sort of very simplistic manifesto and yes someone is going to say oh god not another map a ya know file and in part yeah I are like I hear that you know it's it's a you know it's not ideal but from from a process perspective it ends up just being like a logical definition and we'll talk about that a little bit but by having that it allows Nelson to focus solely on like this with active lifecycle management we know what's needed all the time so we know what to clean up and what to garbage collect Nelson also has this optional routing control plane in certain scheduling systems you want to use that in other in other systems you don't so we'll talk a little about that too and we have comprehensive auditing so no matter what gets done in the system deploys clean ups etc and Nelson there's an auditing function you can go and basically talk to that and say hey like what happened why did my application get get garbage collected or whatever happened and Nelson will sort of be able to say stuff like yep it was garbage collected because all the traffic was shifted to the newer version and it was superseded by 1.1.2 or something like that so that that's pretty neat and we have the sort of extensible you know ways of different arbitrary scheduling around more artifact systems and that's based on free monads and co-products we'll be talking about that later which is pretty fun so I started this talk talking about rigorous systems and you know to have these sort of theoretical foundations you know for Nelson that's primarily two things which is so you know this idea of graphs and this idea of co-products so graphs we have a very nice library called quiver which some of us worked on rune or sat in the front here I poured that from from Haskell so the FGL from Haskell was ported to Scala with it so now it's called quiver and so we have a very nice library for this so it allows us to model several different things so so graphs have this idea a vertex so a vertex is simply vertex end with some some type level L and then we have edges so from some from vertex a to vertex B we have directed edges and it turns out well you know there easily model evolved you have an edge with edge with a type or a laboured add label edge with a type and a type for the label so if we have many of these we create graph now it turns out that this is very nice property about graphs which is that they're annoyed so if I'm moving too quickly if I'm sorry if people don't know what my notes are I'm not going to go into what they are what they're not and but yeah please look out but it's a very nice property very high level has these are these function signatures in most implementations and so this allows us to basically merge the grass so yes someone will pull me up about associativity laws and stuff like that so so anyway that's a secondary thing that the from the high level takeaway here is that like if because these graphs are monomial we can merge them all together now if we look at what more applicable graph when we think about deployment well we always talk about our applications just like their graphs except that like we never modeled on this graph so when I think about my application as a user I'm like well I need like B and C to work because I'm a and so we can sort of logically say that a logically depends on B and so a logically depends on C which depends on e so from an operational perspective I know that like just from a human perspective I know that well a sort of transit Li depends on e and so if we can make these kind of like rationalizations ourselves then why don't we just codify that into the system so the system knows about all these transitive dependencies and so we actually already do this in our dependencies for a local build this is exactly what IV or either any of the other dependency management systems is doing it's just that we just don't do that at runtime like for some bizarre reason so if we take off the if we take off the labels there and we add some more sort of you know deployment ready labels you know we can say okay this particular version of an application requires this other version of an application and we can kind of you know it's build up this graph and so the nice thing about this is that from your applications perspective you only ever need to tell Nelson you know what you depend on so if every single person in the system or every single agent in the system every single service is just telling just telling Nelson what they depend on then all we need to do is essentially sum of sum all the applications in all their graphs and we build the whole world and so it's like you take all these little partial graphs merge them all together and then you know what the whole world is and if anybody sort of like squinting thinking like really does does that work and yeah it does and so we can talk about some of the interesting things about you know there are some details about how certain things work but we like this allows us to sort of derive several useful pieces of information so firstly it means that like if you have different vertices on the graph that aren't connected there's no relationship so there's no routing so likewise if those if those vertices aren't connected that means that system a can't talk to system X so therefore never provision credentials for a system X because it will never need it and so it turns out that there's you sort of very you know interesting things that we can derive from the fact that this is a graph and essentially just leaning into what it is already because we do these things are you know continually so for example many systems lean towards systems like you know like open tracing or Zipkin or something because they don't know what they deployed and then they look to tracing to figure out okay what are the relationships in the system and it's like you know what they are because we built the system it's just that we just forgot to codify it anywhere so instead of instead of doing that if we just say hey when you're building your system why don't just tell us what it is you were using and then you know we'll you know we can figure out many other things and make a sort of safer and more secure system so the entire world view falls out of this union of all these sort of discrete graphs and there's a sort of nice sort of aspect of this which is that the edges of the graph which were formed the ingress to the system so those things like typically like anyone familiar kubernetes your Inga's controller is essentially the edge of the world likewise you're a load balancer as the edge of the world these are just different systems that allow inbound traffic to the rest of the graph and so we use those as kind of like an anchor point for the graph or so route of the graph I'll take questions at the end if that's okay so so I won't talk about a little bit about life cycle with this stuff so life cycle is something we we typically in the like most companies end up doing this would like some hacky Bosch grip door some like Python script and a once upon a time we used to do this with Python just a Python script too until based on an AWS bug it like nuke to hop off the system because it wasn't expecting an empty array so that was pretty much the time when we decided hey you know we should really like codify this in a more formal manner so anyway so this is a great tweet from one of our goes he tweeted this you know a couple years back this is Amazon system in 2008 so like many large enterprises like this is 10 years ago nearly think about how more complicated their system is in all likelihood 10 years from then this is a gigantic graph we also have a gigantic graph and it's like many large systems have these big grass and it's like for some reason we just never like we don't we haven't looked to basically do this automatic graph pruning or handle these like automated lifecycle swings and accept what we are you know and actually codify or audible automate all the other cleanup and an Associated tasks with these things so so one took a little bit about the lifecycle of a given application that gets deployed with Nelson and and how we do garbage collection to try and avoid a you know running all these things over time so an application is deployed but Nelson goes through a set of states so it's basically just a small state machine and that we have internally so an application gets deployed so in this case it moves from pending to deploying to warming and warming is kind of this interesting thing by default it's based on console so we have these health checks and the idea of the health checks is that an application can tell us when it's ready to receive traffic so for example and if you have an application that boots up and it needs like five minutes to do a cache warm or something then you know it can basically stay stay in the warming State and then just indicate oh okay now I'm ready to receive traffic and and so we delegate that to users and that's sort of a nice thing but by and large most applications get deployed and then they move immediately to the ready state if they don't have any any such at any sort of warm up things and so over time and it doesn't matter whether that's today or tomorrow or next year you know the application will eventually become garbage and we'll talk about why that is in a little bit but then there's this neck the next state they typically move to is garbage and obviously then they get cleaned up and become terminated now this is sort of other workflows it's kind of interesting which is decided application so over time by the way I still love the main antenna size specs of all this because it's something we never really address typically what happens is it like you know I produce an API then like you know maybe a year goes by maybe two years go by whatever it is and then like I'm on like version three of something major version three and then somehow someone else is still using the old version of my API and I need to do some like schema change or something that's like some now incompatible change to the old revision I can deprecated my system and at that point it basically triggers it Nelson looks through the graph and say who's using an old version and then it goes and says tells everybody the people that were using that system you gotta upgrade like like that thing is deprecated and so this is sort of another nice thing that as an API producer you now know who all your consumers are and this is actually typically like something that most people run afoul of it and in large debate systems simply because of Conway's law like like in a large company you you end up in a system where you produce an API and you might have five API is maybe three or read API to write API is whatever and someone uses your API in a way that you are not familiar with or or they do something you weren't expecting essentially and so this idea to be able to like know who you know who your consumers are sort of you know inverse traversal of the graph from your position it is really useful so so yeah so this is basically sort of the internal workflow about how Nelson decides you know it sort of traverses the graph and figures out like you know whose garbage basically so there's sort of a fundamental tenant here which is that like every single application is is has the sort of TTL like we will delete everything eventually like period and so it doesn't matter whether or not it's a you know today or tomorrow or next year you know some at some point that thing will not be needed and so based on this concept there like everything is running on this extended you know this essentially TTL we can just bump the TT hours to retain the application and then if we don't want to retain it well we just don't want the time we just don't bump the expiry and so there are set of policies that we can use to evaluate this but once we've you know once that thing has expired then it just gets marked as garbage and another process will come and clean it up and so one of the policies we have is this idea of graph pruning and so because we know the graph as we were talking about up to now so in the event where the producers of sister meth they roll out a new version and that new version doesn't need the G system so Nelson would be like oh well G is basically orphaned now so it's not being used so that can also be cleaned up and so when we do this we can shift all the traffic from F 1.2 to 1.3 so we can sort seamlessly know that you know we moved all the traffic norm was affected and you know once that's done the F 1.2 becomes garbage and then eventually gets terminated so that's very very nice like that there's not um you know this is sort of not comment and it's a very nice in practice it becomes very nice because you never end up in those sort of discussions where someone's going around saying do we really need this system anymore and then by the way you've got like 100 of those systems and it's costing you like thousands of bucks a month like you know just to run those things so so so you know it sort of does away with those things another common kind of workflow typically for data science is like you know objective model comparison so I might have like a my production model which is like you know three point four and then I've got like you know four point one and doing some other stuff under one objectively compare and contrast those models you know at run time you know so it can do that too and keep around all these different versions to allow you to iterate last two features so on you can see these are simple boolean algebra is essentially so this is pluggable and you can pretty much pretty much do whatever you need to you would like to do there so we mentioned security too so trying to stay on the theme of graphs and all the all the interesting things that we can sort of derive from the graph and by and large I think security is that thing that everybody wishes they did better because they kind of like know that they or maybe they rely on like you know promote offense or whatever it is but there's like many janky things that people do you know pertaining to the security in it and I I would propose that those things when you derive them from a graph like a sort of authority of graph about you know what the system should be like the desired state over the world you can actually a sort of you know have a safer system and so some of the things that we derive from this graph is this idea of deriving security policy not pro kleh nearing and so typically what we sort of do you know is what many security people do is that this application should not to this application or something like that right and and like saying that is basically like useless like like because someone somewhere has to then go and implement that and so if that policy ever changes or there's a change in the relationships usually that stuff doesn't get updated and it's it's Allah so this bullhorn thing like it's like an afterthought in many in many systems so instead Nelson can actually generate these policies and it based on the graph because it knows that oh you used to use this one or now use this one okay yeah remove that policy now it's generating the actual policy that will be used to derive your credentials so that that's that's super useful so basically getting like policy clean up for free and now because we know these are internal external dependencies for any given application it means that we have an easy way to audit the system both inside and out it's like oh you need to talk to a vendor oh the vendor needs credentials okay great well they get provision to vote you also have to declare them so otherwise they don't get put into your it put into your security policy so you don't get access to them so you end up in the situation we get a single point that knows all the things that could talk to all the things and so you can run a simple audit this is really really nice and so from a practical perspective when containers are scheduled like every single application instance ends up with these unique credentials so unique certificates unique database access like all that stuff can be generated per container so if you get compromised only to the blow away the container and you know then you haven't lost anything because those credentials don't work anymore so these kind of aspects of a really nice like if you go with this completely arbitrary completely ephemeral you know sort of security model based on there's a desired state of the world and you know you can automatically provision a lot of these things without result with a minimal of hassle now just on top of that like if you know that you're doing this if you want to use a service mash like envoy or a link or D or something like that you can just do transparent MPLS for everybody because you're you're automatically generating the certificates again per container and because you're doing it per container you can add all the extensions and all the other things that you would want so if you want to do like you know sni or any kind of like you know old name stuff for your security depending on whatever you want to do you know it's a breeze to because you're just generating it for a container like you're generating that stuff anyways so yeah this is really cool I know this is like high-level if anyone wants to talk about that stuff like after the fact I'd love to I think this is one of the really interesting parts of doing this kind of system so so I'm going to move on to another sort of component that's graph like in Nelson or indeed is a graph so we have this concept of namespaces so this is an optional part of the system by and large it's if using the control plane part of Nelson for routing this is in play if not then it's not but it's all very interesting a interesting thing that we that we built that not many people are doing anything like so so so a very very high level I'm not too sure how many people familiar but so we have this sort of shared young substrate you have a set of machines and then you run a you run a scheduler on top of it so you can address all the machines you know simply as a pool of resources pretty common so tools like nomad tools like kubernetes tools like mezzos you know these are these are resource managers that allow you to address this pool of resources you know as a single homogeneous you know available set of resources so for example instead of saying I need X machines you simply say I need like you know I need 400 gigabytes of RAM find on the cluster for me you know I don't care where I am you know these kind of these kind of things so what we did is we basically built the sort of sab namespaces on top and so this is a completely virtual concept there's no you know there's no physical thing here it's simply it's simply like this virtual construct to say okay this pool of this this this pool like not pool even not even related to the resources but this group of things is is this this route this group of things is another through now in this diagram I've used dev QA prod but like in actual fact like these arbitrary like you will you name them whatever you want but it sort of most illustrative for most people to kind of think about it in this way now the sort of interesting thing about these things is that the namespaces they're also trees which are subsets of grass so now the root thing what we call the root namespace the root of the tree in this case you know let's say it's QA now I can define another sort of sub namespace under that QA unstable and I could even defy like another one could staging and then I could have another one specifically just for me and so if we're doing resolution about like discovery who do I need to talk to there's sort some very interesting things we can do in the sense that because I depended on a particular application I can be like okay does that application exist in my namespace so typical use cases for this are like like this is a hilarious thing about most dev or QA environments people so they want to try their system but they want to try their system with the rest of the world that doesn't change and so when everybody wants to do that you end up with a system that never works because like everything is unstable because everybody wants to try their thing with everything else but everybody is doing that so it makes the whole system unstable and so subsequently we realized that there's this kind of like multifaceted aspect to it which is like what you want to do is you want to say okay deploy my thing and I'm just gonna deploy my thing in my special namespace and then it's gonna resolve up to see okay is the thing I'm looking for available in staging oh no it's not available in staging is it available in QA oh yeah it is so then you get this like sort of multi faceted view of like okay I'm using my application just with special custom one that I deploy with the standard QA one of everything else and so because these are completely arbitrary you can have as many as you like every dev could have their own namespace 51 at its it's like completely virtual but in terms of like doing discovery and stuff it means that people can have like these of sandboxes just to do whatever they want and it's just completely a you know it's just shared on the same cluster without any without any sort of concern about you know stepping on someone else's toes essentially and so if you want to deploy even if you have a tight coupling with another system you can even deploy that you know on their behalf in your namespace for some special testing so all these things are really nice and this works for both services databases any other dependency we have in network and you know so it turns out that it's it's some really nice property so in terms of discovery and routing I think they're they're like pretty related concerns but sort of important to understand why they're different I think discovery is simply about taking this logical name like like accounts or software update or like users or something like that like some kind of logical part of the system and telling you sort of what that fully qualified name is you know where specifically is it and routing on the other hand is sort of telling you about how to actually make you know move traffic to that particular endpoint so for example convert it to IP port names or something like that and there'll be various distinct revisions you know all of this saying inner communicating systems I think that's the difference between you know discovery and routing so in our case if you are using the control plane in Nelson this is we have this discovery protocol which we call the lighthouse protocol and so I'm not gonna talk too much about this so and just get forward a little bit run a little bit time time because apparently I'm slower than I anticipated but and this is online I totally totally recommend you go and look it and I've done a couple of talks about this before so anyway long story short there are some very nice things you can do with them with Nelson and its route and control plane which is you can say okay I want you to shift traffic but I want you to shift traffic in using a power curve so like for example I can I want it over over over 50 minutes move all my traffic using a power curve or use a logistical curve or whatever it happens to be and so this sort of curve based traffic shifting allows you to move traffic in a way that makes sense for you over a time that makes sense for you if you're a front-end system and you've got like you know ten million customers like obviously you're not gonna move that thing and like you know five minutes you know it's gonna take a long time or much longer time to do that without causing a you know you customer impacting outage so anyway yeah this is documented online at super if anybody wants to talk about it I think this is really interesting and doing this stuff it allows us to that the namespace and things allow us to do this kind of traffic shifting without without impact impacting users so I've got some links at the end can I come to that thank you yeah so yeah so workflow so workflow is kind of this idea in Nelson it's like it's sort of like how the sausage is made how we how we hold all the things together and so by and large you know we have this so internally we have like a set of executives and this executive basically just pull off work on the queue and then they execute these workflows so the workflows you know they're also this kind of graph like a you know graph like system and so they do this thing and then they do this other thing you know and so on and so on and and you know it turns out that you know workflows sort of conceptually are graphs and in other more generic scenario is you know internally we do actually you know model these workflows you know and rule engines as graphs but in this particular case and you know we made a specific decision to use a discreetly less powerful thing and that power less powerful thing in this case was co-products because what we didn't want to do is we didn't want to fall into the same trap in Jenkins which is where you can basically build these complicated DAGs of things and and so forth so basically choose once again to use the sort of minimally powerful thing so I had some background on Co products and things so I'm just going to skip through that because we're running a little bit tight on time but based on the audience I think most of you sort of familiar with Co products and that we essentially built this algebra and we built this algebra internally which takes all these discrete parts of the system like how do we deal with docker how do we deal with the scheduler how do we deal with these all these other things and you essentially end up with like co-products of co-products of co-products of co-products all these discrete programs so our program is simply a program of programs so when we're actually building the workflow the workflow reads like almost like English text in some senses is simply a fork on french and it says okay let's do this thing okay right they look right the policies to vault and then I've removed some stuff because it wouldn't fit on the slide there's a bunch of other things then launch it and so it simply just says do this do this do this do this and so by doing by by having this co-product approach instead of doing a graph there's no complicated DAGs there's no complicate like people don't build logic into Nelson that would be an abuse of the system so it's like it's specifically like disables that kind of that kind of thing so this is super interesting I show this only to show that it's like it's very straightforward so please go and have a look if anybody's got any questions on how that you know workflow subsystem works it's really interesting so you know this is definitely like a non-trivial system right we didn't build this thing overnight and right now we've got fairly tight integration on the open-source version with hashey stack right now there's the community or adding support for kubernetes in SEO which is awesome I'm stoked to see that happening and also good to see that even people who are using other systems recognize that they still going to end up in the in these in these same problems where they can't clean up and then they need some kind of lifecycle manager and so yeah so yeah so we have several challenges I think it's a I think if you're a tiny organization this is probably overkill to be frank but for many for if you are running a scheduler or you have more than 100 nodes in your system you know to the point where you are thinking about running a scheduler like please consider like you know like all these side-effects and all the other things we've talked about there you know sort of running up into a proper that could cause you problems so so in summary using principle systems systems like Nelson having been built on on this of you know foundational concepts it can provide some rigor to the rest of your application and I think you know not having this manual housekeeping being able to choose whatever CI you want being uh secure deployments all these nice things that fall out having graph-based structures you know they they they get you a lot and they don't cost much and so yeah anyway with that thanks very much I know run tight on time there's a link here at the end so if you're interested in documentation it's all online if anybody's got any questions I'd be more than happy to talk to you and I've got some links in the end of the slide about you know other projects and stuff that related like the graph library I mentioned and all the other things we pull it to Scala so I'll post them online and put it on Twitter so anyway thank you very much for listening and I'm sorry run a bit tight on time [Applause] you mentioned documentation that that link yep developer documentation yep API on tiny more like comment with your slides I've gone into the theory of except more in-depth that's a good question so we do have some stuff on there depending on your questions it may not be there but if there's anything new yes so that's a really great question so so we did have to make some some choices and so at the end of the day you have to kind of boil down to something that people could understand and so we did originally our original aspect of this system had a much more complicated system like the one you're talking about and in the end we just decided hey you know what like the system is becoming complex so as I was mentioning in order to sort of reduce the platform complexity we had to make a trade-off and so the trade-off was hey the users they have to basically follow semantic versioning and the interface is essentially defined by December and so it's not ideal like we tried to retrofit some stuff like like our plan is to basically retrofit it so that the contracts for the contracts for the systems themselves was basically but you know published like we've made a project called remotely it's on github you can check out but like it was an experiment and the idea was that what we wanted to do is to basically like have compiled time verification on the contracts and so that if you were publishing December then like you could actually say okay like system X upstream said that they had this contract did they actually have it and we didn't get too far along with that recurrently we basically just rely on December and essentially continuous testing to make sure we did the right thing but yeah it's it's a it's definitely a trade-off so for security and stuff we like pretty heavily on that like depends on if that we might need to talk a little bit more after it but by happy to thank you very much [Applause]