Scale By The Bay 2018: Adelbert Chang, Nelson - Functional programming in system design
Recording: Scale By The Bay 2018: Adelbert Chang, Nelson - Functional programming in system design
you yeah so my name is an alert Cheng I am a lead data engineer at Target do sometimes it feels more like I'm a distinguished animal engineer given amount of Yamma laughter right day-to-day but today I'm really excited to be here to talk to you guys about Nelson deployment system I've been working with Tim Paragon who's also in different room on for the past year or so so let's get started so earlier this year I gave a talk of the same title and I confess the same abstract at the North East Scala symposium as well as scala taiwan for so if we're in the original talk I covered some things that I won't cover today so if you've watched it before Dobby new stuff today and if you leave this hog wanting to learn more there's that talk and Tim pair it also has a talk that he gave at a meet-up last year so dad talks can cover stuff like traffic shifting graph pruning automatic shutdowns and some more discussion on implementation of nelson widgets in scala which is right there at scala conferences for this talk we're gonna cover more a system-wide view as well as more focus on how users interact with Nelson and then also how operators are going to deploy and maintain also goals for a talk I hope you guys can leave curious about Nelson and maybe even if you don't wanna use that work maybe you want to steal ideas from it or offer ideas for us to steal so please talk to us afterwards so those who know me or follow me on Twitter know that I've been kind of a functional programming that for the past several years I treat constantly about it it's been about six or seven years now and even as as recently as last month I give him an interview for this conference and I talked about a bunch of different stuff but even then to take away was apparently I still find I still like functional programming which kind of begs a question why am I talking about deployments which is notoriously known as kind of a terrible place to be in and also like on a reactive fact which when we do have like a separate functional programming track so to answer that I got into functional programming because I wanted I was looking for a way to program that would give me confidence and would sort of make me happy day to day I can like write my program leave for a couple days come back and still know happening and it didn't feel like I was getting that with so does mutable imperative code and I said that's when I got it a functional programming again been sold on it ever since but I noticed that as I hung around with like other programmers who were interested in like software methodologies and stuff we would often build these really nice pretty programs we were to construct them strongly typed well tested and all that good stuff but when it came time to like build or distribute or deploy their programs people had kind of different sentiments than they did for actually writing their code right so no matter what companies I've talked to you or I've been ad tooling get em structure seems to be like kind of almost universally hated and that was kind of weird to me all right that we had a lot of catching up to do with tools our languages were great we had a lot of advancements with languages Brian Kantor will give a great talk today about how rust has changed the state of system software but in terms of like tools in infrastructure I think there's a lot of work to do there so for me then this was kind of the same problem that I had working with a mutable imperative code where I wanted to gain more confidence when it came time to say distributed deploy my code and I didn't always want to have to move in between like a yey programming phase followed by like oh no I have to actually deploy this thing so with some programming hubris I was like all right is that gonna play No oh yeah so with some programming hubris I was like alright next next rule I take I'm gonna do this myself how hard can it be famous last words so a couple weeks ago your arm in ski tweeted something if you can't tell I use Twitter a lot but the arm in ski of Jane Street tweeted something that I thought was really interesting which is that one of the most important perks you can give to your developers is a really good dev tools team and especially having worked in this space for the past year I've found that tooling an infrastructure have had very they have very profound but kind of hidden effects where these are things that people are using day-to-day and people sort of either get used to or get desensitized to sort of like the terribleness of these tools but they have very large effects I've seen so like one of the one of my favorite examples for this is for like running builds when CI or doing experiments where if they're slow for an or buggy you do well thing happened as developers going to kick off a build or someone's going to kick off an experiment they know it's gonna take a while usually like I say 15 20 minutes they go do something else for a while oftentimes much longer than it takes to actually run the build or do the experiment they come back later have to remember what they're doing reap age into context realize like the Brill Brookwood experiment didn't go as planned they kick it off again and they go do something else I've seen that tons of times and so the amount of wasted time on like bad tooling or buggy cooling isn't just the time it takes to make that build or run that experiment it's like the time it takes for this is the person to come back and re remember everything they were doing similar things four deployments if deployments are scary or unreliable oftentimes I see like people become scared of deploying things they deploy less frequently when to deploy less frequently the next time to actually go to deploy something it's like a bunch of different changes things go down they get scared people start inserting like manual processes and it's just like a vicious cycle what I want to try to achieve are people who have sentiments that like this for people who aren't so much scared of deployments or or dreading deployments but they're sort of like yeah I'm gonna deploy today whatever it is just part of my day to day and especially when you get into like a continuous or like a really nice deployment cycle it's nice to be able to say like I'm gonna write this feature today and then it's gonna go out in like 20 to 30 minutes so that's that's the kind of state I want to achieve and I think Nelson serves an interest sort of an interesting data point to achieve that so to start getting at that goal I asked myself why are people scared of deployments and to answer I currently have is because deployments are often manual they happen infrequently and they have cascading effects and to drill into that last point by ask cascading effects I mean a lot of times deployments aren't going to be isolated right when you deploy something is not just like when you're developing code you're just on your local machine and you don't care about the outside world at that point in time and nothing about said nothing and now several is happening it's gonna affect your development but when you deploy oftentimes it's being done in a way where everyone into org or everyone across your micros services architecture or whatever is going to see your deployment and if either you messed up your deployment or de been handle errors correctly that can cause cascading failures or other cascading effects and just to reiterate often times when things do have cascading effects and things go down people get scared they deploy less frequently when they finally do have to deploy the changes that have been we're gonna insert like manual processes like all right let's all flock out 3 hours over time I walk through changes or something like that and that just slows things down even more and it's all terrible so to address so so my theory is if you can address if you can solve this cascading effects problem we can at least make deployments slightly less terrible than a lot of people think I think it is so the premise that I'm gonna present and then Nelson more or less operates on is at the root of Cascade of this cascading of that four element is mutability mutability as always Jimmy causes problems so let's talk about that what do I mean by mutability in the context of deployments so here pretend this is like a microservices graph each circle is going to be a micro service that does whatever and then it's annotated with the version so and then the arrows are gonna represent dependencies so in this case services B and E depend on service D for whatever functionality then the owners of service D want to upgrade it from two point one to point two and oftentimes what happens it'll do a rolling update or something like that and at the end if everything goes well the service D 2.2 is going to live in the same place of where do I live does this is an in-place deployment now so this is why I meant by where deployments aren't isolated where D has made a decision on their own to deploy something B and E have to be ready for that deployment whether like whether or not they're ready to have to be prepared right and as is commonly as is common some people aren't gonna be prepared for whatever reason and things go down right things go down and this is because again everyone is sort of context switching all at once and I attribute this to one way I like to think about this is like similar to how we have problems with mutable data structures in the concurrent environments we have different teams maintain their own service to operating concurrently they're doing their own stuff but they're all working on a a mutable data structure which is a service call grab to serve as the pendency graph and when someone is mutating in that data structure everyone has to be ready whether or not they like it so how do we how do we solve this problem given that lens in a programming case we're going to try to just alright let's try to limit immutability let's try to make things as immutable as possible so let's talk about what an immutable deployment model might look like so same service call graph service team D is wants to do an upgrade but now this time instead of doing an in-place upgrade we're just going to deploy it somewhere else but the key difference here being that the old version of D is still there and services a and B are still pointing at the old version of D right so this is now more isolated than before so this now begs the question all right how do services a and B and other dependents actually observe the change in B so in the case of E or even B we'll get to that later in a case of E as an example it's gonna do pretty much the same thing as D it's going to redeploy itself and make an explicit choice somehow saying that I wanted to pass thirsty 2.2 this time instead of true that one at this point again the old version of e and a new version e are coexisting and those familiar with like red black deployment models this is essentially the same thing right if the new deployment looks good then we can shut down the older version and we roll forward if the deployment the new deployment looks bad for whatever reason you can shut down the new deployment and you're you still have two old one everything is ok there let's assume things are ok we shut an old version we roll forward at this point in time the new version is depending on a new version of Dien the old version of B is still depending on older version of D which is completely fine right sort of the point of this exercise was that we weren't gonna mutate the world for everybody we were going to have people pick when they when they want to upgrade with their writing so let's say B is ready B is going to redeploy itself explicitly making a choice to depend on service D 2.2 and because B itself has a dependent who does service a a is going to redeploy itself and make an explicit choice to depend on a new version of B and then again at each point in time there's a new version and an old version of the service the old one of the new one looks bad you can shelter the new version and you're right where you were before and if the new one looks good then you can shut down the old version and roll forward so in this case let's say everything is good a shift on its old version that frees up B to shut down its version and then the whole version of D it can be cleaned up and that's our final service call graph so this way of managing services for those familiar is very similar to how persistent data structures work in programming languages and just like how persistent data structures give us a nice vocabulary for talking about changing data I think immutable deployments give us a really nice vocabulary for talking about service deployments so how do we now actually implement this right it's easy for me to sort of stand up here and talk about would it be nice if we had this model but we have to actually implement this for this to be useful so we can start by identifying that the core of this idea of immutable deployments is the idea of explicitly tracking version dependencies so we have introduced a new problem with immutable deployments that wasn't existent in mutable deployments which is because we can have multiple versions of the same service running at any time and moreover we can have more than like even more than two we could have like five different versions of a service running at any given time there's no longer like a single place that a service lives right oftentimes you'll see like we'll have D prod whatever com now we have to talk about D at a specific version under specific namespace or something so we have to have some way of explicitly tracking where each version lives as well as connecting different versions of equipment service to different versions that potentially different versions of its two dependency so one of the ways we can implement this is in the deployment system itself since every deployment has to flow through your deployment system anyways it's in a good position to sort of track and publish information about what the state of the system is now oftentimes service discovery is going to be a separate notion from deployment right a lot of times deployments are going to be like here is a jar or here's a dock in there deploy it somewhere and I'll figure out how to connect to my dependencies myself usually in a mutable way and so because of that the coal graph is invisible to the deployment system so there's not much the deployment system can do what we want what we want is for a deployment system to be call graph aware which basically means that every service that goes through our deployment system has to declare the name and version of itself and of all of its dependencies and from there to deployment sister can then begin to build the graph the internal graph of all the services and their dependencies as well as begin a published service discovery information where it says where if say he says I want version 2.1 of d3 points as it can first confirm that 2.1 of d exists in the system and then also potentially give e or put services company of information about d in a place where you can pull it down other stuff that you can do once a hat once the deployment system has full knowledge through graph has also do things like traffic shifting graph pruning lifecycle management stuff like that so this emphasis on immutable deployments by ratifying a service graph is the core of Nelson and when I think of Nelson I like to think of it as starting with the idea of immutable deployments and taking it to its logical conclusion and everything else we build an else in terms of tooling or sort of things to help make things more economic now working with this new model right you get some stuff that sort of comes for free when you're immutable deployments an example being like you know where a given service lives so we do lose some stuff but we built everything Nelson has interpreting or because we're gonna we're doing a mutable deployments we're going to do it right so that was a conceptual basis of Nelson now I want to show how Nelson actually works from users and operators perspective so as I'm discussing those keep in mind that Nelson is doing this immutable deployment model internally so Nelson from a user's perspective first things are going to start with two files so the first file is what we call a Nelson manifest file it is yeah well I apologize I'm part of the problem it is a check so this is gamma files gonna be checked into the root of your repository and it essentially details there is someone to he'll like if you're familiar like a travesty amo file or a bill kite file or Jenkins file this is sort of like an nelson equivalent where I'll describe a couple different things I have the bare minimum of a file here we have several other fields you could put in here but I needed to fit on a slide so the manifest file has three main sections the first one is the unit section which describes what to deploy and I say logical here because we haven't said anything like what we're actually deploying right there's no like docker image or path to a jar dam acupoint you I'm just saying I wouldn't deploy something called foo with these dependencies and these versions the second section is a plan section which describes the how to deploy it this will include stuff like how much CPU to give it how much memory to give it how much disk space how many instances and stuff like that we also have a blueprint here which I'll talk about in a bit you can think of this as essentially how you want to at very high level how you want this where you want this thing to be deployed so in this case will be I want this to be deployed on a machine that has GPUs on it and then finally at the last section we have namespaces for just actual where to deploy you can think of these as kubernetes namespaces which is true for the kubernetes back-end of nelson i'll talk a little bit about that later for nomad I believe these are more or less logical maintained internally and in this section is also where everything is tied together we're saying for the dev namespace I'm going to deploy foo with this game plan the second file that you're going to work with is called deployable file and instead of being check-in this is going to be attached at a per release or as of recently a parade github deployment basis so this sort of ties together to what the logical definition of what to deploy with the physical so here I'm saying I'm deploying the thing with logical name foo this is going to be the version of myself and this is the actual docker container I want you to deploy that corresponds to it so starting with those two files users are going to then make a deployment that should be capitalized actually because we're when we say make a deployment I mean make a github deployment which is like a github concept older versions use github releases for the same reason nelson enabled repos also have a web book installed that correspond to whatever hook they're gonna use so in this case we're going to assume that the repo has a deployment webhook so a deployment is made it can be done manually and you can attach it deploy gelth all yourself or we provide a tool called slipway that will makes it gives a easy command-line interface for you to do it and so you could also do it through in CI if you want to do continuous deployment for instance but once the deployment is made the webhook is gonna call Nelson Nelson is going to based on the kana Center web hook go back and go back at a repo have fetched and manifest into employable files pars it figure out what we're how to deploy it and then from there it's going to deploy it a scheduler and your deployment into its graph and that sort of job done from the users perspective from then on the user is only going to interact with their workload using the nelson command-line interface which i'll talk about in a bit and using other tools like Cabana or Performa to look at logs and metrics and stuff like that it's assumed that whoever's operating your infrastructure has those tool set up for you other stuff Nelson is going to be doing in a background that's sort of separate from the deployment workflow is stuff like automatic traffic shifting cleaning up deprecated workloads and stuff like that so I can show you a little bit about what the CLI provides so sort of the most basic thing that you can do is to ask Nelson to essentially promote a certain workload from a dev environment to a production environment for instance by default everything goes into whatever your administrator has configured to be the default namespace switch off in a dev environment and then once you've determined that that work looks good and you want to release it down to your world you can say you can tell Nelson to commit it into a QA environment out to a production environment and Nelson will execute it essentially the same workflow as it did for the default deployment but for just in a different namespaces time one of the several implications of this of course is that your workload is expected to just do the right thing just by moving from namespace to namespace you can do some runtime inspection of this the environment variables that Nelson injects so you can put as expected that say the way you pull secrets or the way you do config management can just happen transparently from the user space other things you can do you can figure out what our current what's currently deployed based on the namespace or a data center there are other filters on this as well such as give me only stacks that are failing or give me only stacks that are that are healthy so this gives us a high level one of the more important things to note here is we have a gooood there given that gooood you can also ask Nelson to what we call inspect essentially that gives you diagnostic information on how Nelson itself interacted with your workload so the status history and the logs here are on logs for your application to launch with how else interact with it so you can essentially see if as if Nelson successfully deployed a workload or if it fail to deploy for whatever reason you can also see what time it was deployed what plan would namespace it was using yeah Nelson also lets you do some basic runtime information so this is the Nelson Stax runtime command that will give you just health checks for your workload where does health care come from those can be written inside that initial in Nelson manifest file so you can use Elsa's next runtime to get that kind of runtime information but beyond this Nelson is the tool is intentionally not intended to give you like application logs through like how cute control logs might give you logs that's all assume that you have that in Cabana or Splunk or something so that's that's the main commands I personally use there's a lot of other commands in the CLI it's God could help a page we have a website that show later on but that essentially ends the users perspective of Nelson and it was kind of brief but that's also kind of the point of Nelson one of the goals of Nelson is that service owners you just have to focus on their service and not worry about all the complexities associated with the schedule or like kubernetes so this is sort of benefits on both sides right we don't want service owners to think about schedulers or ops we want them to just focus on the service and everything else should be just like just deploy to think and something will automate it on the other side operations doesn't want to be in a business of deploying other people's things so that's why Nelson is has a self-service deployment model but at the same time we don't want people to have like say Cube control access to the cluster because that's a really powerful tool you don't want people messing with your cluster without you knowing alright so sort of the reason we keep a limited is to benefit both sides so now I can talk about Nelson from the operators perspective so just what happens in a back-end that hopefully your service owners don't have to know too much about at least they shouldn't have to maintain so and it's full nelson is going to need a github application a system that can do dynamic credential provisioning this typically means hash a core vault a system that can do service discovery and health checks and in a scheduler at a minimum get nelson only really need to get up application a scheduler and the service discovery system and if you're using the kubernetes backend of Nelson you can sort of fold the service discovery and health check part into your just kubernetes itself it's easy it's in terms of implantation I'll talk about it briefly it's pretty easy to extend right now we in the newer version of Nelson we're currently only a supporting kubernetes in one iteration before it was both kubernetes nomad and Mase's it's pretty easy to extend as long as you support your scheduler or whatever supports pretty basic functionality essentially can you watch a container came to lead a container and can you give me the health of a running container so if you want to use this with a yarn or like Netflix is tightest or something it's pretty easy to extend it's also built to work with multiple data centers and if you really want to get wacky if you have like multiple data centers each with each which run their own scheduler Nelson can deal with that just fine right you can have one day two internet runs meso is one that runs kubernetes one that runs yarn you're good to go as long as they can all deploy containers and you have like more or less a cohesive way of like if you're running like a spark job it better work in all the environments but for like micro services it'll work out of the box in terms of deploying Nelson itself those that could be deployed pretty much anywhere you just need a network line-of-sight with github and the various systems you're interacting with of course if you if you're just doing like a single DC you can even deploy Nelson inside kubernetes itself right Nelson comes as a container so you can deploy Nelson inside a kubernetes cluster that administers that way you can get like lifecycle stuff and an ingress stuff for free but yeah by default Nelson comes with a container if you can make it run somewhere you're good to go so I stopped the users workflow by saying like Nelson's can deploy it into the cloud but the clouds gonna have to be something right it's just hopefully to the user it's deploying to the cloud and it backends something is actually happening so I'll talk about that the second half of that workflow so we'll start by Nelson receiving and parsing the manifests for the repo Ted did the deployment Nelson is then going to dynamically provisioned credentials if that functionality is enabled it will write service discovery information to a service discovery system either making an entry in console or critical kubernetes service they'll be tied to to a specific name and version of whatever is being deployed and if finally it's going to actually deploy two thing to a scheduler after these steps are done Nelson is done with this job and this is a very intentional thing that we're doing we're not Nelson is gonna try to do anything fancy like dynamically rendered their credentials into the Ronin container or automatically connect containers together Nelson is just pushing that information somewhere and it's up to people to pull that information and this in part is because we don't want Nelson to be in a hot path if Nelson goes down everything should still continued at work the only thing that should stop with Nelson goes down is just people can't deploy so that's a very conscious conscious decision that we made now given these systems and given these various backends Nelson can do generally like one one-size-fits-all way for Nelson to interact with these systems right we have it wasn't to github web hooks is really only one way to implement that doing given Nelson's model this sort of one straightforward way to push that information to vault and push that information to console but when intire gets it's time to actually deploy it onto a schedule or somewhere that's when things get tricky right because different organizations are gonna expect things to work in slightly different ways especially for kubernetes you can have stuff like different people can use different kubernetes versions people want to schedule their stuff on certain pods or certain nodes people want magical annotations so like some places will require magic planet you just annotate your workload with specific labels and if something is going to come and scrape metrics from it at a specific specific end point all right so there's different organizations are going to do something slightly different in a previous iteration of Nelson the way it's submitted jobs just scheduler was various was in a very prescriptive way right we were saying we're gonna expect that you're on kubernetes version 1.8 which happens to have this schema you're not gonna you're gonna have the labels we tell you you're gonna have and you're going to you're not gonna be able to define stuff like toleration Zoar or no depending to you anything like that so as tim and i we're trying to deploy this and work with it we constant we're asking questions like what if we want to schedule a GPU node what if we want to persist in volumes what if I only wanted to schedule a certain notes what if I need a certain annotations what if kubernetes one day decided that I wasn't gonna we're gonna use Y Amal they're gonna use I don't know what's doll or something then then then what right we have to always be playing catch-up so we're answer do this and I think some people are going to cringe when they see you next slide but our answer to this we just worked out well is our notion of blueprints which is essentially a fancy way of saying we're gonna have templated specs of whatever your scheduler wants in the case of kubernetes this means templated Yambol I'm sorry in a case of nomads this is going to be templated hclo for instance but essentially we're now allowing instead of having what is sent you to scheduler be and sign bells in source code whoever is operating at deploying Nelson will have a template spec that will still be interpolated in a more or less deterministic way right we're not giving you free access to variables you just say this is where I put the stack this where I put the image and stuff like that but we allow the user to operators to control what blob is sent to a scheduler and you can have multiple different blueprints and you can have them under different versions and users will then specify a blueprint by name and version and there in their manifest and so you can have one blueprint that says this is for experiment so I want to schedule have a GPO enable node and I have one good print that's for like sort of vanilla service that does that has metrics on a prescriptive port or something like that and don't use that blueprint and so this still allows you just to have sort of a high-level way of controlling where things get deployed without having to do questionable things like allowed to splice in arbitrary Yamma which we entertain for about 10 seconds or allow them to or like just give them cube control and I cope for the best I now on the flip side of for operators it allows them to control a lot of the stuff specific read or their organization while again not having to deal with like if you give people if you let people splice an arbitrary yeah I won't they're gonna do some weird stuff with it so we think blueprints serve solve it pretty aim like it's a good middle ground between the different trade-offs and I spend working well for the past couple months we've been using it so to to give an overview of now to full-nelson workflow end to end assume we have the github repo is nelson abled which means we have a web bucket soul we've got the Nelson manifest file we make a deployment that deployment triggers a web look which calls Nelson Nelson goes back and pitches to manifest diamond dynamically provisions potentials right service discovery formation deploys a thing and then from there on that's the end of the workflow end to end in the background the user is then going to use the Nelson CLI or other tools to interact with the deployment and then in the background Nelson is going to be doing its own sort of automation to make things nice traffic shifting health checks clean up with that stuff so this is a roadmap this is some of the bigger items that we're working on moving forward so one thing is for the kubernetes back-end we don't really have a one-size-fits-all answer for services service right in terms of like how do we where do we publish this information that like service d2 tulips here and service D 2.3 lives there kubernetes the crew mets ecosystem is sort of still evolving we could do this through services and having labels that kind of has almost because you can mutate labels halfway through and you don't want people to do that we could do this through like an envoy sidecar and publish it as a CR d sto was coming out we could do it there sto so there's not really a clear answer of where people are going so we don't have a one-size-fits-all answer if you have opinions about this we'd love to talk to you but that's something that's on our roadmap another thing we want to do is currently Nelson only supports deployments off of like a map like a single master branch it's really useful to be able to both for the mono repo use case and also just for like testing for people to do deployment off of branches or pull requests so imagine like if you make a small change to a service and you create a branch for in Nelson and you tell Nelson to deploy a branch it will do the same workflow but just deploy it somewhere else that doesn't affect the Deaf names base for instance and you can send of this business guys or whatever a link to your fork up to service and they could play around with it and once everything looks good once the brat closes Nelson can then detect that with like a web foot or something I automatically shut it down similar to how it does lifecycle management in general so that's Tim and I have been talking about that I'm not done we're also in terms of internal stuff right now Nelson does like JSON over the wire we're moving that to protocol buffers that GRP see if people want to build other tooling and other languages and dumps that they don't have to write their own JSON parser and can actually just formalize that we have an API Nelson currently doesn't have a good like a web UI and so we're working on that as well and then right now it's kind of a hassle to get started with Nelson because we you need to have github you need to have a coop native cluster running at the minimum we're trying to if you have any if you're trying to if you're playing around with Nelson and you're having trouble you knows your pain points or you have ideas on how we can have a smoother demo experience please talk to us and we'd love to get your help on that we do have a website for those forbidden Elson website in the past you will notice it's different about two months ago having known Tim for like two years I realized that he knows how to do front-end stuff so we have a hip new clown native slick website now so you can visit with a slick new dot io domain we were gonna be getting else in that cloud but I decided that was a bit much so what do you did we'll just stop at I know futures not covered today and that are either covered in either Tim or mines past talks or that are just on a website automatic traffic shifting how Nelson handles like load balancers and ingress how graft routing works we have audit logging and Nelson so you can sort of see if you like it people like wondering like why didn't my workload like automatically get deleted and that's an auto ogz there's stuff like slack and email notification whenever things get deployed or cleaned up and we're stuff like deployment off branches PRS etc other than that we have slick website we have a repo that contains everything that Nelson uses including slipway and other tooling we have a gator chatter get her channel that we monitor I do have Nelson stickers which out there here I have Nelson stickers of anyone's interested and if you have questions I think I have like five minutes to take questions and if otherwise Tim and I are around today and tomorrow so feel free to approaches thank you [Applause] this is bad because the person was about to ask a question is on my team the persistent deployment Paul could you get in a situation where you have a service that depends on like a person to point to but transitively depends on e 2.1 and without problems so like can we get into a situation where services either knowing we were unknowing they lie about what the dependencies are probably you can't do that third reason is obvious the problem is management so your core service you directly depends on e 2.2 then you have some other package C which also develop aid but it depends on e 2.1 that's an ce questioning also and also goes Nelson doesn't account for transit dependencies right like it doesn't matter to us that C is incidentally using the older version of a as long as you can get into issues if like C has semantics at the pen on a and your searcing like one semantically incompatible version of a and C is using those different ones that's possible but there's nothing it's possible yeah so we don't we don't provide a way to sort of force people who are existing dependence off of you but you can deprecated a workload of Nelson which essentially means any additional deployments after this will fail at the dependence old version right so if I have a version one at a version two and I deprecated version one and someone tries to deploy a new thing after I deprecated that dependent version one the deployments Nelson will actually stop that so you're depending on your dependent you're trying to deprecated that you're trying to depend on a duplicated version of a stop that yeah the only case where you have like multiple versions as if like one team is depending on you and never deploys ever again in which case you can probably walk over there and yell at them for a variety of reasons why aren't you deploying are you talking about self like workflows for like run this job and that job completed to run this job Oh for staggering deployments not for like not for look creating like a dag up the flames but just like deploy and like the West Region first and if that looks good didn't plan to use region I don't think we have that right now that's an interesting use case maybe talk to us afterwards because we don't have we don't have an answer to that right now but that's definitely interesting so right now Nelson is fairly prescriptive about that so there is support for in terms of automated deployments since everything is just based off of a github deployment if your CI is doing like get up deployments off every commit you master don't you you basically have continuous deployment so I briefly mentioned automatic traffic shifting I'll talk I can talk about it really quick so Nelson is keyed off semantic version if there's major minor patch if you're deploying the only time when Nelson is gonna force someone to explicitly depend on a new version it is as if it's only keyed out the major and a minor version so if you're if you're dependent 2.1 and some of the place you got here you have to make an explicit choice to say I want 2.2 now but if you depend on truth I'll 1.0 and I deployed to dot if you depend on my service to toe to dhawan Otto and I deployed to dot one dot one then Nelson's gonna automatically traffic shift based on some some pattern and if if things begin looking unhealthy of your health check and detect that your thing looks your thing what my thing looks unhealthy whole reverse the traffic shift there will be in a Nelson manifest file I believe so we this is coming up a lot because tags are very common it turns out Nelson purposely makes a decision not to support Dex as part of a principle that we want to be we want to deployed a thing and did not be in your way so where I answer to if you want to deploy like one job and once this job completes run like these two jobs or something like that is to just as part of your containers entry point delegate to something that does handle that so like for instance if you want to run a spark job as container we're not going to have like Nelson spin up like run this job and then run this stage and run this stage we're you're just going to instantly have container of that on entry point it's going to run sparks submit or something like that so essentially we're delegated to a separate system and that keeps Nelson tabatha out of hot path thank you very much [Applause]