SF Scala: Christopher Hunt, Container Cluster Management

SF Scala: Christopher Hunt, Container Cluster Management

Recording: SF Scala: Christopher Hunt, Container Cluster Management

Thanks so I'm not carrying a mic presume you can all hear me yeah good at the back down yep nowhere it's okay fontsize good fury there at the back if I just go to the terminal window look alright a bit bigger okay okay no or is so I'm going to you know leave the speaker notes at the bottom because I haven't really practiced this at all this was kind of thrown together very quickly but what I'd like to talk to you tonight about is our containers and the state of the nation as I see actually state of the union is the correct thing isn't it so let's correct that straight away you know we enter real time amendments to the presentation just because we like living on the edge here so you know maybe containers is a little bit of an odd subject to talk about that you know a Scylla meter but I kind of wanted to do so because production is very much a part of our world now as developers whether we like it or not you know our employers are increasingly demanding that we look after what we develop in production because the consequence of that being that we are going to take more interest in that that quality aspect of things right we don't want that too a.m. wake up call from our pagers iPhones or whatever because you know something has gone down there in production by the way my crystal has always introduced I work at like band of course we make a significant contribution to the Scala compiler and and Martina dusky the creator of Skyler was is one of our founders and that's my twitter handle so you can also describe to me and make me very popular with my children so so yeah I just talked a little bit about containers I I'm kind of assuming that we know what containers are about here you know they are discussed a great deal but then sometimes it's bad to assume so please don't be afraid to put your hand up right now if you know you don't know what the heck I'm talking about when I say containers okay we're all good with containers we show of hands that using docker a lot of docker going on anyone using rocket at all mmm that's kind of what I expected so you know I was thinking well depend depending on the reaction I could sort of you know show you a little bit about docker but then I'm glad that I don't have to but what I want to talk about in particular is the open container initiative now I'm also interested in whether you guys have heard much about the OCI show of hands have you heard of the open container initiative okay that's great okay but again the reaction was kind of what I expected and it's been going for a couple of years now and was formed by dakka dakka inc that is an coral s and coral s of course our behind the rocket container and the intention is to formalize the container image specification right so that nobody no one vendor owns that you know up until that point there are many representations of these container images and also to provide to standardize on what the runtime for that should actually look like as I said you know number of specifications and since they formed the the initiative that the likes of of Google and mesosphere and others have also joined this group it's been going a couple of years and the specification is about to get to 1.0 I don't believe that it's been released at this point but what it means for you is really significant because it means it's all about avoiding vendor lock-in right we want as developers of course to be the authorized organizations to be locked into as good as they are you know to darker ink or to core OS or you know to anybody else that wants to create a container standard and you know that is fundamentally what it's about and to me that that's you know what excites excites me and so so that's really OC I what I wanted to show you was sorry is a container or container cluster manager that we have written at light Bend you know and of course there are a few out there there is kubernetes there is me sauce marathon docker native they're calling it now which was formerly docker swarm hashey quartz nomad I'm probably missing some that I think that's appear you got one as well okay Oh will they oh yes absolutely sorry whoops hey big Spotify fan by the way you know I subscribe I can show you in fact my whole family subscribes six of them but yeah and you know that what that shows certainly to me is that there are a lot of people in this space and increasingly so because there are no problems that need need solving there so so we have one as well the project's itself is called conductor but it's part of a larger suite of technologies that actually will become free you know for it for usage and available very soon we've it's actually about three years old now and we use it in production ourselves like been calm and reactive summers and our other websites and web assets and our own micro services all run on on this infrastructure today but what there's something I want to show you here about that which I'm particularly excited about and we've we've embraced OCI right so you know we have OCI cluster management now as part of the tool and I wanted to show you then what that actually looks like so so can conductor has this command-line interface here you know what this is actually showing me is I've got a few things running we've got an eighth look at the proxy running we've got a logger running and we have a visualizer running as well which I'll show you in a second now all of this is written in well predominantly they could knock to core an agent are actually written in Scala and akka we use a cluster and anyone using occur around here as well bit of a code going on okay so if you guys do of course so what you know there's it's kind of like if you believe in in the actor model and its ability to manage distributed systems and you know here's a working example of of occur in in action you know that is highly scalable highly performant and you know follows the AP sort of bias you know of the cap theorem you know we were very much focused on that from a knack of spective what that means is that it's very lightweight it has no dependencies you know when running and therefore less moving parts listening to to manage right now sue Cooper's in this world here but before I actually show you the OCI sort of thing I want to show you this is just running a little app provided by dr. Clegg called hello world that's going to expose port East port 80 on port 9000 ok and so if I if I go over here let you wrong window okay that's going to be this window here what wrong window this window right so you can see you know it it's outputting to the line just to show you you know the real node there's no smoke and mirrors you know that's that's just stuff running in docker but where this gets interesting is where now I can save that image and then pipe it directly into conductor itself this is exactly the same docker image with no change whatsoever alright then I can I can run that as I say and I'll go through you know exactly what this is actually just done in a second this is show you over here running on the wrong port here and let me I just got to do a I can do a conduct info on this this bundle here and it will show me that's running off port 1006 0 9 let's change that and there we go we see exactly the same docker image running unchanged and pipe through into conductor there you go okay well fine you know I can I can get that with kubernetes as well but what just happened here is that we normalize into the OCI image format we load that into conductor core which becomes a bit of a file server a bit of a registry in its own right and then when we run it it actually packages it on disk and then applies run C to run that images itself now an interesting point about this is that that package we call them bundles actually contains all of the image layers in it as well right so you you know the idea of a bundle is it has everything it needs to run there's no requirement to go and source layers at runtime from the registries over the network other than the core itself that it gets all in one go and this enables a very fast and reliable startup ok but the other takeaway from this is that you know even if you were using this purely within your development environment to sort of you know pin down and verify what it means to write a distributed application to make sure your service discovery is in place to make sure that you're consolidated logging is working and so forth then and then at runtime you know you want to choose a different target be like ooh vanitas or any of the other OCI compliant orchestration systems out there then you have to freedom to do that right and that I personally find that really quite exciting and we of course we hope that you sort of stick with us right but you know you don't have to and again you know underscores that rule of avoid avoiding vendor locking so I wanted to pretend up because I you know I again I'd that I hope you know a lot more about OCI now and what the impact on our own industry is going to be I think it's personally going to be quite big I know for example the kubernetes community are really excited about OCI because of course you know they can have a reliable image format without other vendors having to change things under them and you know I think it's very exciting for us as developers but that's all I got for you tonight so hope that was useful do you have any questions at all we were just totally impressed this is the awkward moment isn't it you know you can clap actually if you want to cut really what makes it better for Skylar is you know our obviously you know we we totally get the benefits of scala akka play Lagaan you know because that you know we support these projects from a company standpoint you know and you know what's carlo fundamentally brings to the table at least in my mind i have this mantra of needing to sort of you know write less code but more code of higher quality ryan scalar is a language that helps me attain that goal more more so than a lot of other languages that I've used I know that the more lines of code that I write the more bugs I introduce into a system right as we all do right that potential is there so you know I I would say therefore that there's a high degree of quality associated with what you're seeing and of course by using akka as well it's naturally distributed you know distribution and the handling of failure and that and you note that the resiliency aspects of the reactive manifesto they're not afterthought you know they're in it in their in day one you know and just not in code form but just in design form you know in terms of how it permeates how we think about the system an example of that being that there is no master in this world it's masterless right you can contact any of the core nodes split-brain resolution is there with auto recovery you know so we've put it you know we've we've basically leveraged akka and akka cluster and SPR and associated technology akka streams is a lot of streams going on here with first users of streams pretty much and actor HTTP so you know so to me it's a celebration of everything that we go out there and we talk about you know it's it kind of makes it real you know that's that's what I would say yeah yes well I'm a bit worried about the word better right because better definitely sort of a personal angle on things I don't know of a comparison chart I certainly know of places that compare that you know the more popular cluster management solution so that Rancher for example have a really great document identify heard of rancher at all they they were there another you know example of a cluster manager but they they provide a normalized model over me sauce and COO Benitez and docker swarm right but they on their website they they provide a great comparison and talk about the strengths and the weaknesses of each but but not ask you know we were not known to them you know so we need to do more of this kind of stuff you know to make ourselves known as well right so the help right okay okay conductor like been calm well actually give you the documentation at least yeah I should have put that up there sorry yeah yeah but if you go to light Bend calm you know you should be able to eventually navigate yourself around we are actually overhauling the getting started experience and making it you know stick as possible it's a little bit awkward right now to be honest but that's going to change like imminently within days itself right but conductor like been calm will actually you know get you some of the way there hopefully yeah this is the converting of the docker container yeah yeah that is cool you guys rotor is very another open-source project great question actually there are a number of technologies that there are L tools out there already that will do this the maquis is one of the the ones in this area there is another one called scope you or I can never get the name right and indeed we actually that there are a couple of transform transformations that occur and the first transformation in the CLI when you do that docker save and pipe it to conduct load it goes through another command called bundle that we initially started using an off-the-shelf tool but then decided to eradicate our dependencies there and did the transformation ourselves and it actually turned out to be quite easy and we should be not of no surprise you know docker and core OS have both contributed to this image specification but you know it looks a lot like docker in my opinion and and run see of course is run c is the core of docker today right we and they've they've contributed that to the OCI as a reference implementation ok so it's pretty solid there you know as and you can see you know quite performant as well yeah yeah anything else no cool that's all I got Thanks [Applause] [Music]