SBTB 2014, Connor Doyle:: Run Applications Like a Boss: Fault-Tolerant and at Scale with Marathon
so my name is Connor and I'd like to talk to you about a system called Marathon so you can tell by the title slide downloading there's supposed to be a picture here it's coming from the Internet it's on its way but I work for a small startup called mesosphere we're based here in San Francisco um we're about 30 people in two offices and we do tools and Frameworks on top of Apache mesos in addition to contributing to mesos core marathon is one of those tools okay so here's where I'm going first of all we're going to talk about what marathon is then we'll take a brief detour we'll call it a lightning detour into mesos internals just so you can understand the the model we're working with and then we'll dive back into some more features that you can it's more things that you can do with Marathon we've got a very very pretty logo okay so marathon is first and foremost the Scala application it's a Scala application to run your Scala or other language implemented applications it can run any Linux binary and it runs things on a cluster it's open source Apache 2 license it's up on GitHub and it's a framework for Apache mesos okay just have a quick show of hands who who's heard of mesos okay it's about half who's used it less okay we'll talk more about what this means in a little bit um all right so the goals of marathon first is to be a self-serve interface to your cluster the idea is that any any one of your developers or Ops people can go to Marathon Supply it with a description of an application along with how many instances you want to run and Marathon just makes that true through Magic in this way it behaves kind of like in a knit or an upstart system for long-running services um and this this is kind of another way to to say that but it's a it provides a private pass so you can run it on AWS or GCE or on your own machines be having trouble with the internet here let me just try to reload this uh image here it comes okay we're depending on the internet here um but this is a view of the UI oh here we go so um basically you have this this form this is how easy it is to launch launch an application in your cluster you you fill in the simple information about your application oh it just okay skip ahead okay um all right so these are the things that you can do with Marathon uh there's a there's a nice UI it didn't work but uh you can start stop and update the definition of your running applications you can see what tasks are running you can kill an individual task or you can do all of those with with both the rest API and the web interface um I also have a nice rest interface so first example here is you just do a simple post to launch an application you can do a get to see what applications are running you can put against the against the application to update the app or you can do a get against the task collection to see what tasks are running you can also do a delete against the specific task to kill that instance um so let's jump back into into mesos so maybe just suspend disbelief for for a minute here the idea is that you give mezos an entire cluster so mesos is a top level Apache project it's a obviously open source free to use and what it does is it Aggregates the resources in a cluster so it keeps track of all of the CPUs and RAM and what other whatever other resources you want to account for inside your cluster and it keeps track of what is used and what's left over in addition to doing the resource accounting it also enforces those resource limits using Linux c groups um it's pretty uh pretty robust it's been in production at some of the largest web scale companies uh eBay and Twitter two of the sponsors every tweet you see runs through systems that run on top of Apache mesos it also provides somewhat of a of a SDK for distributed applications it provides some Primitives that you can take advantage of so that you don't have to re-implement them one of which is distributed State another is message passing um but it can really accelerate the the development of new distributed systems so at a high level the motivation for this is that cluster scheduling with heterogeneous jobs is extremely difficult and so mesos breaks that into two levels the first of which is the resource accounting level so it keeps track of of what's available and it delegates more specific logical decisions to the applications that are running on the framework or running on the cluster and the main idea here is that you have a resource offer so mesos will take a chunk of resources in the cluster hand it to a framework and the framework can decide to either launch tasks or to decline those resources um so an application that sits on top of mesos is called a framework there's two parts one is the scheduler and that's the part that receives resource offers from the cluster and the other part is the executor which is the part that runs tasks okay here's the same idea in picture form so we have a scheduler that is receiving a stream of resource offers from mesos it can decide to launch tasks mesos will spin up the executor for that task and then report back status to the scheduler and this is all no single point of failure works on thousands of machines so the mesos master can die the slave can die the executor can die your framework can die and we can survive all that so let's just slip marathon in from the previous picture so since it does implement the mesos scheduler API it's receiving the stream of resource offers from mesos at the same time it provides a rest API for you to Define your application so your application definition is just a blob of Json and one of the items could be Uris which is your application package ZIP or a tar what have you and then also the number of instances of each application you want to run and Marathon will build it'll compute a queue of tasks that need to be launched on the cluster and then drains that queue using the resource offer stream and it also provides an API for introspecting what's running and where it is running so at a high level Marathon puts an abstraction above the tasks that are running in the mesos cluster called an application and every task is an instance of one of the apps and Marathon will launch tasks for applications it receives updates from the system about whether the task is still running or not so if a machine in your data center just like explodes Marathon will know that that task was lost and it'll start it somewhere else in the cluster just a look at a couple of internal components so we have an application definition which is an immutable thing and we have an app update which is a diffing type for application just so it's a way for for us to keep history so we have the marathon scheduler which is also the part that receives resource offers and then we have a task tracker just so that we can keep track of what's running and make your state what you want uh so here are a few more exam Advanced features of marathon it has an event bus so that you can you can subscribe an external program you can provide a rest endpoint and Marathon will push updates into it as events happen task comes up task goes down as I mentioned applications are versioned so you can look at the history of your application definitions you can post a previous time stamp and effectively do a rollback it's highly available so it does leader election through zookeeper it supports placement constraints so when you spin up your your meso slave so every machine in the cluster has possibly attributes you can say this machine has fast disk this machine has like a infiniband or whatever and then those attributes are surfaced in the resource offer and you can you can hang off of those for scheduling decisions in your application definition it also does health checks so you can set up like an endpoint on your application and you know a frequency and timeouts and everything like that it'll keep track of whether the application's healthy or not if it's not healthy you know past the maximum number of failed health checks and it'll kill it and start it somewhere else and so the main idea is just to make your operations completely automated upcoming in the next release which is maybe next week or two rolling deploys and restarts namespaced applications so you can Define basically groups of applications you can have a a dag where you have a product with a front end and a back end and then applications under the under those and you can declare an explicit dependency say the front end depends on the back end so that when you issue a scale against the product it'll scale the back end first and then scale the front end so health checks on the executor if you've got 10 000 nodes you don't want like a one to end fan out and so this is a way for health checks to be executed on every node and then only the changes in health status are forwarded back to the scheduler it's a much more scalable way of Health checking your applications artifact staging is a new feature this means that if you have Uris like basically your application package you can stage them into hcfs and then you don't have again that one day and fan out where you're like pulling from your Maven artifact repository from every node exponential back off this helps when you have a flapping task so a task that for some reason the configuration is just bad it's never going to become healthy we back off so that you don't fill up your disk with uh with dead things also first class Docker support so this is a feature that's coming into mesos core and we're going to support it in Marathon as well you just Supply a blob of Json that points at a Docker container you can run a custom command inside the docker you can just do a Docker run and that's all for free out of the box that's it for now uh yeah so it's up on GitHub Pierre's are welcome it's all in Scala you all know Scala so let's be friends and we're also hiring scholar developers and go and C plus plus if you swing that way that's it yes okay so the question was how do you compare mesos with yarn and uh so first of all they're very similar systems they're both doing cluster scheduling yarn or mesos predates yarn by quite a bit um but basically mesos is model is supposed to be more General and Abstract than than Yarns Yarns tends to be a little bit more hadoop-centric but in all you know there there's they're very similar projects processes are they like devices are like what's the greatest okay good question so in these cases uh the the task could be just as just a Unix command and that would be executed by the default command executor inside of mezos but in general it's it's uh it's up to the framework so the task is just basically the task semantics are dependent completely upon the framework so we have an example that uses just a URL as a task and we run a run a web crawler and so the executor interprets the tests and spits back results to the scheduler yeah in the back rolling updates feature sure so the main idea is that you can in the in the next version every put to an application is is a deployment and you have as part of the app definition a minimum Health capacity and that's a normalized value so it could be like 0.8 that means that we'll use 20 of that room to roll to spin up new instances there's also a maximum health capacity that is by default one so if you have extra capacity in the in the cluster you want to use through the rolling update you can you can specify that all right oh yeah in the notes foreign oh so for applications running on top of mesos what's okay so the overhead is simply a c groups container it's pretty negligible okay thank you