Writing distributed Scala applications with Mesos

funconf 2013, Florian Leibert: Writing distributed Scala applications with Mesos

Recording: funconf 2013, Florian Leibert: Writing distributed Scala applications with Mesos

thank you guys all for coming by um today i want to talk a little bit about how to write distributed scala applications with mesos and a little bit about me so i've been uh i used to work actually with mario's at uh at twitter and uh i worked on user search there i um sort of helped bring ben into twitter because i've known ben who's the one of the main authors of mesos i've known him for a very very long time 13 years and after twitter i went over to airbnb and built out the data infrastructure team and there we built logging systems etl pipelines pretty standard stuff but we also built kronos a framework that was built directly on top of mesos and that was actually really fun so um the other stuff was fun too but yeah and now i'm actually i moved on from airbnb and i'm working on a startup around kind of around mesos so yeah i'll tell you a little bit about the structure of this talk so i kind of want to go through the state-of-the-art deployment of apps and cluster management initially and then i i'll give you a brief intro to mesos and how it works and then we'll actually do something a little bit more hands-on that you can follow along and it's basically how to build a framework in in mesos and if you if you want to go if you have a laptop and you want to check out this url in the meantime and start building because you'll need to likely compile something which takes a while um you can go ahead otherwise you can just follow along and do that at home afterwards if you care to do so and yeah so how many of you guys have actually tried to play with mesos before okay so about like a third or so cool awesome so well first of all kind of the motivation behind behind why mesos was built generally when you when you start with a data center or when you start as a web business you the first step is to provision vms on either public cloud or on your uh on your physical boxes on the bare metal and what that leads to is you start out with maybe 10 boxes and then all of a sudden you have 40 boxes so you have more stuff to manage but you get finer granularity then you usually use something like chef or puppet that's i think pretty common nowadays to set up and launch hadoop maybe ansible so you set up your hadoop cluster and then you decide we also want to serve a website so because we don't have any log data so you're like well we'll use jboss because we're an enterprise company or rails or sinatra so you basically provision your jboss cluster and you already see you have now two statically partitioned sub clusters basically and now you get a big big hit on techcrunch and you need to crunch a lot of a lot of logs so you actually resize your your hadoop cluster and the resizing is actually pretty painful i don't know if anyone here has done a resize of hadoop it's actually not as as seamless as it seems and well what you end up is you end up with a larger statically partitioned hadoop cluster and yeah so what's the problem with this well i mean obviously that works and and a lot of companies are doing this right now but it's really if you see all the steps involved it's really difficult to deploy new frameworks you have to provision you have to set up you have to install you have to resize and also the static partitioning leads to a really low utilization and it prevents elasticity and i'll show you in the next slides why that is so let's assume you have like you have your memcache cluster you have your jboss cluster and you have a hadoop cluster and most of the processing at least at airbnb that we did the automated batch processing took place at night whereas the peak of utilization where people actually checked and started to book an airbnb was during their working hours of course because nobody loves to do work but um so as you can already see during this time this cluster has the hadoop cluster has really low utilization and during night time basically the the jboss cluster is not it's not very busy so before we jump into how it looks with mesos let's let's give you a little bit about a little bit of background about mesos so with with mesos you get fault tolerance you get scalability to tens of thousands of nodes you get isolation between tasks by using linux containers you get you get basically a multi-tenant cluster and the apis are that are available at java python c plus plus i think shopify just released the released ruby bindings and in general mesos has a coordinator worker architecture so basically you have a bunch of a bunch of workers that that feed the resource offers into the into the master and i'll show that in in in a slight further down but but i guess like also why should you care about this um there are a couple of companies that are really using mesos in a in a big way i think the biggest the biggest user right now is twitter i think they're using five or six thousand notes i don't know is that about right at least yeah so and i think twitter the entire ads infrastructure runs on mesa so it's a it's a serious i mean ads is very serious for twitter so all of that stuff runs on runs on mesos and at airbnb on the other hand all of the analytics infrastructure now runs on mesos and job orchestration is done by a tool called kronos kronos is it's basically a it's a distributed crown with added support for dependencies and fault tolerance and well that's distributed and i can show you a little demo later but many others also run kronos d-park d-park is a python variant of spark and yeah so so it's really it's really gaining a bunch of momentum but to come back to this slide that we saw earlier about the resource utilization with mesos hypothetically you'll have something like that you can you can really um you can really leverage the the peaks and troughs by by basically putting your by basically merging your entire cluster into one large resource pool and um you can you can obviously also give different priorities to each of them to each of these frameworks but this is essentially what you're what the view of your cluster from mesos and if you contrast that to the way that that you have the view which you have a bunch of vms it just seems it just seems a little bit simpler right so that's that's one of the big that's one of the big value propositions of mesos and by the way feel free to ask any questions uh in between so i talked already earlier about the way that measles works on a on a high level basically you have you have the masters or coordinators and frameworks register with the coordinators so an example is spark spark how many here have heard about spark okay awesome yeah so everybody was at the talk before um so yeah so spark is a framework that was actually originally written against mesos i think it now provides a standalone mode and i don't know if it works on yarn as well but um it's one example of a framework chronos is another is another framework that i've mentioned mpi runs on um runs on uh mesos and uh i guess what else is there a couple of others there's for example um a storm storm runs on mesus as well i think natively yeah and we'll talk about one one more later as well but essentially the scheduling is done the scheduling of these of these tasks is done by asynchronous resource offers so um when when a worker comes online it basically says hey i have i have x or x gigabytes of ram i have x cp n cpus and then this is handed to the master and the master has an allocation policy and according to this allocation policy these resources are offered to the frameworks that have registered with the master so for example you could you could assume that your front end resources or your front ends should be getting a higher weight than your batch processes right because you don't want to if a customer comes to the website you don't want to serve a 500 because you can't you can't you don't have capacity because you're running a hadoop job essentially so you can really adjust this adjust the weight for each of the frameworks how much what a percentage of the offers they basically get and well mesos also monitors your your tasks that you're running and it will notify your framework if a fault occurred and then it's up to you to really say hey i want to retry this task or i i just want to abort and so so it's really like made for writing high avail highly available and fault tolerant systems so a little bit about resource offers so resource offers are consumable they are basically cpus they're ram and they're disk and i think they're working on i o as well but i don't know what the what the latest status on that is but so so essentially if you're if your machine has four cpus and one task consumes one then you only have three available and uh it's the same for it's the same for ram and it's the same for disk space but there's another really neat thing attributes so meso supports attributes and you can basically just hand those to the to the slave process that you start they're essentially just a map of string to string and they're non-consumable they're just they're an attribute so it's a rack id uh it doesn't go away if you if you advertise it every it's it's basically it allows you when you write a scheduler to make a scheduling decision based on the these attributes where you want to which which resource offers you want to accept for example a license could be another thing that you that you expose an attribute so if you have a mathematical license on one box then you could run mathematica tasks on their box i have a question go back to the architecture site could you clarify how all those individual pieces map to either like physical machines virtual machines or linux containers sure okay so so um a slave is essentially a process that runs on that runs on on a box and we've set up our boxes to run uh one slave per box and these slaves can then basically fork a a linux contain a linux well a c group and in that c group an executor will run and uh that executor then will will be if a framework for example sends a send a task to that to that slave that executor will run the task and we can i think it will become a little more clear in a couple of slides does that answer your question yes okay awesome um okay so well i think i think um it's kind of important to point out here that i've already hinted at that but there are these two these two main areas that i've worked uh with mesos with and one is kind of in the request response uh part so request response are your services right there they're up and they're always running and then batch that just kind of well it's like hadoop right you're running a hadoop job and hadoop should be up at the for the time uh it doesn't have to be up all the time or it's a it's a task that finishes basically and in the request response part your tasks actually don't ever finish at least in theory so and those are some frameworks that are that are currently that we can currently launch on mesos so spark mpi hadoop chronos are all in the batch in the batch part i think storm is also considered batch and kind of to to to come back to your question earlier about about c groups and isolation well mesos itself doesn't actually provide isolation it leverages the isolation that the linux kernel provides or that if you're running on solaris you can use zones so the linux kernel since for the past like three four years uh on every standard distribution has contained c groups so uh you can get process isolation and memory isolation uh so sorry you can get cpu and memory isolation by using c groups but you don't actually have to do that so for example at airbnb we weren't using c groups for a while we were just using process isolation meaning that your process could grow its ram as until it basically was shut down or swapped and um but of course it's really nice if you use c groups and you have analysts that that like write really large queries that generally take down your cluster if they now write really large queries their processes just get killed and but so another interesting aspect about this was that at twitter we ran mesos and bare metal they still do not not on top of a vm and at airbnb we actually ran in the cloud we ran everything on aws and aw aws uses zen underneath so you can actually get the benefits of mesos even if you run on top of a vm but you don't have to run vm auto scaling on top of the vms underneath you you could do that yeah yeah you'd have to i mean right now you'd have to build some of that wiring yourself but yeah that is definitely something something you can do so oh yeah i guess one one aspect i forgot to mention here so um obviously when you when you're running on bare metal your host operating system can do optimizations that you can't really do when you're running on a virtual machine because the hypervisor is kind of a black box right so in general you get much better performance if you're running on on bare metal but that's pretty obvious so to go back to airbnb and kind of finish off this this story on how airbnb actually got involved in mesos when we were evaluating mesos last year at airbnb after i've seen it at twitter the primary reason for for using it was we wanted to move off of emr that was but that was kind of the long that was the long-term strategy back then and we wanted to first start playing with mesos and and basically introducing it uh and so we wrote this system called chronos and uh kronos ran as this distributed cron-like system on just like 10 notes or so initially and then we added we added other frameworks so we added a storm and now as i had mentioned they moved completely off emr and run everything on mesos so hadoop runs on mesos at airbnb and the cool thing about that is you can run actually multiple versions of hadoop on the same mesos cluster so you can give one hadoop cluster to your research department and they can and you can actually like constrain the resources they use and you can run another another hadoop cluster that runs all of your production jobs so it's pretty nice um yeah and i think i think when we go back to when we go back to this slide um i think like airbnb is now uh looking at least from some of the engineers i've spoken with they're also looking of extending the use of mesus to this request response part of the equation so yeah there was a there was an art there was an article of on gigaom about um about airbnb so if you guys want to check it out it's it's kind of interesting they write a little bit about uh how how i don't know if they write about how much money they're saving by running it directly on mesos instead of emr but uh it's a really interesting read so does that article like i didn't see like your group mentioned there like this is giving enough credit to people who actually build all this stuff or is it just i think it's more of an overview of things yeah i'm gonna i'm gonna sit down and give you guys a little quick overview before we continue here of the of the mesos ui so that you kind of maybe this makes things also a bit clearer so this is kind of this is kind of running mesos locally on my box i just i just told the slave that the mesoslave that i had four gigabytes of ram and 785 gigabyte disk and so so you see one slave is running uh there's there's a there's a resource offer out and there's a framework registered which i'll show you in a little bit it's called marathon but i think what's also interesting to look at is you can actually start mesos really once you have it installed it you can start it really really easily by going by this presenter screen thing is kind of annoying so the command line for starting is basically here mesos master you give it you give it the url to zookeeper and that runs your master it doesn't run the embedded zookeeper though it's it's mostly for the library bindings and then for running a for running a meso slave um i'm setting some some library path things here but you can see here i specify the consumable resources and here i specify the attributes in this case i specify i have a mathematical license which i actually don't but i'm also not running mathematical on here it's just an example okay so so this is um this is this part okay but what i really wanted to start doing is i really wanted to shift the focus a little bit because i think every everything i've i've mentioned so far was more from an operational point of view so i think there's also there's also this view for developers so in general i think when as we're building more and more distributed systems failures happen right so for example you lose you lose a node or you lose an entire rack and you have to deal with that as a developer developer so and i think mesos gives you a really neat way of handling these sort of these sort of errors and the same for elasticity i mean the cloud promises us all this elastic stuff and at the same time i see people over and over re-implementing the elasticity components so mesos mesos really like makes that also really easy and having said that let's really jump into uh kind of the anatomy of a framework and we'll build a small mesos framework i'll sit down with this stuff oh i gotta give some credit here regret credit is due um a friend of mine toby actually came up with this tutorial originally and i just modified it a bit so all right and i'm actually going to switch to mirrored screen now awesome so i don't know uh is anyone kind of like did anyone start doing the tutorial or give it give it a shot no okay all right so how this actually how this actually works if i find the right terminal here okay so basically i i cloned the repo and i checked out the as the step zero s0 so this is a little bit confusing there are some tags in there and um we i just wanted to upgrade it to run with the latest version of mesos and so i created branches and they are called s0 through s5 in case you want to check this out yourself so i have it open in my ide already and initially there's just i know a lot of people will hate me for this but there's a pom file in here for um not not a spg file i'm sorry oh oh sorry guys um so you see this pawn fight can you can you read this properly or should we should we turn the lights down much better okay cool so um basically basically all i'm doing is i created like this this template this template palm file here and um that creates an uber jar in the end and not much happens actually so let's check out let's check out the first part where we added some code so this is this is the example scheduler and this is actually these are all the methods you just need to implement that are that are defined in this uh in this abstract scheduler class that mesos pro that meso ships with and in fact like only two of them are really interesting initially the two that are really interesting are resource offers and a status update so you get a resource offer when you get a resource offer you can you can say hey i want to accept that resource offer or you can just decline it status update is relevant if one of your tasks that you had launched has an updated status such as like it's running it's terminated it failed um and uh there are a couple of other ones slave laws slave loss generally also results in a task loss so you don't unless you have some special semantics that you need to do when when your slave leaves you might be able to ignore this but um this is this is kind of just the the layout of the scheduler and before i actually continue further i kind of mentioned earlier that you have schedulers and then you have you have executors so um which mesa ships with a command line executor that basically just forks a new shell so you can you can that's what we've actually been using for kronos and um for a new framework that we've we've been writing as of recently and please interrupt me if you have any questions so we'll we'll go through this uh through the next step so we'll check out step two we created a main method a main main class object i mean and we give it the framework a name we accept an input argument it's the master url that could be either local which means it will run an in-process mesos or it's a url to the zookeeper including the path to your mesos node and we also include a command to run because this is the framework we're building we want to just we want to just build an init d for your cluster so where you can say i want to launch a process five times on five different nodes for example and i want that pro i want five instances to be up and running all the time so if one dies we should launch another one um and then we say how many instances so so these three arguments so then the first thing that you do is you you build a framework id and that's just an identifier for for this for this scheduler basically uh and then you build you construct the framework info so under the hood by the way protobufs are used for all of these for all of these data structures and uh when when you determine you basically set our parameters like failover timeout which is the amount of time in seconds uh when your framework goes away and if it doesn't reappear within x seconds it will all the tasks will be killed so if you set that to infinity the tasks will continue running if you set it to zero if your framework if a framework dies all the tasks are killed why is this kind of useful well hadoop i don't know about the latest version of hadoop but the job tracker for example the job trigger can be seen as a scheduler so if the job tracker dies and if it's not recoverable because it's not ha you also might want to just kill all the tasks because there's no there's no you're not going to be able to recover them that's kind of the use case there and all right next step is we basically build a new scheduler we say new example scheduler we give it the command and the number of instances and then we construct the driver out of the scheduler the framework info the master and run the driver and in fact we can probably already do that so let's see ah yeah so so here you see mesos mesoscala just registered just now and it's running so we'll terminate it and that that should go away within now within 60 seconds that should actually disappear but let's let's continue if you don't have any question we'll go to the next step so next step well what do we do we actually implement some logic here we say so as you see the only method we fill in is the resource offers so for each of the offers we so we loop through all of these offers and what we do is we say if the current number of instances is less than the number of instances that we um we want to be um we want to be running then basically construct start launch a new instance and how do we do this we we basically say how many cpus we need how many um i think we only set the cpus we don't set anything else here yeah so we only set the number of cpus we need and i think in this case we just set it to one and um set the command so so the command info builder basically allows us to construct just a shell command and then we say hey we give the task a name so with a task id that we generated that's that should be unique and we set the slave id the slave id is handed in from the offer we add our resources that we need and set the command build that and then what we say is we tell the driver to launch the task and we get the offer id out and give it a list of the tasks we want to launch so and then we just increment our current instance count so let's try that what do we what should we run well let's run sleep 60. actually let's run echo temp demo one and and let's run this hopefully works how many did i say it should launch oh maybe i did that wrong oh i told it to launch more i told you to launch 10. that's why you see this going up so i only have one slave running here that's why it takes a while to go up but but as you can see it actually it actually launches it tries to launch all these tasks and it actually can only launch four because i only have four cpus available so it will actually not be able to get any more offers once it has launched four tasks so um and you can see there's a bunch of sleeps in here so if we check out demo if we oh obviously i i overwrote it each time rather than appending so okay but you can see this kind of works so let's any questions ah it actually does take care of deployment so you can give it a you can actually you can put a uri in or a list of your uris and it will automatically download them if they are tart archives it will automatically untar unzip and uh place them in the local directory so it actually does take care of distribution and it will place them in a um in a temporary directory that's active while the executor is active it will eventually when the task dies will be cleaned up according to a global policy i think actually the policy is not global you can i think set it per slave if i'm not mistaken yeah or like you can i mean you can actually give it a htfs url and i mean like htfs is super fast right like even if you have to download 200 max i mean that's that's pretty quick and um otherwise you can do some bash magic and figure out if it's there already and done i mean you don't actually have to touch any of the any of the mazes code you can make all of this work outside of mesos and and that's what we actually did at airbnb so we did some we did some local caching and we wrote some fancy bash scripts that took care of that so yeah how do so you have you can start multiple masters okay so so as many as you want and one will be elected leader the all the other ones will do a redirect to the elected leader and so so basically if you want to go to the web ui uh it will be redirected to the to the masters web ui after a couple of seconds and um yeah and then you can also run as many slaves as you want so that's how it's how it's highly available uh you could you could actually do you'd have to bootstrap it somehow um and i think the latest version of mesos actually has a has a little hack in it where um you can actually execute um you can actually bring up a framework um with a single command and the the timeout of the framework is just uh it's just infinity so the task will always be up and running so uh but yeah so so you could could essentially do that but i mean you kind of need to still get the slaves onto the boxes and need to get the initial slave started right like because they need to these the boxes where you want to start this so so i mean yeah you need to start the slave somehow in that box so it knows to communicate with the master and there's not enough resources so it's like uh i mean whoever has the higher priority of the frameworks will get the first resource offer and if that if that framework accepts the resource offer then the resource is consumed so the second if you give it if you give it the priority yeah i mean but it but it actually how it's implemented right now it's like i think it's weighted fair sharing so it's like so it's not guaranteed but there's a in the current master there's a patch in it's called resource reservations resource reservations are really cool because they actually allow you to guarantee for a framework that that framework gets resources on a given box so these frame these resources are then pre-consumed and no other framework can take them so and that's actually like i played with that last night it's working so i mean according it's working in a test setup i think actually ben also told me that it's work i don't know you guys are probably using it already at the twitter yeah in the same line so let's say my app wants this set of resources in order to run and it needs all of them so it's all nothing basically gets all the resources but or it gets nothing and it doesn't consume anything can you specify a set of resources with which i will start or i will not actually consume them oh yeah so you oh you can you can say hey i want i want x amount of ram x uh x memory and x disk and then it's like a and and for all of them yeah there's actually no way to i mean you would have to do some magic and do in order to do an ore like because generally you you want your process probably consumes all of them so yeah so how does an application auto scale an application actually does an auto scale within meso so so the scaling becomes really easy because you can add slaves on the fly and these slaves will just advertise their available resources to the master and then they can run tasks so you don't need to reconfigure your application in order to launch something on those on those uh slaves because really the the idea is that we don't care uh what what if this is a box if this is a if this is a physical box or or we just care about this pool of resources and you can add to that pool but it's really up to your application to scale and the application when i say application i mean like your framework like the framework we're building right now it's really up to that that framework to decide how to deal with with uh if if it is um how do you basically deal with claiming more resources yes yeah okay let's let's go to the let's check out the next the next thing here we're almost done with this one and then i want to show you guys something else so okay we'll check out s 4 well what does s4 what does s4 do we we added some this is a really low resolution very hard sorry so so basically yeah we basically say hey if we now add uh kind of the ability when a task when a task dies we can um restart them so let's try this out actually am i still running here sorry i really can't see much on the screen right now okay let's start it so so what we do is like when uh when when we kill one of these processes that's supposed to be running and actually let me let me make this a little bit more clear because so what we'll do as you see as you see i'm specifying we should run ten of them let's just run two of these and let's actually append okay so it's supposed to run two two times the sleep statement and by the way like one of the things like when you run zookeeper on localhost and you have a bunch of other java processes running it's kind of uh sometimes there is a little bit of of a weird behavior where your zoo keeper drops the connection so just if you see that this is likely a zookeeper related thing okay so because we had the i think we haven't handled in our framework the case yet that we so task was already running it was already scheduled to run and since the frame of the the timeout of the framework is 60 seconds that task was not killed now it's killed because we we um we are over this threshold but let's say so so now we want to run this fancy program that we wrote this uh this echo and sleep program we want to run this um two times in our cluster all the time so let's kill one of them and see what happens so let's kill this okay so first of all let me reconnect my framework here i don't know live demos so much fun okay so i think maybe it started right up again so let's kill another one so if we look in here it has it has finished two of them it's it started two more so so that there are two of them running um okay so so you see like even if you kill them they come right back because uh of our logic here where we get the status where we get the status update and let's go through those logs really quickly just so you actually believe me so received status update task task running and do we actually print this out here task failed here we saw there was a task failed because we killed it earlier and then it started back up so um there's a last step and then we then i s5 so what we added was oh yeah oh this is this is kind of neat this is the last thing i kind of wanted to show as you as you as you might have seen right now we don't actually persist the number of tasks that we want to be running but mesos provides a pretty neat abstraction it provides a state abstraction so this is a this is basically a replicated state and it can be either backed by zookeeper so right now it's for the java part of it it's only backed by zookeeper and um so it's only backed by zookeeper but the guys at twitter actually have written a a replicated log that is actually shipped with mesos and i think eventually it will be backed by that that's backed under under the hood by level db so but in fact chronos uses the zookeeper state abstraction it scales quite well so um some okay so so as you see this is this is the framework you go play with it download it if you want i wanted to i wanted to show you something something else that we were that we were working on that is related so i'll show you this i don't know if have you guys has anyone heard of pronouns it's it's this this to this framework that we wrote as well yeah okay cool so another thing i wanted to show you is this marathon framework that we've currently recently built and this is the last thing so so marathon is actually a more it's it's something that it's like the inner d for your cluster like the or run it for your cluster what we just built but a little it has more features so um let's start this up okay so we see marathon is running we terminated our other framework so what does marathon allow me to do well let's say we want to run kronos locally i hope this works okay so as you as you've seen i've i've just used this command line tool that i that that we built for for launching stuff with um with marathon and now we can see if it actually launches kronos oh yeah so cronus is running in this cluster sorry the the screen resolution is so bad that the ui is really terrible but um so so yeah chronos kronos is running and you see here wow we're running we're running this and i cheated earlier i actually launched another framework i launched sinatra the same way and that was persisted so it actually brought up sinatra as well and what i can actually do in order to show you what's actually running by a marathon we can take a look at this so we're running we're also running a fu command fancy program but this also shows you the ports so for example if i want to connect to if i want to connect to sinatra hello world so so we basically launched a sinatra but it's it's pretty nice because we can also scale those and this is the part about the scaling that you had actually that someone had mentioned earlier okay so i sinatra and 2 h so what should now happen is i should have another sinatra up and running and i will query my end points to figure out where that other sinatra app is running oh there's another one running on so there's one running on three one zero zero one and three one zero zero zero so let's try this out yeah so same fancy app that's running and actually one more one more thing we did build well i have to figure out where where marathon is actually running i actually know that so we build a we build a ui i don't know i think the the resolution is going to be terrible but we built a ui for marathon uh last night and and that gives you here this is kind of the ui you see the applications that are running you can launch a new application for example i could launch hadoop but i'm not going to try that because it's likely going to fail unfortunately there is the resolution the resolution is really terrible um is this better and the it's not very bright but essentially what you can do is you can you can put in a url here you could put in well what we can do is like we can just kill chronos here all right bye bye okay so okay what i need to do is i need to so this is the command i can run so first of all you should trust me chronos is no longer running it's dead we paste the this is not the url this is the command if i do a typo please let me know okay so i don't know if this is going to work but we'll see oh wow this unfortunately it's dying why is it dying i kind of knew that was going to happen all right um give me one second we'll figure this out i think something wrong with her well the ui doesn't work as well as as well as i thought so um maybe this was maybe this was the issue we'll see no i remember what i had to do i had to do some some crazy stuff with the ld library path oh gnarly um all right we're almost there we'll try this again and by the way as a uri you could be you could be putting an hdfs hftp https whatever you want in there um s3 is not supported right now but you could okay let's try this again this should work because you've seen actually that it works via the command line interface so ah but actually this time it works i must have i must have uh written something i must have put something weird in there so so now it should be up and running again oh yeah so here here we go no died right away why it was running for a second wasn't it so for some reason the coroner's launcher doesn't like doesn't love doesn't like to run right now well you'll just have to trust me that the ui looks pretty and will eventually work but um anyways um i'm going to move on because i i don't think it's worth your time to debug here right now but um yeah so so that's kind of a marathon we'll release actually marathon pretty soon as an open source project and if you guys are interested i think twitter is also releasing their project which is aurora it's kind of a it's it has many more features than marathon marathon is a it's kind of the sinatra and aurora will be the rails and yeah so um i guess marius do you know when you guys are releasing that but um anyways there's some there's some resources on on um the stuff that i have been talking about i hope you guys uh enjoyed the talk and let me know if you have any questions you have so we have built something into into into marathon so so one part of marathon that i didn't show you because we have don't have a ui it is we have aj proxy configuration into marathon where we have a local and in fact uh we uh while we were at airbnb we did that already so basically all of the hosts to run h.a proxy in tcp mode on localhost we don't rely on zookeeper uh well we rely on zookeeper but we pull the zoo keeper configuration dump it into an aja proxy configuration do a graceful restart or reload of the configuration of aj proxy and then basically any app can just connect to a global port on localhost and will be directed to the to the right server with that with a different with a different port so it's because whatever poor mesos gave you uh as a as a resource uh and yeah so so that actually works and uh we're gonna open-source that as well so yeah what kind of uh so load tests of mesos have shown that you can run around you can run on the order of tens of thousands of slaves with 200 frameworks and still get scheduling in less than one second so uh yeah i mean that's it's really like it's really fast if you if you anyone is a c plus plus hacker you should check out this uh my friend ben wrote this library that mesos is heavily leveraging is called lip stout and lip stout actually looks a lot like scala it makes your c plus plus code look really cool it has tries it has option and all of that stuff so you should lip stout it's uh github.com third party slash lip stout like stout like a beer yeah no questions so you can add tasks on the fly right so can you also like increase uh like let's say about framework running and it wants 10 cpus can you pop it up to 20 you you could you could do that on your you could do that in your framework um i mean yeah you could do that in your framework actually that might be a pretty cool feature um also to to for for marathon because right now you have to give it when you launch the when you launch your job you have to give it the resource consumption it wants uh yeah i i totally i i think that that'd be a great feature to add yeah cool well thank you guys for for going hey guys