Scaled: Ben Hindman: Scaling Clouds as Operating Systems

Scaled: Ben Hindman: Scaling Clouds as Operating Systems

Recording: Scaled: Ben Hindman: Scaling Clouds as Operating Systems

[Music] lower body Milano's Alexa crab Robbins in the second episode of our scale podcasts and series where we explore skilled companies people in software and a very fortunate to be here at d2 I keep with Ben Hillman who is the co-founder of Detroit even the creator of messes partial messes orchestration framework and first I'll ask them what is messes what's messes well message is a piece of software that we built how we started back in 2009 so that's been quite some time now and we build it to make it easier for people to build and around their own distributed systems at 2009 the challenges that we were seeing where people were having difficulty trying to either scale up things like Hadoop or build their own versions of Hadoop build their own new distributed systems and from our perspective we felt like building an a distributed system should be as easy as writing a piece of software for a single machine and really we just needed more software to make that easier so we effectively created this resource management platform that's what Mesa says and and made it so you could build all sorts of other things on top probably the most famous of them that we built on top was Apache spark mm-hmm which has been oh very well they successful which is that remember was a toy application a that's actually exactly right yeah when we first built it we were just trying to prove the fact that you could build Institute system easily I think the first version of spark was maybe only a thousand lines of code and you could do a lot and that was really really speaking to the power of you know what leverage something like mesas for for for building and running distributed systems so a lot of folks now take you know clusters and super sister for granted right so the whole cloud phenomena appeared after that shortly after that or in parallel so can they bring us back a little bit to the motivation what is it where you're scaling yeah right what was the pain point which was hard to scale yeah well it's a multi-faceted answer really first I think the hard part was from a very technical perspective there's no there was no way of being able to write software where you could just get resources from somewhere in the cluster mm-hmm we just we didn't write software that way we have that today but it's still pretty pretty coarse-grained I'll talk about that in just a second but the analogy I like to use is when people write software on their laptops or for you know a server machine or further phones and they want to do more computation they fork a process or they create a thread if they want to do more things they allocate memory that's just what they do that's just what you write is as a programmer you do these things to get these resources and then you can do your computation the way that it works in 2009 if you wanted to do more things in on a distributed system as you you know went to another human and you said can you allocate another machine and then with that other machine I'll run this other application or I'll run another version of my application and so forth and so on and I'd to me that that was sort of that was sort of the fundamental the fundamental issue you know the fundamental challenge in scaling distributed systems was there wasn't this natural it was an immediate impedance mismatch between writing the software as a programmer and then running that software in the cluster and and sure we had frameworks at the time they were being developed like dupe but they were specific for data science you know they were specific for doing analytics they were specific for doing big data hmm and and really you know the Holy Grail I think is that you have a you know programming model which naturally captures this distributed setting and that's what we really set out to do with maysa so that was that was kind of the fundamental limit from a burglar perspective of trying to scale you know building new distributed distributed systems what was really interesting from our perspective was while we the way that we approach the problem was let's make it so more people can build distributed systems as I think often happens with with technology you know you you build something with a particular problem in mind and there's a different problem that ends up being the thing that most people end up using the the software for are ours wasn't necessarily a different problem but it was it was that whole issue that I talked about where you have to go to a human to get a machine hmm that was where everybody really focused on the resource management you know crazed so lots of different projects came out that was really sort of trying to solve that particular problem and just make it so you didn't have to go to a human to get more machines to run things mm-hmm but less of this programming model thing you know it was it was still it was it was it was less of people writing code where they could just get some resources from somewhere in the cluster and then they could run their computations and more just like hey run five of these but I don't want to go talk to a machine so she codes gonna figure that out but not the code itself is the thing that's requesting the five machines does that make sense so so and that's sort of you know I think that was really what kicked off this whole container orchestration phase is you had this strong desire by developers and and and operators alike to to make it so that developers could very easily get resources and run some of their applications on it but not from a programming language from from from you know an API call and of course everything in computer science is a little bit blurry because the minute that you're making an API call like isn't it kind of like your programming yes yes it's like where is your wrong time yeah yeah yes so so would you say this was kind of the in line with the DevOps movement make developer run operations through software exactly right and so this is cattle not pets for that right so you don't want to know the names of this machine so what was the typical use case so what are the resources right so we have typically we have memory we have disk right we have faster CPU is this the class of resources mostly yeah III mean I think what's what's really interesting is that in the early days we actually spent a lot of time thinking about how we could expose all of these resources in particular disk and and and really there was such a benefit that I think people saw from even just CPU and memory nothing else I mean to be perfectly honest there was such a benefit in having some layer that could just run these applications instead of a human that some people just didn't even care how many resources were being consumed in a lot of cases machines were so underutilized that any system you know resource manager like meso so a resource management I left like kubernetes or any of these systems even if they just didn't do a great job of running multiple applications on a single machine there was tended to be plenty of resources on those machines that people didn't like they weren't even specifying necessarily the right amount of resources that they needed to run their applications hmm the resources would just be consumed by the apps and everything worked now as people started to push the limits harder and harder they cared a lot about how many resources were actually being consumed by which apps and we started doing some really interesting things when it came to things like oversubscription you know learning what was actually the right you know allocation for the individual applications there's a lot that happened happened over the years and storage I still think is one of those ones that will grow over time we focused on it really early because the most difficult distributed systems to run are the ones with state and so it from the early days we really wanted to figure out what primitives we would need to build to make it easier to actually run stateful distributed systems and so obviously figuring out what your storage resources look like and how they can be allocated and how they can be split and shared and all these things was it was really important for us it continues to be important for all the work it's funny Larry because now this whole serverless revolution came to the point of stateful server that's exactly right they discover state and so no it's right how do you do that yeah yeah no it's exactly right now when you you know when you describe basically the kind of the golden early days of messes yeah I just realized this is basically your little private cloud like you basically people like you know if you give people free Amazon they will just spit up a bunch of stuff Yeah right like why should they care they just pick up a bunch of instances the corporate datacenter and then I need to pay for it though just so you can just do a good job I spin up all the machines and just yeah yeah I mean I again I think with the cloud and the ability to hit an API endpoint that gives me a new VM or gives me a new container I think we're getting closer to this realization of being able to truly program the cloud but I still think we're not there yet this is that you know I think we'll be there when people build software they build applications where when they want to do some other computation just like they would have you know allocated memory or fork to process or or spawned a thread on their single machine to do that computation they can do the same thing but now it just runs somewhere anywhere in the cloud I think that will be true cloud programming I don't for sure that programming the cloud cannot be placed much of llamó yeah that's exactly right like this is you know I still don't understand you know the progress is not linear because this seems like regress to me so but you know let me kind of dig a little bit into the history because I think what I'd like to understand in this series is really we have some inflection points and some things first of all they scale other companies and the things themselves are scalable what is the metric of scale I won't understand this a little bit so so you do this research at Berkeley right and and so this was the research project a Templar Apple yeah right and and if so can you talk a little bit about the original dispersion then he brought the Twitter and how did it catch fire what made it right because I think people had to go on the land and so I think I'm really interested like what like some magic happen there yeah yeah well so the origin the origin actually came out of parallel computing so we were doing a lot of research in parallel computing and specifically new parallel computing models that lets you more dynamically move resources around in these these applications we had these really powerful 128 or 256 core chips that we wanted to much more efficiently run we run with a single application and what we noticed a collaboration between myself and a couple of other folks that were doing a lot of Hadoop work was there a lot of these similar ideas of trying to scale Hadoop on distributed systems in fact in some cases I had more CPU cores on those 256 core boards than we had in our clusters of Hadoop said the that was really the origins and again you know everything I was discussing earlier about you know the goal of kind of creating this truly programmable cloud or programmable a distributed system was the goal and then Twitter amongst many other big big big companies started to pick it up because they saw as you pointed out earlier this this devops thing happening mm-hmm and this opportunity less to get people so much to program distributed systems and more to simplify it so that people didn't have to go and talk to humans to get to get to get their applications actually launched in the cluster and it had a lot of benefits of by doing it through an API like this these companies like Twitter and a bunch of other organizations they could abstract themselves away from any particular cloud or any particular virtual machine-based system whether they were using OpenStack or VMware or any of the clouds they could kind of abstract themselves away from all that and just say alright here's an interface where we're you know developers can show up and they can be responsible for operating you know DevOps they're they're their own applications and again you know that that was sort of it that was you know you talk about sort of inflection points mm-hmm in the industry there was a very I was very right that was a very violent unity for you know or people had been doing a lot of puppet yeah they've been doing a lot of shaft this right they'd been doing a lot of these these other other tools which are all great great tools that sort of helped us in this progression of you know where we'll ultimately get to as it's an industry but I think still think it was a lot of work you know you talk about people writing the amell's it's not the the right strategy well the stuff you had to write in the chef world or the stuff you had to write in the puppet world and then still the interactions you had to have with humans to do all that it was still just too much I did that I remember that it was basically a bunch of bespoke Ruby which includes other Ruby and so everybody had their own I just critical representation of the system yeah but the ground was laid by the infrastructure scold exact rights order to appear exactly yeah yeah exactly you know that that that the ground was laid there and then I think people were very very excited about this opportunity of going one step farther beyond the chef puppet world this is something even more automated mmm-hmm I mean that's really the opportunity that we had with with Mesa set places like Twitter you know and then a bunch of other organizations started using it Netflix Apple uber you know a bunch of these big organizations that I think it was also a good time because there were a lot of engineers there that weren't excited about the chefs and the puppeteers no they felt like there's got to be a better way and they had such massive scale compared to a lot of the organizations that were doing chef and puppet and it was just fine mm-hmm that they were looking for something else they could really help them scale and that's that's exactly what maces could could help them with now again people weren't using maysa with that original intention of programming their own distributed systems you know really you know for this data center they were using me so basically just to launch you know and to em instances possibly dynamically but you saw some number of instances of their applications right kind of remember what was instrumental in heaven meses adopted it with right so because he hears you know PhD candidate Berkeley right coming in to Twitter and here's a bunch of other work guys who certainly need stuff running yeah right need stuff up yeah and and and obviously this is there is this new thing yeah so I think it's pretty magical that you know and they obviously this so the curve of their own data center yeah and they're having some patchwork of scripts yeah keep at the top right so so who took the risk how did this actually materialize yeah yeah well you know I think one of the things to me that made you know my experience at Twitter so memorable was there was a very real need and at the end of the day as we all know a lot of things only happen in industry when there's some real value that that's associated with it as well as I think there was this intellectual curiosity mm-hmm and that that's the thing that to me made the time at Twitter so special is it wasn't like we just went in there because there was like some need and we did it there was also this intellectual curiosity of hey can we here at Twitter introduce a new novel way of getting people to actually launch their applications but the need was they had puppet know and a lot of it and people were effectively having to map mm-hmm which machines ran which services there was some Excel spreadsheet or spreadsheet that was attaching all this stuff is just chaos so there's it was it was nuts I mean it wasn't chaos it was state of the art mm-hmm but it was state of the other so we discovered was Excel spreadsheet so service discovery was yeah let's figure out what IPS you're actually gonna talk and and what happened during that time at Twitter which was so fun and and and such a memorable part to be a part of was this whole idea of like alright let's let's decoupled is stuff let's do service discovery where applications can actually just say hey where is blah blah blah service and then I can talk to that kind of sounds like DNS yes but something that massive scale and potentially changing even faster being even more dynamic than something like like DNS um we used zookeeper and we build a bunch of software around that so things like that we're evolving other really fun things that were evolving were decoupling services into smaller and smaller parts I think it's a natural thing as companies get bigger as they can't have hundred people all working on the same application it's just natural of that happens we actually used to talk about it at Twitter as as Sowa as software or innate architecture just without the message bus because ooh keeper and some kind of service discovery mechanism mm-hmm to figure out where other people were and then did point-to-point communication with also each other those those different services versus say the so a days where you stuck something on the message bus and everyone listened on the message bus it wasn't until more companies started to think about that and we're doing that as well and more people talked about it that that in term microservice really came out just sort of what we were doing there and all these old services yes service-oriented architecture and they were real small and then and then then micro service became yes Micro was not the new era in services it's just different yeah name yeah but you know you talk about scaled and again inflection points the inflection point for an organization like Twitter was they couldn't scale the engineering organization unless they thought about different technical architectures who are what they're wanting to do and it was really a perfect storm if you choose to break up your software into many different pieces then you either need a lot more humans to manage running all those different pieces or you need some software to help manage run all those pieces and that's really what missus was at Twitter yes it says a good storm the company was scaling we were changing the architecture to deal with that scale and and maysa was able to come in and really help so they're kind of in terms of who kind of was your champions so the first time I heard about masses force from Morris Erickson mm-hm advised kind of informally my startup versal he we invited him because we did Scala and he was well known in the skull of world with finagle and and he actually mentioned message to me so in 2012 it sounded like revelation just tobe like you know it's just launched of these machines and so to me I really needed from this point on yeah there's this guy Pantheon man she's not really like talking too much of composition so you know like I am going around like he's a Twitter so from this point on you guys were on my radar and at that point I remember vividly it sounded like magic books we did saltstack which was a Python version of puppet on share which was a bunch of crap so whatever reason there is something not a bunch of graham people used to note in the same league yeah right so so Marius was the clear champion of messes he just you know he talked about it as as a given it's like it's something through the users so we're engineers like Marius you're champions right yeah was it and like how do they get buy-in from the VP's like what what was the process there yeah well so first one of the things you just mentioned that I just feel like I have to have to repeat there's a great quote and I don't remember it but something is a you know every every good piece of technology has a little bit of magic or something like that like I always feel like when I see some technology my cool magical it's always fun yeah so I again I I think there was a perfect storm at Twitter in in our case we had some strong engineers like Marius and a bunch of other people like that that were advocates of us actually trying to do this mm-hmm but then in addition to that I actually when I first got to Twitter I worked with the chief scientists there a guy named Abner Chaudhary and Abdur had actually come into twitter through an acquisition of his company doing search mm-hmm and he was kind of tasked with you know bringing in I think people like myself to think about how we can do some smarter things inside Twitter across the whole spectrum great across data science across analytics across infrastructure as well and so myself and a couple other people basically on his team kind of had this Charter to go and see what we could do from an infrastructure perspective and so combined with him and engineers the engineers like Marius there was a lot of support for us to actually try to make this happen and then I think as we made progress and we started to actually move real applications onto the platform more and more engineers within the company just kind of were clamoring hey we want to do this too we want to make our lives easier - we want to take advantage of this - so forth and so on so eventually we had a bunch of people internally that we're interested in doing it not everybody you know some people still you know we're just happy with do it at the old old-fashioned way and that's fine and so we did at some point there was a mandate at some point that everybody had to actually move and do it in this way but um but but but we had plenty of people that wanted to take advantage of the tech so again I think we were really fortunate and again Twitter was scaling and that helped tremendously to getting people that were thinking about what could we do to actually help help the company scale even faster it was a really fun time I know you came around Twitter at that time there was a bunch of great tech was being billion yeah you mentioned finagle and and there was a bunch of other really really fun technology that I thought was was getting developed at the company and being open source at the Twitter was was a you know a good you know icon at the time for being strong supporters of open source and getting people to connect it was even its own github org right so basically the fooding year-old open source yeah from using it in production to me that was very impressive yeah I was it was really fun so this I mean this is great insight that if you have you know a chief scientist who is motivated to bring this you know operations solution right I think this is really this is a very interesting to me a hundred percent you know I I'd you know when I think back on the all the ingredients that were necessary for some of this stuff to happen I think that that was a critical one you know to have that champion and not just for him to be a champion but also for him to have you know the direction kind of given to him of like hey what what kind of things can we do I think there's a lot of organizations that have R&D organizations and I think that actually there's some really really good stuff that came out of this one some of them I just kind of almost they just exist and I think this one actually produced a ton of good stuff you don't know it strikes but this is very interesting because I you know work with Twitter on committee syrup we do you know subscribe meetups there it used to host scale by the bay it's been a major sponsor this year and so my goal of 0 was always to bring together end to end demonstrations of Technology right and I always wanted to see you know like they do in life of a tweet how it passes through the system from API of the user to the internal systems and then passes through all this and some ads are run and you know inject his money growing on some bar shores alright so this is let's do what what I always wanted to do it has was very hard because it seemed pretty solid and I basically thought this is the just just necessary feature of a company growing that you know everybody becomes specialized and fanatical people often angle people machine learning is here and database is here when happen and so so but what you described is cross-sectional and I did not sit myself at this scale so I wonder if this is it was easier because the coming of smaller and more fluid and they did a lot of transitions and so I think like from from a describe I think it's very important to keep this spirit and dynamic alive right that if the more you grow I'm not just like people just become settled in their jobs right and so in the way they they they it's very important to think of it much harder to bring like a massive change like this the bigger company yeah so that I think that's also probably a good point in life will be to Scala yeah so the quarter the right moment yeah yeah so I wonder now kind of I'd like to take that and so you know you took this framework which succeeded in scaling Twitter right and you took it it at the wild and so was the idea that now you will scale the world you skill others yeah it will bring this so I will learn now how this works out right rubber hits the road yeah so so this is a little bit different so so you kind of you you you saw this work and I remember floor did this at air B&B and they did Cronus right and so they implemented as I remember kind of indestructible Bosch right running jobs for Airbnb and I think that really worked out right because they did this graph of jobs and so I went a tech talk at their B&B and I saw them presenting him like I think they said they trained civilians to run jobs yeah on Kronos yeah right so was that the first kind of proof of concept you this can be done outside yeah well so yeah so so so flow my my co-founder here at D to I Q he so he was at Airbnb at the time and you exactly right he built he built Kronos so what was Chronos to flow used to he'd get into the office on Monday and he'd find out that all these batch computations that they'd been running since last week had failed over the weekend you know failed on Sunday or failed Monday early whatever it was and he'd have to go and he'd have to know figure out you know which machine was down and then come and get the batch thing back up and running and get everything going mobile sometimes I'd take him a couple days you know Monday Tuesday Wednesday then he'd get it up and running and then Thursday be like cheese okay my weeks over that was it's awesome and then he'd start all over again on Monday when someone failed so he basically proposed to me said hey you know you've got this thing mesos I want to build a distributed system that's gonna run a bunch of other things so that when stuff fails I want to have that API I've just been able to say just give me some more resources and I'll run it somewhere else and it was a different programming model than say Hadoop again there were you know distributed systems out there but he's a vastly different programming model it was a dag just as he said it was half a directed graph and so he needed to kind of create this system for letting people express their gags and then and then go run it and that was Colonel Sam is hugely popular it was one of those funny things where I even remember when when he was building it I don't know that he he being Flo even really thought how valuable or popular it's gonna be I think he was just trying to solve a problem I'm he's just trying to build some tech and solve a problem and then all of a sudden we started to hear all these people they were like oh I have that exact need you know I have that exact need of being able to run this dag of things I have to get launched and I've hacked up all these scripts to try to figure it out and deal with dependencies and follow along and this is perfect I can just do it now at the same time the other part that I thought was so was at that time in an interesting aspect that I think is really relevant is there were commercial pieces of software that did this and I'm forgetting some of the names of them right now but there were a couple of things that we would show up and they'd be like oh yeah we've bought IBM blah blah blah and it does exactly this thing for us and then these people would be like wait yours is a hundred percent open-source free thing I can just download and run it you know I'm spending all this money it's a big company so so I think that was kind of one of the really interesting aspects to me was this was before I really became an entrepreneur but I'd you know there was this moment of thinking like whoa you know people need this and there's already a product out there that's not as scalable or you know has as many features or something else perhaps perhaps you know there's an opportunity here for something to go get built but but yeah that was one of the first companies we started working with outside of Twitter and then you know again you talk about scale there was a little period of time where I gave a lot of talks and did a lot of meetups and had to sort of scale myself to really meet with a bunch of companies to talk about what we're doing and I still think that that's a pretty important part of all you know scaling all tech projects is you you sort of do have to go pound the pavement and chat with people mm-hmm I think it depends really on the project but especially if you're trying to do something that's a little bit different that is in the face a little bit of how people have been doing it and has the risk of totally and completely failing and not being successful I think that that that you know it really requires you to go out and and you know and preach the good word you know and get people excited about trying it for themselves so that was kind of the other thing that happened to really get it out outside of Twitter and again I think Twitter did a great job we hosted a ton of meetups at Twitter to talk about what we were doing you know we wrote some blog posts from from Twitter we did a bunch of those kinds of things to help help get it get it out there and and and it was you know while the successful as I mentioned there was a bunch of organizations that were able to take it and and and even to this day run it on millions of machines right so basically so again I think there is some magic going on here because there is this open source at heart company Twitter right and now there is Airbnb with first of all really valid business so the result of data is coming in jobs around but it's not it's not a software company per se right it's it's it's a resource management in the real estate it's kind of weird way to to mess about with buildings yeah right and people and and so they they just have a need to run stuff and by virtue of this economy where people move around right so flow comes there he comes with open source background obviously you know nobody is gonna buy myself commercial thering right people are gonna use what they know yeah and so this is interesting to me and obviously I think around this time probably if you would describe my sister bunch of other engineers like myself then they will jump on this right so so it's a you know I wonder one thing and if a little bit going to puppet and chef can you like I I have a feeling dealing with them that they're basically kind of the the rich the limit of you know BofA and the Western operator from hell ability so basically there is this isn't man who juggles more and more stuff with his association scripts and bespoke bash scripts and Ruby right and so so in some way these are the things on steroids she reached the natural ceiling of manageability so any engineer would represent messes now kubernetes right that's why I think people jump in a cab religious can you can you know kind of formalize verbalize what is if you would have to kind of separate in hyperspace puppet and chef and soul ii systems like that from meses incorporates what is the key difference yeah so i mean the fundamental difference with especially the original designs of the salts and the puppets is this actually the chefs and the puppets is you were well let me rewind just a second how did we manage computers to begin with well first we didn't have a lot of computers so most companies they had a single server and they had all their web requests or you know whatever was happening would come to that single server and they do all their there's some and it was stateless they were just kind of serving web pages right when they started to add more machines because they had more users or whatever it was you know when you have five machines like I can remember five IP addresses yes I can remember the complete IP addresses I can remember five passwords I can enroll at stuff I can like go get the other cat and so aids exactly you know you make it super simple and then and then you handle it I might be able to do ten you know maybe I could even do twenty but after twenty I'm probably not going to remember all those things anymore so what did they do people started to take all this information the IP address what operating system was on that machine what you know users were allowed to go access the machine what applications and what did they do they wrote it down somewhere right because like they couldn't remember it so they wrote it down and in a spreadsheet or they wrote it down on a sheet of paper and like hopefully they didn't write it down on a sheet of paper that they spelled coffee on then they you know didn't have that information anymore then they started storing it you know on actual files so birth so there's the chefs and the puppets they came into this world of saying let's take that stuff that people are writing down in an ad hoc way mm-hmm and formalize it so that's and do the automation of when someone writes down on a sheet of paper this machine should have this operating system with this these libraries and running this application then some some software can just do that instead of a human having to go in and do all those steps the stuff the human had written down now now just gets rung but and this is the key difference it's still about a single machine yes the chef's and the puppets of the world were still about setting up that single machine and kind of having this perspective of MIT this machine based perspective hmm versus having the application based perspective so the application based perspective is you know it's not what runs on this machine the machines are cattle or as you said earlier you know the chef and the puppet model was still a pet this machine was a pet this was the manifest that described how that had operated right in the maysa world in another kubernetes world it's a cattle it's just I separated out these pets who the machines are and I just take these applications and dynamically I figure out where I'm gonna actually run run the application so application mind mindset application centric mindset so I wonder if this came from you know because you come from from research world where people actively avoid to be insisted mills right so so in a way mmm you know a typical you know popular Chef user it's a kind of really you know glorifies this admin who loves his mission because she said me loves his machines he wants them perfectly configured he knows them and he doe he doesn't want others to mess with them that's exactly right and engineers don't care about all yeah yeah right so I think that's that that's I think that's very good yeah very good division you know it to me it's one of the reasons why I also don't think that that the MS you know have an infinite life I mean they're gonna exist for a very long time and of course they were wonderful addition in the work you know in the path of auger thing we're doing a computer science but the sysadmin who loved the VM because they could finally get their users to stop mucking with the actual physical machine and the SIS admin's could just give their users of VM and say there's your sandbox now don't screw up my machine yes but they weren't really helping the end users because the end users still had a machine they had to do right you know the reason why Isis admins that existed in the first place was to manage the machine for the people now they were still giving a machine to the end users and they still were just screwing it up just as much as they're secure the machines up to begin with right you know I think the real key thing is I you know I don't think people actually should care about the machines you know when you're just writing your applications at the end of the day I think all you should care about is the resources right do you have the resources you need to do your computation the answer is no and the next question is is are they somewhere available where you could word that you could access you know and that doesn't have to be I need to go talk to a human to figure out that those are available that should just be as simple as like right it's like nobody should sanctify some weird UNIX box which grew out of you know Bell Labs people run out of memory and haven't been user Bill and like forever perpetuating that we don't have to do that yeah totally yeah that that's I think the phase change ultimately that I'm interested in seeing and in the industry is this one where we get to a place where you know when you build an application and you deploy it's just it it itself can find its own resources from whatever it's allowed to talk to you know which could be a cloud a private cloud it could be some edge device whatever it is you know if there's resources and I can run my app if some software can run more stuff for cath read on those resources that should be able to do it that's the future I hope all right so no we have messes in the wild you have a company and I remember early days I said I think I've been at every mesosphere office I wasn't this kind of chapel building yeah right and then the spaceship building and now this actually my former nitros office where I run a bunch of meet up so this is all kind of a big circle right and so I remember you know at some point you guys are basically like there are different positions different pitches right I think you pivot it as any so I think at some point so early on I was pitching to Paco mmm all right that you guys should do enter him data pipelines because I came up with this Mac stack with wideband and but you guys are focused on data center utilization right so your your target audience you thought we're managers of the center's who should really save electricity and fill the machines of the help right and I think recently we've I've seen kind of the AI focus right and and so so can you kind of describe to me how did this evolution appear let now you know you you know the magic of scale you know how the scale stuff so you want to bring into the world and see where it hits what the customers who have the need so you need to vote it's super complex because you need to evolve the business model but I kind of in my mind that the magic of scale is driving this because you can you know skill something and so you need to find something you can scale and I remember you even had a carnival cruise ships as a customer and I think it was like so here's the cruise ships resources right so so how how did you find this kind of niches where customers need scale like how do they realize they need scale how did you convince them this is the right way to scale how did this thing about yeah yeah well I mean so I think the early days there was a there was always this question of should we as a company go and focus on helping people do the resource management in their clusters for I'll call them you know the web apps or for data apps yes that was an early early question that we kind of went back and forth on and yes smack you know spark Mesa osaka' Cassandra Kafka that was a that was the data apps right the people that were trying to do is really do things and I think that was that what that was a area in which we were wildly successful mmm-hmm but I think because the people who were doing smack were already kind of they were already kind of Greenfield you know they were already they were already kind of at that place in their in their projects or careers or anything else to say oh you know I want to do this thing where I've got this analytics that's running that's you know storing data in a key value store not a you know database but a key value store that's using a message bus to doing a bunch of our processing that's using actors to you know to serve all this stuff back out to the users that was kind of already you know the people that were doing this actually more greenfield one of the areas we wanted to focus on in addition to that which we obviously spend a bunch of time in that but the other area that you mentioned that we thought we would get the massive scale from was I'll call it the brownfield mmm-hmm the data center operators the cluster operators who just had tons of apps that they wanted to run in far more efficient ways to drive up utilization mhm and that was that was kind of this in this other area now in our case you know to help to get to the scale that we're at today we honestly did both we honestly put put put time and effort into both of them mm-hmm but I think that there very much was a strong interest that we had in in you know getting the brownfield apps to run probably because that was what we lived through right you know at the Twitter's we lived through really taking these more monolithic esque apps transforming them and and and and you know helping everybody in a big way but I I'd you know I think that it was easier to talk to people about the data apps the smack stuff than it was to talk to people about that the Brown show that was a big learning for my know over the time I think I was more expecting that we would just get massive scale quickly with the brownfields and actually we had to do a lot of work to get people to actually want to to take that on why well you know sometimes when stuff just runs people don't want to touch it even if it's expensive you know it's like well it's just kind of working it's doing its thing perhaps we shouldn't actually make any changes right now and you know even if you could save a bunch of money you know that yet money talks sometimes but sometimes it doesn't people can't really sometimes put a value on the ease of not having to actually deal with stuff mm-hmm to this day I still think that there is a massive number you know of applications out there that if we put a little bit of effort into them could be moved to you know platforms resource management container orchestration of platforms that would save a tremendous amount of money and electricity around the world mm-hmm but again I also don't I don't see that as much being the focus I still see the focus in the container registration space being for a lot of the greenfield applications and for container orchestration less so for data apps and more and more for for for these web apps that we were talking about earlier I just thread on the way here that if you binge watch several episodes of snowflakes it's equivalent to driving for miles because here wasting the cloud resources yeah so I think you're right like if we go to cross kill this is actually more and more relevant yeah right and so I won't actually explore one thing so recently there is this new company created by Brian Cantrell who is also speaker and skill by the base as you are called oxide computers and so they basically want to develop OAM data centers so they want to just build data centers with the idea of running them properly yeah right and so and using rust as a kind of layer to do this and I asked Brian kind of you know why not kind of put passes on this and so forth and so I think the kind of my understanding is that some Isis software system seen on top of it taken for granted existing hardware architecture and DCOs was the offering right and so the question in my mind is you know why if we disarm everything we should disrupt the data center we shouldn't take hardware as a given right because it was created by different people for different purposes so this brownstone projects right they just bought a bunch of stuff from best practices as they knew it the CD was and CIOs and right and so it's like in my mind it sounds almost like we need to do to the hardware of a data center this primitive early Hadoop which is a bunch of primitive species which we essentially build for different era and control plane slept on top of this so that's I think that price has a very kind of solid idea so what do you think about this and kind of did you see that you need to actually reach inside the hardware instead of working around all the stuff with software and all these resources do you see kind of evolve it into taking over the hardware and doing software hardware co-development yeah I mean I think it's a really really it's a really interesting question I think theory there from my experience but what I've learned is at the end of the day if your mental model is just I've got some computation and I want to run that computation somewhere mm-hmm I think what that's wherever that somewhere is it's gonna have to have something to execute code mm-hmm a CPU it's gonna have to have something to store memory you know whether that's registers a cache actual main memory or a disk right so then that's a third one it's gonna have to have somewhere to you know just potentially store things even longer like a disk but maybe not maybe not I think at the very least it's got to have memory and some kind of way of processing it so if that's kind of your mental model then I 1000% believe that there could be a future world where we're creating hardware which is not like your standard form factor hardware where we clustered a bunch of machines together to run I do hmm because that's what we had I could very much see a future world in which there is hardware that is kind of purposely put together for running in particular kinds of way whether it's because it's got accelerated FPGAs or it's got you know other kinds of hardware advanced advancements that certain software it can take advantage of and then if those things kind of show up wherever you know from my perspective I think you know everything's fair game mm-hmm the things we're gonna need to know from the software perspective is how far away are these things from one another mm-hmm from a latency from a bandwidth perspective so forth and so on what are the failure relationships with respect to these things if this one fails that this thing also going to fail so fir and so on but I don't think it needs to be an x86 box you know I don't think your standard box I just think it needs to be some kind of resources that I know I can actually run some stuff on I don't even think it needs an operating system I don't think it needs you know we're gonna Linux or you know it probably helpful if it's there but I think it doesn't need that what it really just needs that you just need the hardware you need some software that's compiled to be able to take advantage of those resources on the machine and then you just you just want to run that so I do like the idea of of you know taking a step back and saying hey can we potentially you know think a little bit differently about about hardware how hard it gets built but I also think that we're really going to need some software to help with that you know it's not just gonna be hardware it's gonna be software and I'm biased I'm more of a software person than a hardware person I think that if you did some really powerful things with software you could get away with the current way that we're doing hardware for quite some time I think you could actually you know just keep kind of doing hardware the way it's been doing for quite some time once you had more people doing software in this more abstract way then I think you could start to do some interesting things with hardware so and I mean you almost described a very general way to look at things and I think they kind of enclose the age as this is understood right now right because if they have these little devices mm-hm and you know they have latency then you know they can be far and they can be weak so how does age 50 of current thinking in the Skjold messes I did the age of the are you chief it was DC us implied the DC yeah and so edge is now kind of yeah yeah no I mean edge to me is the perfect example like at that that's the perfect example of I think we're starting to push what the form factor of hardware looks like but still when when we're at the edge its yeah you know what when I'm running software these days that runs on my my laptop I don't think of it as much as a laptop mmm-hmm the way I actually think about it is I have for 2.1 gigahertz processors and this much memory like that's like more how i think about you know it's just it pools of resources the edge is just another a bunch of these pools of resources so that's I think where I where I would love to see us go as an industry is where we stop thinking about particular form factors of machines we start thinking first and foremost just more about those resources mm-hmm just pull the resources chunks of resources and then we start to get program models which are more about programming of this pools resources you know it doesn't matter if Amazon spun up the VM or Google spun up to VM who cares mm-hmm or the resources oh I have a GPU there and they have it FPGA there that's far more interesting to me than anything else oh I've got a you know six gig Nick there and a two gig Nick there you know that's far more interesting what can I actually you know do with those three I mean you have some expectations of there is so much bandwidth you don't have right and I think what's interesting with the blockchain which is a distributed system I think there is no expectation that something has just died and never coming back and partition so does the current paradigm break anywhere for instance you know some some people heard actually advocate for the death of a virtual consistency because the world is so complex yeah you will never have the full view yeah of the whole system so you know like as humans we have locality and if you know a million people somewhere disagree don't care mostly right and so sometimes you will never have actual consistency because just too much too much that you will not never have the full view yeah and also because you have now the edge some systems will never come back and some systems will be guaranteed to die so do you kind of do you think of this how do you manage resources which might die never come back totally I mean the concept of ephemeral ephemeral resources that I'll never get back is that it's becoming a much more standard concept that we think about a lot spot instances on the cloud is a good example and I think that that does have to be baked into when I'm describing those resources I was talking about like a GPU or two gigabytes of something here you need to understand is that two gigabytes I get forever or is that two gigabytes that is ephemeral that might be two you know might be taken away from me and even forever is a pretty harsh word there really know is forever so so even there you know it's it's forever is kind of like a forever which is you know as long as I can but if I've got something really important there I probably have some other entities in the system that's also doing the same thing but yeah I mean if if we you know if there's some catastrophic failures that caused us to lose many data centers simultaneously does this idea of just running lots of software and a distributed way you know break down it totally breaks down but personally I don't think that that scenario is any more likely than a scenario in which we just had a bunch of humans running around doing the these computations you know manually or something and remembering this information themselves you we would do the same thing if we wanted to not deal with failure we'd store information and multiple people humans at the same time as we'd have multiple humans to things redundant lis and all that's in document and there could be a redundant you know there could be a catastrophic failure where we lose all those humans to you so you know I I don't think it's any it's it's any more likely I think it's it's just the reality of of there can be failures and therefore you have to program for that and and yeah I personally think that we could write software which lives and moves around across all these different resources for decades just keeps going you know just keeps moving around and as we should I think we should do yeah so I want to kind of wrap up with the current age of Detroit you and also the scent of kubernetes and coexistence oh so it's very indecent to me that you guys basically took the skill set you have with messes right and you know a bunch of about the citizens every kind of every few years new bunch of people discover what other people knew right that's so kind of it seems to me that kubernetes folks basically found what messes folks knew and you guys are basically evolving by by embracing it right and so now you're called attract you know just mesosphere and you told me that you develop apparatus for kubernetes gonna talk a little bit how this kind of transpired and how how like well you look at yourselves what how did you kind of quantify your scalable skills yeah for this frameworks and how do you kind of apply them to something which you know came from the outside like kubernetes yeah yeah I mean as I mentioned earlier one of the areas that we started to focus on pretty early with my sauce was stateful distributed systems mhm why cuz we felt they were the most complicated ones and so we should make sure that we can run those ones if we ever want to be a place where you can run everything SMAC has a bunch of state yes Cassandra and Kafka at the very least have and then you'd colocation if you know this is much properly auu and yeah you can do a lot of really efficient things if you can manage that the state properly so we did that a bunch in May so sand and and and that was kind of one of the areas in which we we were really successful in making it so you could run a bunch these state fault things so as kubernetes started to get more and more popular it means to be perfectly honest we just had a lot of people in the community that came to us and said hey can you bring a bunch of the stateful ideas that you had in the meso swirled to the a nice world as well mm-hmm one of the interesting things about the early days of kubernetes was people weren't even specifying the resource requirements for the machines there wasn't actually a resource model of how many resources were being consumed or how many were getting allocated you just launch your application and really like say one CPU one Giga RAM and then you use whatever it wanted it just ran and that's what I was talking about earlier how there was just such a pent up demand for separating the machines from the people you know the cattle and pets as a scenario mm-hmm that people didn't even care exactly how many resources were being consumed so state and thinking about discs and stateful stuff was was something that the kubernetes community didn't get to as you know as quickly or spent as much time focusing on is we had in the mesas community and and it was a natural place for us to step in and work with that community and doing stateful stuff so what have you we've been building we've been building these things called operators so in the maysa world if you wanted to write some code which managed other things that were running in the system a great example being Chronos he talked about earlier which managed all these all these computations that are being run in this dag and in this graph in Mesa so she built what was called a scheduler mm-hmm why because that thing Chronos was scheduling which part of the dag should be should be running in what place and in in kubernetes a pattern emerged it wasn't built in that endocrine idea so you could do it this way but a pattern emerged for doing something similar to this called the operator pattern mm-hmm I like the name operator pattern because it's kind of like instead of a human operator doing this some code is actually going to do it instead didn't want us with algebra of some sort yeah yeah we we called it scheduler because we already phoned them marathon's a scheduler exactly or our schedule Chronos scheduler we called it scheduler because we came from the academic community and that's exactly what was happening we were scheduling different tasks and said we call this scheduler yes I mean hindsight scheduler was a because whenever somebody hears the word scheduler they think they need a PhD to go build one of those things yeah yeah exactly yes people get really really scared about this stuff operator on the other hand has less of a scary scary name but it's basically the same concept hmm and and so we've we've been working in the kubernetes community to build um operators specifically around a bunch of stateful storage stuff in some ways we're kind of building the operator for operators we're building a higher level operator which lets you declaratively describe how it should manage all of its different components mm-hmm versus in the Mesa Mesa in the early maysa days when you were to build these schedulers they were imperative you basically wrote code to actually run them and our years of experience have taught us that actually if we can if we can enable people to capture all that that in a declarative way then they could you know write less code from my perspective I always like the idea of trying to do it declarative and then if you can't falling back and having a way of being able to write some imperative stuff to actually get whatever you're trying to accomplish done mm-hmm and that's that's the model that we really been taking do you have the language should you sell for declarative yeah so you're not gonna like this but it's enamel okay that's that's like that's the current state of the art what people are doing that's effectively the DSL is you right yeah Mille and and the project I forgot to mention the project's called CUDA who don't yes yes it's the kubernetes universal declarative operator so Universal operator operator declarative declaratives what I mentioned and kubernetes operator so just it's interesting because it sounds like you're scaling the scale it's a meta scale yeah because you're scaling the engine yeah which scales they're the jobs so that's yeah that's very interesting because you know we have a limited time so we didn't go into marathon and our rora but the thing that's also very interesting kind of a direction of itself so kind of you know I'd like to finish with kind of if you can look back right so we touched all upon Messrs spark which started a stoat project and now grew into a multi-billion dollar company which is key bunch of other things right and kubernetes which scales a lot of things and has similar ideas so I wonder if you can kind of reminisce on this you know somehow all these projects became a skill themselves so you know meses is used at Apple and another and Twitter and coverage is used in a bunch of places and spark itself is you know - cool right so what is it about the software projects with the principles what is it that made them scalable in the real world like they people you know they they can scale companies in different aspects of this companies right you guys are motor operation Science Park more of kind of data pipelines right on often sometimes on top of this other things so what is it like if you would you know kind of cut to distill and open up the questions right we don't bother so this whole series is about it right so so I'm not looking for the full answer but like do you see because you did all the work of messes and you're sort of spark take off what would be your kind of highlights why these things to kovai Mira Don things died never saw the scale which wish these things enjoyed yeah you know I I think if I had the perfect answer for that then I'd be able to just go turn out you know tech and company one after I know this and I know that you you you know that you know there's probably not any any one particular answer but I'll give you my you know the things that I've seen over the years that at least I kind of look for in future future projects one is is there a real need mm-hmm you know and and that that need interestingly enough is often that people are trying to do something and you're gonna make it easier for them to do it mm-hmm so we talked about Chronos before where some people were actually buying this product to do this product and everybody that we chatted with had some kind of you know Frankenstein bash scripts to do it themselves right so I think that's kind of that's part number one it's like is is there a real need that somebody has to that can be solved by some piece of technology that can make it simpler when it comes to the technology actually taking off and and scaling and it may be companies going as well I like they think you know what's interesting is as I say some of this this out loud in some ways this is not the point of academic research is to figure out stuff that's gonna take off in scale yes the point in some ways of academic research is just to try to push the envelope to create new ideas yes to get people to think differently you know you know do differently but for projects that in particular you're trying to get to scale I think you need that need in sparks case we had Hadoop mm-hmm but people were not happy with it you know and yes there were advancements in sparks in particularly Veen things in memory so that we could do different kinds of applications faster mm-hmm specifically things like linear regressions which is really critical for folks that were doing a lot of machine learning so there was there was you know there was something there just like in the Khronos case there was something there but it wasn't doing everything people really wanted to be in Chronos this case there wasn't it there's no dag for what people had there was like new functionality they really wanted that you know if I would have known what I know now I would probably would have created the Chronos company you know because we probably could have just scaled that out massively there was a need people could have moved away from what they were currently using God more functionality and gone but I think that's kind of the number-one thing is like you've got some tech you've got a real need that people have and they're out there doing it in some way you know they've got bash scripts or they got old product or something else that they can get we probably could probably good yeah yeah so yeah you know I think that's probably my biggest nugget of insight there and so you know now even today when I see some new technology projects you know come out or some companies get created that's often the question I ask myself is well how do people do it today you know what's the new value they get from from doing this and then how many people do I think are gonna actually do that and and if a number you know if there's enough of those people then I think that that thing will actually scale what's interesting honestly is that a lot of Technology you know you've probably heard this quote as well the best technology doesn't necessarily win but whatever technology wins becomes the best technology mm-hmm I think that's a really interesting quote because more than once I've seen technology that didn't know wasn't necessarily the best couldn't for example scale necessarily I think kubernetes was a great example it had a hard time scaling to the scale that we could scale it makes us mm-hmm and yet as more and more people were doing kubernetes the technology gets better and better and better and eventually you know it you know it gets to the scale that it needs to actually actually perform at and that's a you know so so I don't always you know I don't think the answer I guess to put it this way when you look at like what like whether or not a technology will go and become super successful I don't necessarily it's how good the technology I don't think that's nearly as important because I think that any technology will could get there if there's enough scale behind it in terms of a community and end users and all those other pieces docker is very familiar to people all right so maybe you know if the catch maybe the kind of gives a spark what I've seen is that because it becomes super easy because just first of all it's old memory yeah and it's you know your expectations are fulfilled because I have a big collection let's say dot map and it's actually there I say dot filter it's there not reduce is there right so it's just what you expect or think docker is a very well expected because it's a it's a it's a holistic image of a machine right there so like I think at Twitter you guys use special packages for our alright so so once you have the full control you actually nobody needs docker it's a it's it's a regress yeah right so but people think of this as their package in the way and then you write like I think this very interesting what will be the end of scale and then we need to run them somehow right so I think that that maybe one of this but yeah I think something should scare maybe some men some some understanding human understanding of how this would be mhm so but I mean you guys did a great job so it's really it's been really you know helpful to get this and I don't think we even scratched a bunch of things so hopefully you know I won't come back and now I seven yeah yeah this project is always kind of is gonna move further but thank you very much man we really create your time and wish you continued success [Music]