scale.bythebay.io: Helena Edelson, Disorder & Tolerance in Distributed Systems at Scale
Recording: scale.bythebay.io: Helena Edelson, Disorder & Tolerance in Distributed Systems at Scale
you this talk is really about rethinking a lot of the ways that we approach scale and distributed systems so just for Alexi I just wanted to really bring out the entire topic of scale I'm not really good to be talking about Scala and this because it's kind of a keynote it's very theoretical talk bringing in a lot of new ideas and I kind of constructed this talk because we all work in and I'm sorry I'm kind of small you may not be able to see but we all work in really different areas on different domains and have different problems to solve but my hope with this talk is it brings each of you ideas are triggered by it and specifically to your own experiences and I'd be really interested if you have some ideas or things you know you want to talk about please come up and find me throughout the conference I'm here all today and tomorrow all right so let's a will so you can find me where I work now and have unlike Dan I didn't really want to list it but at the moment I'm also a committer on fellow DB which my colleague of and Chan is going to be talking about later and the conference I think later today so I've contributed to a lot of different open source projects I've spoken a lot of big data and Scala conferences but everything I've do in the last done in the last eight or nine years has been in Scala and so everything I'm talking about is kind of coming from things that like ideas based on experiences rolling scarlet based applications and infrastructure into production at large scale and so primarily this talk is about concepts and complex adaptive adapted systems and how do they apply to distributed systems and how biological systems innately solve things like distribution regrowth and rebalancing and also the possibility of new approaches to building intelligent adapt systems that are resilient because these are things that for any of us that deal with you know I don't think it's just even infrastructure but any of our systems have to be able to be resilient and survive be fault tolerant and this is just a really repetitive pattern I found over what we do over the years and so the foundation of this talk is based on before I started working in tech I was doing a lot of scientific research in biology in different areas but it kind of centered around finding better ways to handle system dynamics and thinking about how could we re-engineer entropy in systems at a very high level I don't really want to get into it but also it was all this talk was also triggered by things that I've experienced in this industry over the last you know 18 years and particularly the last decade and then starting to question up how we approach these problems of distributed systems and scale particularly at scale and this is really the underlying idea and question problems cannot be solved with the same mindset that created them so there are problems that we have to solve and I'm not the only one I'm sure but we run into them some version of them over and over again but primarily we approach these problems from a computer science point of view and what I wanted to bring up in this talk is you know that there are many other scientific approaches that kind of overlay and could possibly lend ideas to this and it's it's very hard and I talked about this at the end you know we're all on some kind of assembly line if we're not in research trying to deliver products and we have timelines and we have other teams that need things from us and so it's very difficult to step off that typical computer science approach and try and solve the bigger problems and that's something I'll talk about at the end but what I find really interesting is that if you look at other scientific disciplines their technologies are advancing quite rapidly so that's an important point to consider so when we're trying to design and build intelligent systems we could define AI as systems that exhibit intelligence and ml as systems that learn to be intelligent but we often approach systems to have some kind of governing or orchestrating awareness outside of the nodes specifically and outside of they running up applications that we're deploying and since this is all about also scale I from my experience it all comes down to data it all comes down to information that we're trying to pass around in these systems information that we're gathering information that we're transporting how are we storing it where are we storing it what's the replication what's the compression what algorithms are we using when we're doing query patterns and what I kind of wonder and this is I have a lot of questions in this talk to the audience to think about are we possibly missing some kind of common intelligence substrate because we keep you know at every company at every project we moved to every job we keep having to solve some version like I said before of the same problems and I just wonder if there's some way and like I said there's really no time to do it unless you're in research I think but is there something that we could have and I don't mean some kind of governing thing like mezzos or or kubernetes or anything I mean something that actually we're going to talk about in just a few more sites but something embedded in the system so just for a basic side about what we're talking about with systems to be clear we're talking about systems that are comprised systems are entities in and of themselves they're usually in a larger system they're usually comprised of subsystems and those subsystems are comprised of entities and also you have to look when you're looking at these systems we're talking about what are the interdependencies with all the entities and how are they affecting each other and Systems Theory is a really it's been around for a while of course but it's a really interesting way of thinking because it's an interdisciplinary study of systems and it's all about its coming out of engineering originally and it's about discovering how elements of the system and their subsystems interact to produce given States and change it one part affects other parts in the system these are all principles that were very familiar with but what's interesting to me is that out of Systems Theory there was a biologist in the late sixties Ludwig von I'm probably pronouncing his name wrong bertalanffy and he proposed the idea of general Systems Theory because he felt that the scientific investigation needed something even from the technical layer that system theory was coming from needed something more general something that could apply to sort of the more unified paradigms of investigation and thought transcending just the technical problems and then from that and by the way his book is excellent general system theory it's funny how it was here's this book written in the late 60s and a lot of the ideas still apply today because he was coming from a time when things were rapidly developing and and this is I think the same kind of situation and what's interesting coming out of that now that's leveraged more and more as complex adaptive systems theory and I'm gonna say a scary word now complexity theory you know complexity is something that well I've given talks on in the last four years about how to stave off complexity how to simplify systems and your architectures and things like analytics pipelines and so it's kind of ironic that I'm talking about complexity now but I want us to think about complexity in a different way than we've thought about it then then the way that we the context that we think about it now because right now complexity is in one hand something to be avoided and what we do at scale because it makes things are credibly difficult but the other way that I'm the another way of thinking about it is something I want to introduce which is it knew just from another scientific discipline but what's interesting about complex adaptive systems there is it it stems from system theory it takes in two nonlinear systems chaos theory game theory so a lot of different ways of thinking that applied to not just computer science and distributed systems but many other ways of thought and in distributed systems what we all find of course is that the larger the scale the greater you know the more moving parts the more things conducted to fail and they can be tremendous failures when you get into large system particularly with a lot of data but what's interesting is in comparison to that is in biological systems the greater the complexity the greater the overall stability and resilience of the systems and that's really something that I want us to look at like if we could just put aside our like a fear of complexity and distributed systems and what we're deploying and running and just possibly take a look at if we could consider complexity in a different way and possibly apply it in a different way a really interesting application of is the butterfly effect and we see this it's in used heavily in weather prediction but it's all about how a very small change in initial conditions like such as a rounding error and your calculations can steadily amplify to dominate a solution causing extremely different outcomes so in weather prediction initial conditions are very rarely we're very rarely known and in computer science of course we can see something like the butterfly effect with cascading failures when bugs are introduced when do we see a ripple effect throughout our systems and from that came ensemble forecasting and this is also used in weather prediction but where you have a range of possible future states so instead of making a single forecast and this is something that we do when we're building new systems are architecting new systems with our teammates a single forecast of the most likely outcome you instead do an ensemble of predictions produce so like when we're looking at our own systems and building them we can we can pretty easily as opposed to other types of scientific disciplines we can pretty easily list out the things that could possibly happen they're not these great unknowns or anything but we never know exactly when they're going to occur any of those failure scenarios and it's heavily used also in welfare prediction because welfare prediction is highly unpredictable situations so from a human and an animal perspective fire is a very destructive force of course but if you look at it from a different point of view and that's what this talk is more about like if we shift our approaching things fire is actually its destructive but it's also a transformative force you're moving from one state of energy to another state of energy and not only that but you're also moving from you know some kind of state that goes down to the kind of ground zero and then what comes out of that is a completely new system completely new entities in that system that are growing that can be more diverse etc and they're much were much more robust so I want to step back also when you start to talk about a lot of this now I really wanted to add a little bit about entropy events in time particularly for those of us that are working on problems around time series entropy is often for computer science talked about in the term in the sense of information entropy so the the basic idea of the second law of thermodynamics is the law of physics stating the entropy increases and we're going to challenge this as we move into another scientific discipline in a few minutes it measures the degree of disorder from the system and the increase in entropy accounts for the irreversibility of natural processes and the asymmetry between future and past and we're going to question that in a second entropy and the arrow of time so here's a pop quiz who's in physics here physics background okay so if given a complete complete knowledge of the universe for two instances of time how would you solve which instance happened first-order disorder Alexey calculate the entropy between the two snapshots the one with the lower entropy was first so in the wild how would you you might in distributed systems be checking you know with your storage weeding something about your metadata but being able to calculate the entropy of the two snapshots without any of that information if you could have that awareness of comparison you can do that as well and in it and it's very interesting that when you look at biological systems this this arrow can actually reverse but in physics it's a given that it doesn't future light cone so I had to add a quote by Stephen Hawking if the Sun were deceased to shine at this very moment it would not affect things on earth at the present time because they would be in the elsewhere of the event when the Sun went out so what the future light cone is it describes really heavily what we deal with I think I'm dealing with events and distributed systems if you throw a stone into a pond it slowly starts to ripple out and so the lake cone is kind of like from that Genesis event if you sort of layered those that ripple of time over each other it becomes a cone and so from the Sun at one point at that Genesis point it ripples out it takes like what eight minutes or something for light to actually get to the earth and you know what we deal with us with events all the time so something comes to our cluster the first node the first node is aware of it something says so if you ask that that entity that node in the cluster did this occur that first entity could say well yeah of course it occurred but if you ask other entities that haven't you know received that information yet they could say false I don't know it I don't even know what you're talking about and so that's sort of the future light cone applied to what we do and you can think of that as I have a really nerdy shirt that says a long time ago but somehow and if but it's really applicable events lie in the future light cone every everywhere so all of the events that were gathering storing processing analyzing compressing all of that we're just constantly looking at the past and this is a really interesting idea time is a derivative of events events occurring in time as derivative events where they're sequence of things happening in time or what people are starting to talk about more and more with quantum physics is pretty fascinating where time is just a sequence of events or time comes out of it and time is just another coordinate maybe it's more accurate to say that time flows as events happen and the flowing of time or passage of time is events by Anthony aguar who's a physicist and so then given all of that how do we rationalize what now actually is so we could think about now is the moment in time that has just been created in the expansion of the universe and it's just a time moves forward in the continual creation of nows so let's get into biological systems building intelligent adaptive self-organizing systems and I wanted to start with talking about viruses because viruses are pretty fascinating and viruses also exhibit if you look at them and compare in relation to for example humans but many other life-forms they adapt they Co evolve and it's pretty fascinating how they do it and if you can imagine our systems that we build and deploy being able to do something like that without some outside governing body that would be pretty amazing we're to go into that a little bit but viruses themselves are highly adaptive they mutate often and they interrelate with the immune system of their hosts where the where the host is constantly adapting to the adaptations and the mutations that are happening in the viruses themselves and viruses have all these different strategies they have things like mimicry to look like the structures in there host cells they have evasion tactics they have suppression so they can suppress the the behaviors that the body and the immune system would recognize them as what they actually are to actually kill them off so they have they're constantly evolving strategies that allow them to survive longer and be passed on in the genetic or the viral from one organism one host to another and the immune system of the host is constantly adapting along with it so if you think about you know how could we apply this to the systems that we're building it's it's pretty fascinating obviously it's hard to do you know when you have you know tickets and deadlines so if something that has to be deployed but I just wanted you guys to to have something to think about throughout the conference and please ping me if you have more ideas about this so the immune system itself is really interesting fascinating it exhibits a highly distributed adaptive it's self-organizing behavior its self programming it's self learning all of these between the immune system and the viruses are doing self programming in run time basically and another thing that's interesting to look at we see things like the domino effect where one event can trigger changes in others so I wanted to start looking at more complicated biological systems in comparison to the distributed systems let's go that we build this one is really interesting evolution and complexity at the edge and the edge is sort of like that edge of chaos where things can start to crazy things start to happen and this is about the proliferation of species at the transition between land and sea where there's thriving transition zones and the way that these complex and biodiverse systems evolve is kind of through a domino effect so you have a system where you have a biotic conditions and you could think of that as some kind of straight substrate you know we have like our you know meso layer or whatever some foundational system that we deploy to but once lower each sort simpler forms of life come like simple cells and other lower-level simple resources and then slowly what starts to happen is the chain sort of the pyramid of what needs what and what relies on what and what the dependencies are they slowly start to build up and what you get at the end and what this slide is talking about is how it's not so much about the complexity but about the resilience with this biodiverse system and how being and how they develop at the edge because of the diversity and the richness of their environment and what's happening and I think about things like this in the sense that you know I deal a lot with building out and thinking about fault tolerance and making things self-healing in large systems and there's a lot of strategies here that I think are applicable but you know difficult to solve and how to build it and the idea I'm trying to propose is we shouldn't have to keep facing these problems every project we start and every company we go to I just wonder if there's some overarching collaborative thing that we could all contribute to where we have this kind of substrate to our systems so self-organization is something really interesting that we don't really get into for more architectural reasons with distributed systems so we tend to assume that organization and order need to be imposed by some external force individuals can be acting and reacting to each other and complex organizations like schools of fish swarms of birds and colonies traffic managed to organize themselves into emergent patterns one second another kind of self-organization is a peer-to-peer that's something that we do deal with of course in computer science but this is peer-to-peer organization where there's no governing body outside of these Meza muskoxen they live in the Arctic but when they're in danger they create a circle among themselves they're not like you know talking but they're communicating to each other and they form a circle around the end the their young basically self-organization and emergent patterns with schooling's forming and hurting that you find naturally in larger groups of animals so fish in this sense consents and keep a distance from their nearest neighbor and if you've ever seen this they move incredibly fast and there are large groups of fish and how are they they're actually sensing and they're they have input biological feedback machines to receive and process eccentric sensory input and you know this just makes me think of you know I'm deploying node huge clusters with tons of nodes and in many data centers you know over tons of racks and wouldn't it be there's some concept there that I think that we need to think about being able to take this self awareness and self organization where it's we don't have to always rely on something outside there should be some base intelligence in there and the problem with this is of course in computer science we're always talking about separation of concerns where there should just be this one thing and also keeping things simple it should know one thing do one thing so it's sort of this you know like fight between how to keep something simple for operational reasons and reasoning about something but also trying to make things more intelligent impossibly a different way so emergence I just have this slide about looking at ant colonies which is a really interesting third they have a very limited set of instructions but they're able to self organize in situations where they're exhibiting complex structures and behavior far exceeding their intelligence and capability alone and it's a decentralized structure to self-organizing and all parts are contributing equally and when you think about economies of scale systems comprised of many actors making decisions in parallel and they share that information amongst each other but it's all happening you know at the same time and they're moving around and if each entity or each actor and the system is in I'm saying the other so when we think about cyclic patterns and resilience where does resilience is something I want to get into now there's something that I call the 3 R's replication regeneration and rebalance this is something that biological systems do natively but it's something that we have to think about and construct in our systems self-organizing patterns there's a lot of these in terms of movement and there's a lot of different type of movements and systems and a lot of that has to do or it all has to do with resources and what's affecting the systems that those organisms live in or those groups live in or nodes so for migration birds traveling in a pattern a bee pattern there's a this is really interesting the annual pattern of movement of Arctic Tern they actually go almost from pole to pole in every single year and they're these pretty small birds but they have the ability to navigate and it's the longest migration on earth and then there's daily patterns of movement and both of these are predictable and that's the other thing to think about is all of these movements all of these behaviors there's two emergent behavior and self-organization it's very difficult to predict but it more easy to program but some of these events and migrations they're much more easy to predict them I'll mention why in a second but the daily movement of for example articles through their territories might be anywhere from 40 to 100 miles per day and you know they're organizing themselves to move through their territories to find you know food resources basically and all this is dictated through seasons it's all you know you have these systems and outside of that system is a larger governing force it's it's not a governing force it's abiotic factors that influence the systems themselves and then outside of that because what's affecting you know what's actually causing the seasons is planetary orbit noxee el tilt and that's a system that's in another system in another system so it's pretty fascinating to think about all this but that's what we're really trying to do and that's actually what we're facing when we're trying to design with groups of people sitting down and discussing how are we going to build this new system what is going you know what's the scale of this going to be what is what are going to be the things that we have to handle what are what's going to be affecting this you know what are we going to have to survive all of these things I think are relative and applicable so for resiliency resilience systems and diversity there's a higher degree of diversity between elements whether it's more robust as a variety between elements making more than making them more effective at absorbing change so the more diverse a system is the more robust that system is and that kind of ties into the idea of the distributed systems or computer science we have roles different roles of behavior in our clusters and in ecosystems we have what we call niches and it's the same kind of thing what do these particular types do what's their function what's their role in the system I think I'm talking to you fast just like in my time so when we tried to pick out anything by itself we find it hitched to everything else in the universe that's what this next section is about everything affects everything that's around it and there's a theme about that relate to autonomy that I keep hearing in conference talks everyone's talking about autonomy and it's funny because I find that I can argue of something being autonomous and something that should be autonomous but then when you really think about that thing that's not nothing as autonomous I mean everything relies on something usually many things so it's kind of an interesting point trophic cascades are something that I think we see in distributed systems in our behaviors of our systems as well as very much so in biological systems and also in you know cosmology but it's talking about a process which starts at the top of the system or meta system hierarchy eventually affecting all the way down to the base and it's a matter and this is a question for you guys because I'm gonna ask what you think about it after is it is a matter of common experience that disorder will tend to increase if things are left to themselves no that's pretty true if you just think about physics from from from that particular approach or many other scientific approaches but and that's a quote by Stephen Hawking now I just wanted to show you a pretty classic case study it's about took place in Yellowknife like Yellowstone National Park they didn't set out for this to be a case study but it emerged as something that really exhibits some very interesting behaviors to look up so Yellowstone is a complex system and constant change a very complex system and we deal with complex systems so I wanted to use it as an example in these systems well in 1926 little history the last wolf in Yellowstone National Park in the United States which was eliminated and by 1994 the elk population which was the largest I guess undulate population herbivore that population grew out of control it grew to roughly 19,000 entities in that system in that system wolves are what we call the apex predator they're at the top of that food chain and what people didn't quite realize at the time was the effect in the system when you take something like this extremely regulatory role on all the entities out of that system how it just cascades through the entire system itself what actually happens so think about the systems that you're writing and you're deploying and and how we could possibly think about this and apply it so elimination of the Wolves caused a cascade of changes throughout the entire system ecosystem with no natural predator they all consumed most of their food resources not really talking but they didn't really understand and they didn't expect the extent to which that happened and what proceeded from there over several decades was large-scale destabilization of the system and again think about the slide where I was shown talking about diversity diversification and systems and the innate resilience that it brings so with the absence of the top predator the elk increased the food sources for all the other top mammals like bear which which grew to an endangered species level because of all this their food source slowly disappeared because the elk were eating it you know you can think about all these different roles and our systems that we deploy where so you know something taking over more resources than you wanted it to because there's something else missing and they can come in and utilize all of that the coyote population increased partially to fill an opening that existed where the wolves were and then what's really interesting I think is the tree and plant height and the numbers decrease dramatically and that affected sort of the smaller animals and different roles in this system and so after a certain number of years in 1995 they decided to introduce 14 great wolves from Canada and back into Yellowstone after being absent so 60 years the system just degraded and destabilized without this particular rule in the system so after after later on after the first fourteen they introduced 17 more wolves and by December 2001 their population had grown to 132 and what happened was really interesting in the system there was adaptation and predatory pressure behavior was exhibited so the because the predatory pressure from the Wolves keeps the elk on the move the elk before were using up all the resource and the particulars owns that they were in and so that there were no more in those areas and they continued to overuse and use up all of the resources and those particular systems that they were moving through and so with the reintroduction of them the elk actually adapted they had to readapt to learn that they couldn't just stick around in particular places and you know use up all of those resources they realized they had to move around to be detected less by the predator so it affected resource balance throughout this since because Yellowstone is actually a very complex system but it's surrounded by other systems with where you know all of these resources and all of all of these entities go back and forth through health and then what happened was large-scale regeneration in the system so the elk started to avoid parts of the park where they were more exposed to the Wolves the forest and the Aspen and the willow began began to grow back all of the diversity slowly started to creep back into the system and then eventually it started to get repopulated and and the whole biodiversity of the system began to come back many different unrelated animals that use up different resources they all started to come back and all of those affected all of the other ones and so all of a sudden you have this system that becomes slowly more complex but it's slowly becoming after they destabilized it it slowly becomes more and more resilient and you can think about it as becoming more complex but I don't think that it's the complexity that we think about in computer science I think it's a different kind of resilient resilience complexity where as something is able to be more like a self adapted and self-healing system's large mammal populations rebalance and this is necessary with all of the smaller subsystems the Yellowstone grizzly bear which were marked as endangered came back and everything started to rebalance erosion this is really neat the actual topology of the entire area the system itself and all the systems around it started to change the river shapes started to change water started to pull up like beaver came in just like completely changed what this entire system looked like and it made it incredibly resilient again on its own so one role can change the entire topology of a system and I think that that's a really powerful idea I can't really stand up here yet and describe you know how we can apply it but I think if we all kind of think about this remove my need over it and please contact me Twitter or you know walk up you know during the conference I think that this is a really powerful idea for us so self balancing balancing systems again Stephen Hawking quote it is a matter of common experience that disorder will tend to increase if things are left to themselves so given the case study that I just reviewed what do you guys think I mean this is something that they removed it they removed a top role and then on its own when they reintroduced it the system became more balanced it went from kind of disorder back to a natural order without anything except just reintroducing that rule so you have to ask was Stephen Hawking right which is a ridiculous question of course I just wanted to ask so innovation it's really hard I think for us to take ideas like this and and I mention this a little bit before but to step off our kind of assembly lines of timelines of when things are due tickets and project deliveries and something that you know other you know product teams need or you know things like that or things that have to be deployed to your customers so the assembly line versus those of us that are lucky enough to be in research situations it's a very different criteria that we're working under and so it's very difficult to sort of step off and be able to think of things in a larger sense differently there was a time though when companies would fund and sponsor a lot of their scientists internally or engineers and some of them have won Nobel prizes like back a long time ago Sony and IBM and Bell Labs and but I think we find this less and less research and development has become less are and more D and this is approved by professor Archie Aurora economics of technology and technical change and one last thing two last things I wanted to leave you with the rate of innovation I find it really interesting back to one of my first two slides where I talked about how we're really approaching a lot of these problems and given given the fact that we have a lot of constraints and there's a reason why but we're approaching a lot of these problems and purely a computer science approach and in that kind of mindset and I wonder if we could think into these larger scientific disciplines and apply them if that would help us with our not so much speed in a speed of innovation because I'm not trying to say that we're not innovative but just as a community as our particular area of the industry be could we speed that up possibly by taking a larger view of things and I bring that up only because when you look at some of the other scientific disciplines and their technologies and how rapidly they've been evolving and where they're they've gotten to it does make me wonder if there's something that we're missing if it's more important to sort of press our employers and our companies to say you know hey we need to stop and think about this and solve this problem like actually solve it and so building on that I just wanted to say be more experimental I've been in many situations over the years where because we can you know predict we can you know list off in a new project all the ways something might fail but if you spend too much time doing what's the word assumption planning then that's a lot of upfront time that you're not getting your stuff out there and not actually looking at its behavior in the wild and testing it in different ways so I think that that's a really important step to take to just get your out there see how it behaves and get some actual data huge discoveries are really the result of giant collaborations and this I love because it's from one of the 2017 Nobel laureates in physics and I think that that's one of the most wonderful things about our community for those of you that are new to the school community I think this is one of the most amazing communities I've been part of it for at least eight years and I really think in terms of that but also the larger open source communities it's so important that we still remain exchanging ideas and working together in turn trying to build things together and solving larger problems together and it is difficult because those of us that work at companies when we can't share things but we want to you know hopefully over time I think that that might be started actually I think that is starting to change so we might be seeing a resurgence cycling back to being able to do that as engineers and as a community so thank you [Applause]