reactive.community: @BigScala: Jonas Bonér, Life Beyond the Illusion of the Present

reactive.community: @BigScala: Jonas Bonér, Life Beyond the Illusion of the Present

Recording: reactive.community: @BigScala: Jonas Bonér, Life Beyond the Illusion of the Present

so many stayed after this long day it's been like a four day marathon how many have been here for all of the four days like half yeah I was here for the first two so it was excellent thanks Alex see okay so this talk is try to lie fill beyond the illusion of present and as John Archibald wheeler said like time is what presents it's what prevents everything from happening at once and this talk is really about the concept of time and sort of exploration of what time is and and and through our perception of reality in a way Israel is sort of divided into two different parts that the first part we we discussed the physics the cycle then this this this psychology and the philosophy of time and the present and in the second part is sort of the boundary might be a little bit fuzzy but in the second half I'm sure I'm gonna talk about how I think that we need to to completely change the way we we think about time and how we need to model it explicitly in systems we can't hand hand wave it any longer we've been doing that for way too long and if that harp has profound implications I believe and it was sort of been trying to fake it and now it's time to wake up so this is sort of a wake a wake up call and I will I will probably almost ask equally if not more answer questions than answers so hopefully there'll be some food for thought so I've spent the last 10 15 years buildings or large scale distributed systems enterprise systems and I really come to the conclusion that the time and so they're sort of the present the idea of the present it's really nothing but we're pretty really sort of stubborn illusion okay everything we see we hear we hear and we feel this was only saw reckless for from echoes from the past but this illusion has really influenced the way we think about and model the world view the world in so many different ways and I believe that sorry this simplified view have been serviced or inspired by Newton's physics I don't know if you're familiar with with with Newton's physics but in his sort of world his universe but for time is Martin is marching forward so you you uniformly and and linearly so accruing knowledge along the way in time is something that is absolute regardless of the perspective of the viewer whether it regardless where everywhere the viewer or the observer is everyone is experiencing the same now okay and I believe that this model is very appealing to us I mean we we we humans are naturally really bad at multitasking like we are bad a thinking cooker concurrently and it's so easier for us to think about time as something that is absolute okay these are maintains that feels like we get to maintain control of the present in some way making be in control of now which is Fila comforting in many in many ways I think and I think this also this model made perfect sense in computer science when we had single core processors this is John phenomena in and in nineteen 1945 he introduced the phenomena retexture I know how many year are familiar with that at least in there reading about about it at the Union at the University and so on and and the the fenoli machine had like a total ordering of instruction working working on a rail array of memory a word at a time sir linearly yeah and it's it assumes like a single process updating mutable state and that we were in full control of the present essentially like mimicking Newton's universe and back then like life was good but then unfortunately Along Came concurrency then and made life miserable again I mean I can I can I can I mean I've experienced this sir this this comic strip countless of time right that sort of Heisenberg's like race conditions and all these all these type of things and then then Jim Gray came and say came to save the days till we got transactions to make sense of concurrent processes like transactions gave us like the illusion or having ordering within transactions even though they are there they might run more than one but this order like a cross transactions and and you know ask them you got some acid and acids the isolation provides consistency here and this gave us sort of our linear time back like we'd like to brought us back into a comfort zone and we everything felt good again right we we actually couldn't could maintain control okay we were back in our in our comfort zone excellent so can we go home now unfortunately I mean or I'm saying fortunately we got more possibilities with that that then came like distribution distributed systems and also we made life miserable again because the problem is that transaction work very poorly in distributed system so they they're they're brittle they're not scalable they're not available there really hard to make available at least and we need to bend over backwards to to to really make it make it work and it's really I think there's really nothing to be supposed to be surprised about because it's really not how the world works you sort of we tried to like fake it in and then and and and make our life easier for us like but they lose illusion is like catching up on us like she was try to run away from this like this Wolf Pack by sort of chasing us behind and eventually caught up they catch up with us I think so the problem here is that there really is no absolute present at least not sort of a single globally consistent present we do not experience the same now the same time okay we it's all based on the perspective of the viewer I believe you know it's been it's been said that future is a function of the past the first reference to this was what was by this by a guy called Roberts and it was an Australian math teacher it and I think this were they what is tries to say is that if this future function does not change the past okay and I think it's it's it's older it's only it's on the right track but I would rather say that the local prescient is a merge function of multiple concurrent pasts I believe that is more accurate like we can we can let that sink in a little bit and if it helps I wrote it up this as scholar code here it's sort of it's sort of the local press Indonesian surveen can be more like assess like a left fold using emerge function overall the observed concurrent pasts this is sort of I think the world that we actually live in at least the best approximation or model that we can get so I think what we really need to do is then we need to explicitly model the local prescient as facts derived from merging of multiple concurrent pasts okay was mouthful but I'll try to like explain why and how in this in this talk and hopefully it will be be fairly clear why I think we need we need to do this so it's so taking a step back now information is really always from the past when we think about it it's really true for everything that we observe I mean when when you do observe something it's it's always although it already happened often actually quite some time ago so we're always for like looking into the past listening into the past right so it's it's it's in the eye of the beholder almost worn out quote but that's really what it is and the present is something that is relative so in a way I think the truth is actually closer to Einstein's universe Einstein's physics where everything is relative to the perspective of the viewer where time is something that is not absolute but actually marches forward like sort of a relatively based on on the perspective of the of the viewer and this is by the way Einstein together with Kurt with Kurt gödel is it is it's a book cover from it from a book a couple a couple a couple of years ago that I found really interesting it's called the world without time and it got me thinking a lot about these things you know you know gala was his logician and it was doing math most of his life but it delayed his later years when it when when he fled from the Nazis but across the half globe right here and ended up were living close to to Einstein in in Cambridge I think it was and they became close friends so the last the last years of his life he actually spent exploring the concept of time and and and and the relativity and actually actually proven the time especially just an illusion at least according to to the book Evans rather than his original papers but but really information is something that has latency we're not just always looking into the past it's actually it's actually it's a fact like that the information travels at the speed at the speed of light and this is as we as we know now I mean sometime years I'm some years ago well it was perhaps more sort of an esoteric fact that everyone knew right but today it's the fact that everyone that right system need need to take into account that you know you know the round-trip time to send data between data centers like I don't know Tokyo London or New York and so on it actually takes is like a noticeable that information has latency and and I think that sort of the cost of maintaining this this illusion can be defined as two things first contention there's one waiting for like we're not queuing up for shared resources okay and that the second one is coherence or the delay for data to become consistent and this does not just affecting our daily life just as well as computer science I think I think we all sort of experience this really when we talk to people and try to have any have data consistence and most people probably heard about arm dolls law that sort of effect contention has on parallel systems you know that contention my contention can give soared amid the diminishing returns that when you start adding more more more CPUs or more more computing power it doesn't scale linearly that's our the the the the dotted line here but what most people don't take into account is coherency coherency can actually give you negative results right we're adding more resources actually just adds the work that needs like maintaining cause consistency completely it eats up the benefit so so so so it's actually like like like this like sir the the solid the solid line there that you actually decrease performance and decrease scalability by adding more resources and this is sort of also I think pointing in the same direction we need to give up our grip on Newton's universe really and as late as it gets higher I think the solutions are just just cracks more and more and more and and you know the difference in a distributed system is a lot greater than honor in a multi-core system even though we can actually experiencing it there as well so I really really believe that now and when everything is distributed system so I mean there's really no way of escaping that it's ok if you write pacemakers or like embedded systems it might not be but if you write web apps for enterprise systems or most type of system today requires some sort of going across the network to the database or calling out to two services over rest or whatever or actually being available and you need to do replication and NFC and you know so I think it really it's it's nowadays it's all distributed system the way I view it and I sort of the the way I've learned to view it through through the lens of reactive systems as well is that it's it's it's either everything is really distribution it's either like distributions and so scaling out on a multiple cores or scaling out on multiple machines lucky either had the qpi link between your two processors and and that as agency or you have the network between the two processes right so if you have a model that actually takes this into account then then then you have a model that works equally good for concurrency for just interleaving for parallel systems as well as for distributed systems service or a sort of a sidetrack that I won't explore more here but it's it's something that I really I really believe in so I think this release tie puts the harsh and to there to the solution or we can just afford freezing the world any longer it's just it's just it's just too costly we need to give that up I believe and stop shoehorning everything back into Utah Newton's university because we're and also we we all know that the network is reliable right one of the fallacies of Peter or peter Deutsch not really I mean as soon as you add in the network things will fail in spectacular ways messages will be will get dropped use often as part of the of the protocol to do to do throttling and and and and back pressure and stuff like that or message will be real reorder duplicated and and and and and we need to have protocols thread really takes this into account so it's it's hard but I really think that we can't ignore it any longer so this is sort of a famous feel like philosophical quote I don't know if you if you if you've heard it if a tree falls in a forest and no one is around to hear it does it make a sound and and the question is really how I interpreted this interpret this anyways it's can something exists without being perceived Einstein wild one I mean you know he brought a lot of letters not emails but actually physical letters and one of the one of the guys that he was he was come having most conversations with what was his with what his friends what was his friend Niels Bohr a Danish physicist and and and in one of these letters Einstein asked Bohr do you believe that the moon does not exist if nobody's looking at it and Bohr answered or however you however hard you may try you can't prove that it does and I really I mean dismissed disaster realities are a metaphysical just philosophical question but I believe I actually believe that it's not actually I think is directly affects how we build computer systems today simply because information can and will get lost right and as I said before like network drops and splits really lead to draw packages failed nodes like malicious nodes if you take like B Byzantine fault tolerance into into account and now if this lost the information and it's actually facts about the past okay what does that mean that does it mean that certain sir pasts have has never happened if we can't observe them how do we know okay and then what does that mean for a present we know that is for the future is a function of the past what does that mean then how can we make sense of the of the present so we can we can we can perhaps get some sort of help through just thinking about how do how do we deal with information loss in real life right how do we do it every day we I think how what do we do when we just receive partial information for example I mean I don't know I mean it was out in the bar yesterday and I talked it wasn't like some some noisy bar or absolutely and we don't really hear each other that that well we usually serve wait for more information and and or we ask again and and we might receive the same or additional information that we give so we can make sense of what was said we do not aim for guaranteed delivery if you would take it with like you're born out computer science term instead we we naturally use for at least once deliver delivery with idempotency really that's how we all behave right that's how it all works when we communicate with each other and at an early age we learn to feel like to take educated guesses and if we were wrong we fill in the blanks no I'm sorry educated guesses to partial information and fill in the blanks okay what we think someone said and and we try to make make sense of things and when we're wrong if we which we often are take compensating actions for that this is very natural to us so we to sum up we take decisions always on partial information sometimes applicate information and we take compensating action sea of things if we if things didn't made it made it made any sense right and this is what Pat Helen calls our apology oriented programming that usually take bets and if you're wrong no big deal I mean you just apologize and continue and the and if the protocol is written in a way that that is okay then then life moves on right even if we were if we were wrong and if we think about it this is how a lot of systems work how many have had to have experience like overbooked flights for example when you build like and when you even you know you have you have your ticket you didn't get to go on the plane right because we airline's do this all the time they over booked the flight and then they sort of asked for forgiveness they try to bribe us like giving us vouchers and stuff that since then that usually doesn't have much much value unless you fly with that airline again and as also how ATMs work I mean if if the ATM actually cannot reach the serve really will still handle money and then it will it will it will compensate by by by charging you later so the bottom line but the the bottom line is that I I think we just simply cannot force the word and to single the globally consistent present and along again maintaining these illusion that we're so used to do but it just proves too costly performance will suffer scalability will suffer reliability will suffer and correctness will suffer okay so the question then should we just give up right sometimes we might just feel like doing that just ain't exit the meeting and jump out the window but I really believe that there is a path forward okay and what we need to do I believe is we need to treat time as a first-class construct we just can't hand way time any longer it needs to be part of our programming model in the way we moral softer part of the programming languages ideally and at least built into the to the standard libraries and data structures and so on so but what is time really then if we should dive into that topic I believe that one way of looking at of course you might disagree but but I believe that it's actually not about wall clock time you know tick-tick-tick hours minutes seconds that's sort of a half leaky abstractions for us to make sense of the world but I really don't think that's the essence of time it's not at least not what makes us feel time and everyone has has experience like when why some sometimes times time just flies through the window and sometimes it just feels dog slow and we can't understand how that how we can be like that right and I think I really I really believe that when you think about it it's usually related to the events experience during this period of time so I really believe the time is all about causalities or the events that happened during this time period so the the succession sort of costly related events I really believe that is the essence of time since we all know that time is definitely not absolute if you send out an atomic clock going around the world a few times and bring it back then I mean yeah they're different different time have passed past was actually provable even so the question then is then how can we manage time the first thing I think we need to do isn't it we need to work and think and design and model the world as facts and and sort of model how facts are causally related to each other okay so what is a fact according to merriam-webster is a fact is something that the truly exists were happens something that has actual exists into true piece of information know what it seemed sorry it represents something that has happened and was interested and what's important understanding that facts just accrue right facts do not change they just accrue you can just add more knowledge in fact sorry immutable if something's happen it have but it's nothing to do about it I mean we can be feel sorry for that sometimes but in way she didn't but but life moves on right we need to need to take compensating actions usually and fix what was wrong but it doesn't change the actual fact so facts accrue but they and they can accrue by either you adding new facts to the system or you can derive new facts from existing facts and facts can often mean mold like alligator using dependency graphs like we have like knowledge derived knowledge based on knowledge like and and and and there are the data structures that allows you to do that like data flow graphs for example you can futures or the data flow variables it's one it's one way of doing that for example this is also how in a way how persistent data structures work so closures were popularized if you can fill fill bag will brought to Scala and and it's also for some huggy hug it works it's a nice way of modeling the world with immutable dependency graphs I believe but but in order to make sense of this at all we need to stand on saw an on solid ground I think immutability is a requirement is it really the only way we can make sense of the world and unfortunately the traditional orbit oriented languages don't help us much with this they treat value and identity as the same as the same thing and and and and I think mutable state sort of see something that doesn't mean it doesn't sort of make much sense as a general programming model that we should use everywhere I think it's something that we invented us to make life easier in a way right easier in quotes like at least when we when we like or in Newton's universe we're like then then it might actually work but but but we all know that that as soon as you write around multiple threads or starts passing this around the network or whatever things start to fall apart I think we really need to bear the base our understanding and modeling on stable values code that doesn't lie and all these things so think we the question is then do variables have in life at all john backus he said he said in history or during a world lecture in 1977 that the assignment statement is the phone Norman bottleneck of programming languages and keeps us thinking a word at a time in a word at a time terms much the same way as a computer bottleneck does a word at a time I mean the assignment operator now is trying to shoehorn or ourselves back into Newton's unit universe again that's really what is what what is trying to do and this implies to a global music global mutable state and this is why we always need to go through like adding critical sections everywhere if you use like you know regular Java concurrency or C++ concurrency of course there are better ways and legrasse trusts ownership system is extremely interesting for example but if you look at like traditional ways of doing it so so what do I think I mean is mutable state some only like evilness I I don't I don't think that is the case actually I just I think it has its place is actually the easier to work and to reason about in in a box so I just these I think we need to make sure that it's contained if if if you it's perfect like for local calm calm computations then it's totally safe but as soon as we have sort of worked out some computation then we produce a fact that is immutable and we publish that out to the rest of the world to to watch and then if then we can touch it and we can touch it but that we can't change it any longer so so that I think that's really the way we need to work with with mutable state so how do we store it or in organize facts yes we I mean a lot of people as I said joined the skullet or the to Scala dates a lot of talks I'm sure you talked about function function a functional program the fractal program is like based on lambda calculus and all these things I'm sure you know all this I think functional program is a great tool for for working with values in facts this sort of a fact machine right you put facts in and you get facts out and as long as you stay true to that and normally work with with with immutable values mutable it can actually be done safely insert them at least context languages within the function right as soon as you probably you get the result out you need to make true true I treat it as an immutable variable logic programming I believe is also even though it is the DES net unfortunately not being that that widely adopted I really love logic programming I think is the ultimate tool for managing facts actually you only only only define the properties working on facts and then you then you let that so they runtime just take care of how like we're solving the different rules and and within within the constraints that you have defined essentially and you feed it with facts and outcomes facts data for programming error we'll already talked a little bit about I think that's all that's also an interesting model but the first rule of facts is that we should can never ever delete the fact I've already touched about that space really worth repeating is it's over sometimes very very tempting to start deleting facts because I really believe by deleting fact we delete our past anyway and we already said that the past is our only way to the present so it doesn't sound like a really wise thing to go around and deleting the past I believe and and I think this also so this means that I really think there's no reason to use in place update any longer like like like we used to we are used to do using looking the regular relational databases it can't really be convenient sometimes but I really think it's wrong and Jim Gray said in his in his like seminal paper that the transaction concept from eighty one worries reintroduced transactions that I'm going to read and I'm gonna read the quote here when bookkeeping was done with clay tablets or paper and ink accountants developed some clear rules about good accounting practices one never alters the books if an error is made is in its in its annotated and a new compensating entry is made in the books the books are thus a complete history of transactions of the business update in play strikes many system designers as a cardinal sin it violates traditional accounting practices that have been absurd for hundreds of years still I mean this is what we at least I am guilty of been doing for years right because it's convenient but it's just wrong I really believe that crowd is dead we don't need update and delete we can just use create think we can create new facts and append them to the history and we can read at any point so see and are affine but but but but there's really no reason to update meaning in place and delete facts any longer because because I mean one of the reasons why you had to do update in place I mean if you should go back in history they would these guys were not stupid right they had they had they hadn't they had a reason was because memory constraints right so so often do that was the only way you can actually right working software so but I think today still holding on to this view of the world makes no sense because disk space is so incredibly cheap often you can actually even suck in the whole database in memory and just use it there this is also great quote from from from Pat hell and the counters don't use erasers paper that when he said the database is a cache of the subset of the log and and he talks about seek seal of a sequel databases the question again said why work with the cache of a subset when we can work with the real thing often databases you know the you do use transaction logging under the hood append only every trance all transactions are store they are ordered but they're but what is exposed to us as developers is usually only an update in place model and if we want to if we want something else we need to create extra tables to restore historical data denormalizing these type of things so I really think that one of the best ways we can manage facts and manage sir make sense of the world really cysts or facts Ino order in an event log an event flow can be done like in a sequel database or we can use this for something that's more specialized for it that doesn't matter this is all about luck idea some patterns than the way to think about things more than looking exactly how things should be implemented in a way like the log is really just database is really the database of the past right it's not just the database of the present like most sequel databases are being used today today like the snapshot now but actually everything so log allows us to actually go on a journey through time you know the log ID can be can be seen so serve as a concept of time roulette ticking forward like tick by tick by tick by tick so it's really the causally related succession of events that's right events in the event log and if you remember that is actually the way we define time earlier in this presentation the causal related succession of a man sir I really believe that they're not really nice way of viewing the the survey the the world of the past and and and the present and time the essence of time you sexually gist or events in and and also I mean since all histories kept around all the time it allows arbitrary time travel and we can even replay the history and and and and and find out what went wrong or if we do for auditing purposes for debugging for replication like replayed on another machine like bringing the state up up to just being locked in for bringing an object up to speed and so on another interesting question then is and can we rewrite the past you know the sort of in the past is always sort of biased in subjection like souls the winners to get to write the history and the history books actually do get rewritten sometimes based on new information but but in any in computer science we we can rewrite you know we can we in the database and tamper with that but that does does that mean that we should sometimes I mean there are exceptions I guess but as a general rule of should not but rewriting the past means I feel the change in the present because we know that the the past is they are all the path to the present sometimes we actually might want to do that but it it often it doesn't make much sense actually like just like like what kind of mess they did when it went when they went back and tried to fix things in this in this in this movie I just watch it with my son and if actually did enjoy it even though it's like from the early eighties so the so I will also believe that we need to shift our focus like for thinking about data at rest to to data in motion okay and and because it's in its when data moves that I think that's most of the interesting things are happen and I also think that sir that allows and also forces us to model the world as it is really not like static data not like snapshot of now but actually with with data that flows and that moves and this sometimes we we need to drink from from the firehose and you know you know in streaming is a very hot topic nowadays and and I'm sure it's been covered in the last two days like a big data you know we have things like spark streaming flink and in the land architecture and and distributed stream processing frameworks like you know storm Sam's I asked for war and stuff like that and we have the react streams in initiative that typesafe have been part of doing specification for together with Netflix and in pivotal Red Hat and and and some other implementer so that is like yeah you know Oryx job or ARCA streams and so on and and I think all these that's this is all good right because these all these framers they force us to think about time they forced us to think about a model causality and and they also force us to design systems that are loosely coupled and I also think that I'm going to talk about both later they also force us to think about different consistency models because we can be fully consistent within the log there were there we have strong consistency we have like single single writer right depend only but between the different logs because you will most likely have different logs you have eventual consistency and this is like two very powerful tools that we should at our disposal that we should use like and and makes me make sure that we that we make sense of I think it's great to allow time actually to diverge into difference or event logs into different streams and evolve independently right but at some point we need often to like to sync back and and agree sort of on on at least temporary shared view of the world okay and then that is when we need consistency oops so am i boring you by the way or okay I have a lot to cover but good let's continue so what is consistency then and and why and when and all these things I think maintaining consistency means that we need to employ some sort of coordination okay and and and the the problem is that too much coordination can violate and sorry too little coordination can violate consistence correctness right and too much coordination can really make things dogs slower even brittle too strongly coupled the coordination is really really expensive and it's the hard to get right especially in in a distributed system it's very costly okay so what I what I think this needs for us like in in practice that the we need we need to start when we design a system to define our unit of consistency in I mean what is what is there like for a minimum unit that that can be strong well that it can be strongly consistent I mean II you know so the other extreme is the way we used to write systems using single sequel database then our consistency boundary you know is it's the whole universe but that is very hard to scale that's very little is hard to make performant and so on so I think what we need to do is like try to minimize our unit of consistency and think about decomposing the system into consistency boundaries and and what's important to think that to know that this consistency within this consists in the battery we can be full transaction we can be strongly consistent everything is like we're back in Newton's universe for this little island so to speak this and everything here now has to be moved as a whole be replicated as a whole because as soon as we start partition that it's almost impossible to keep things consistent so the question then is then how do we up in optimal way divide up our system in this consistency boundaries Patel and once again is one of my heroes he's been he defines a nice framework for it like thinking about consistency where he talks about inside data that's our that's our current local present okay our outside data that's what I called the blast from the past you know information is always from the past and then he he has when he calls that between services between these consistency boundaries that I talked about he calls that hope for the future which I think's almost it's almost poetic so I think that's a nice model and framework to have and and I think I mean bringing some some sort of some sort rendi practical examples like micro services for example I think makes good consistency boundaries it's a pretty terrible name actually microservice there's nothing about size at all I think is for me it's like it's about three things like single responsibility principle good protocol design that's full of decoupled well the asynchronous or communication in between and that they are fully isolated okay share nothing you really have these these fully isolated through islands of consistency so within you have the present between you have either the past or the future okay that's a nice way of thinking about it I believe and and I don't know if there were any DDD heads here but but but but aggregate roots is also can be also a nice way of modeling this thing aggregate this is a group of domain objects that can be looked as a single unit we have where they have to add the area to aggregate fruit syrup sort of guarding that group nobody n touches my mic my sort of mic my good my crew or my my true pair without going through me essentially so he can they ensuring the integrity of the audio of this component to pool or a group as a whole and so within the consistency boundary I think I think we can so this is Robbie so within the consistency boundary I really I really believe that this or the log is a great way of modeling time and modeling the history I'm event sourcing is he's a great tool it's also sort of trendy I don't know if there were any talks about that in these in these two talks succeed see QRS is an extension to events or saying but but but event sourcing is realistic I see I see I think I believe it's a great abstraction learnt on - on top of the log where the word where everything is sort of all state changing events go into the log essentially and then some of them separate between the commands and events as well events are state changing you know sort of commands for state changing well advance or secular the only represents a fact that something has happened and there's also a really nice way to model to model the world with e within the consistency boundary but between the consistency boundaries it's still a zoo it's still really hard right but it is actually the zoo I think that gives us availability and that gives the scalability there when we start deep decomposing the system into these say violence we actually give us a lot of headroom for scalability and for availability which is great but it's a lot harder but often we're forced into that so I think to model this sue we really need two systems that are that I said there are decomposed they are decoupled and there are two ways that we need to think about decoupling the first one is decoupling in time this means that this means like adding concurrency you know asynchronous message passing concurrency can actually be I mean sorry decoupling in time can mean concurrency we can just meet interleaving running in order single on a single thread but I think it's still a useful abstraction because it opens up fork for for decoupling in space we actually run in entities in different universes right in different in different contexts and and and here a lot of sort of the reactive principles leads the way you know you know we have location transparency is a great tool and there are there are nice tools that I that I advocate for like I had the actual model for example that held that really gives you by by then through the computing model itself then model of computation itself gives you decoupling in time and space because it's a model for distributed systems so but I also think so a lot of people ask them ok what but what about strong strong consistency between consistency boundaries I mean again like it's often tempted like pulling transactions and like across across at least cores right but also sometimes actually unfortunately across the network using XA and stuff like that don't we need that I think sometimes we actually do need it right and then there are good tools for that like Paxos and raft and some of this consistency strong consistency algorithms and sometimes we should reach for them but it's the wrong default this is so important that we should not start with that this we might end up here sometimes the way too often just Auto have it and because it feels it's in it's in our comfort zone we reach for that immediately and then we think yeah we can always like try to decompose later and we can always mean you know you know yeah reactive sounds good right and all this principles there and but we don't need it right now right we have we're under pressure time pressure or whatever and sure it's possible to refactor a system decomposing it after the right but it's really hard and it's not actually easier to do it right from the beginning I believe you know you know strong consistency means that we often have to sacrifice things like we said we have to sacrifice availa availability for example you know see the sea in the cap theorem and and also sacrifice performance and throughput in latency and and here if I'm a geek out a little bit if I haven't geeked out enough right here we sort of we're living in sort of the looming shadow of impossibility theorems right perhaps not as frightening at this you know the vampire or this or the count or lock you're in an austere ferrata but but but I think I put it the I put up this a mission because you should be scared right this is a scary guy even I don't know how if you look at that scary but anyway because we know again like the network is unreliable all right so it means we have sake disconnects temporary networked or partial failures and there's no way you're actually knowing if it even if the service you're communicating with this it's actually down it will never ever come up again or if it's just like if there's a temporary network drop or if it's just being under heavy load doing garbage collection if it's all running on the on Java and will come back just just a second later we need to take educated guesses about this and and and and what what makes it hard is that as officially inch Patterson surficial inspired Patterson and that's Nancy Lynch proved in the paper the impossibility of the distributed consensus with one faulty process is that consensus is actually impossible we shouldn't even try to strive for that you know in because it's actually proven to not to be impossible and and and and and and and the cap theorem is another impossibility theorem that actually proves us proves that consistency is impossible if you have a network you know the cap theorem say like pick two out of these three like see staff consistency a for availability P for partition tolerance if you're on a distributed system then you have P so then you have the choice regardless if you wanted or not between si and a you can't have both consistency and availability it's impossible so you just have to embrace the fact that if you want an available system you need to decompose it and live with the fact that it can never be consistent there will be eventual consistency in between which actually as I said it would think about things that way can be good thing it can actually gives us Headroom for scale making systems actually more reliable which is which i think is good dissecting character I'm gonna skip that now so the the the the first principle of successful scalability is to batter the consistency main mechanisms down to minimum James Hamilton said and I really believe that we should embrace eventual consistency as I've already touching up the guarantees gives us more Headroom for a scale in all these things right and and and I think it once again keep track of time is actually keep tracking of core of causality and and you know some systems rely on times some time stamps for that I really think that in general synchronize clocks and time stamps is a really bad way of managing time and and and and and even though some of the most well known databases like in or sequel databases actually relies on time stamps it leads to lost data in corn ink in corner cases it's really not a good way of managing time because it's really hard to keep clocks and sync you will always have have clock skew instead I think we should rely on logical time okay and we will and that that's no news right Leslie Lamport invented lamp or clocks in 1978 right Lampre clocks gives us a global causal ordering between between events okay it's really it's really beautiful in its in its simplicity when when a process does work you just increment its counter I mean that you search attach when a process sends the message include the counter and when the message is received on the other side you take the counter and you merge it with with with what you have so you have a sort of the max local and receive and then you increment that essentially and then you get a global ordering of of events in the system that's often not sufficient though because because because it's - it imposes - too too too much guarantees essentially so so so a calling feature in invented lattice vector clocks in in 88 gives us a partial causal ordering between events and what this gives us is a tool to detect conflicts in systems which history might diverge so that I think that's that's that that is a great tool and logical time what that does it gives us coastal consistency and I think that that is actually what we should reach for because causal consistency is as good as we can we can make it that's also that's already proven I can I can I can read I can I can I can send you the page the papers later it's called the kak paper where they actually prove the Colton consistency is the best we can do and still be available in terms of consistency but it's also very intuitive coastal consistencies like it's what we humans usually sort of think when we think about the world it gives us all these things like read your rights and and and and when I've seen something I mean history is monotonically increasing and things like and things like that so the question is then yeah I can skip this just gonna talk about that causality is actually also expensive there are no silver bullets since when you track a lot of meta meta a lot of metadata but so it's actually very useful but it's not a golden hammer once again I don't believe in these and these things but but but but one thing that I think can help when we when we do design protocols you see is to read for for what it was sort of the popular recalls like a see 200 I think it was part Ellen that defined this this as well in what in one of his papers we all know like acid wandered oh like traditional asset like stands for like atomic insistence isolated and durable and the problem again that's not scalable not available it's brittle but but but as it - as it 2.0 so didn't defiance is that's being associative a stands for associative that means like like like grouping batching does not matter C stands for commutative instead like this ordering does not matter yeah and and and and i stands for idempotent means sort of that that application does not matter and and i think this this and d for distributed i guess to me just to make it a nice acronym did i think this is a really good guidance when we should define protocols and how we should define could the communication between these these consistency boundaries between these islands of strong consistency and there are there is actually some very interesting research going on just about this like I don't know many I mean here I've heard about conflict-free replicated data types see our duties yeah we we use them we use them in in in our code we have we have the Menaka in the archive distributed data module that we use them also in some of our commercial products is it's actually a very nice not nice abstraction to for for for managing sort of causality and and make you sense of data in a fully consistent way still eventually consistent we mean you know that most sequel database or no sequel database at least that came out worky worky worky value stores and key value source they essentially give you one data type that's the register you know key value key Vout a key value and that's and then they came with all of these nice promises you know available and then super scalable you know and and so on but the problem is that we as developers were used to think about in richer data structures than just the register we used to think about things like we have counters we have maps and we have sets and even graphs see our duties gives us that they allow him to model sort of rich data structures but with the same properties that's like you have like in dynamodb for example or or or react and these like we are essentially adding adding adding cser duties so there there's a lot of work going on that help it gives us good tool contemporary research but actually actually that it's very very practical and I just want to say before I conclude that's also another sir very interesting research brass that's all the device that's also that's actually done in in in Berkeley right up here is what they call disorderly programming and the kalam theorem Cal stands for consistency as logical and monotonicity and and you know monotonicity is that like facts just accrue knowledge only grows and this is like distributed computing model that is based on the fact that if you take logic programming and and and and and like monotonic increasing state that facts only only accrue then you get a programming ball that is fully coordination free and that you can that is fully eventually consistent and that you can scale without limits really and what bloom does the language that implements this is actually give you this safe box that you can work in and when you have when you accidentally exit this box when you write things like like code that is not monotonic for example when you need coordination it will you will get a compilation error and there you know okay here I need to use some sort of coordination mechanism press reach out for zoo keeper or at CD or or use the sequel database or something like that like for handing off data and I'm just I'm just bringing in did this up because I think this is very interesting research with a language that actually treats time as first-class I think I would like to see more of that I like to get this into skull I got like to get more research to think about this in these initially modern real languages because I really believe that's the only way we can we can as an industry that I may really move forward with it when it comes to that do you we we just need to give up this delusion right so you need it so you need I think so in a way we've been we've been forgetting quite far I think and there is a lot of interesting research but we're just getting started as I said and I hope this was like sort of food for thought I want inspire people to think more about these things think about breaking free from their phone ointments or of baggage in a way right and and and and and break free about the idea that everything has to be consistent all the time and reach for good tools to make sense of the world anyway we have a long road ahead of us as I said I think it looks quite foggy out there actually you know it might be worrisome but one thing is clear I think again the time I think really needs to be first-class so let's go exploring thanks for listening huh okay yeah some questions yep yeah sure I mean I don't know if I need to repeat the question and essentially what what is reactive systems in and then what are good resources to learn more about reactive systems yeah first I encourage you all to go read the reactive manifesto a lot and a lot of people don't like my manifestos but it was it was it was written as a matter manifest essentially just to provoke people to actually have them go read it and and hopefully actually disagree sometimes so we get a good discussion going it was the way to like one of the reasons why we have why we added the manifest in the first place well most not to just start wars right and have people fight for things but actually to unify different groups there has been sort of the need to to write systems in the way that the reactive systems in the reactive manifested defines for quite some time and throughout many different sort of communities you know the the dotnet community the Java community Scala community and JavaScript community and so on also realize that we need to change the way we work we write systems we need we need systems that are that are first like scalable or elastic right that I can that can actually make make use of all this all the nice new hardware that we're getting like running the cloud and you know the cloud platforms actually support elasticity just pay-as-you-go so to speak both scale up and down like super to be cost efficient so we don't cease it with more hardware or resources than we really have to but the problem is that our software usually not up for the job so we need we need to rethink our system in how we write systems there we also need a system that are resilient that is the second trait or building reactive systems and this is something that we have hand weighed way too long it's something that is very bold you often look just bolt on afterwards actually sometimes they after we gone into production let me start having the for the first failure is everything okay are crap we need to think about resilience with we think we can buy ourselves out of it riot slapped me some cash on top I don't know we're turning a WebLogic server clustering that actually never works that way without that well or or something like that that's just fundamentally wrong we need to Serg a sign for resilience is really bad by design and I did it that was all walk my talk my keynote for the school about buy-in by the bay was all about I really believe that I mean failure is nothing exceptional I mean calling exceptions exceptions is just wrong I think it's all natural failure half especially if you run a distributed systems that I already said if the synaptic failure will happen it's better just have a good way of embracing it to manage it and this is also what reactive system is all about like making it first-class citizen in the programming ball and natural state in the life cycle of the application and I really I really believe that that that sort of none of these two is interesting or even even important unless you have a system that's responsive that always can respond in a timely fashion both under heavy load when you try to scale as well as when things go will go and go down so that is the third trait of reactive systems building truly responsive systems and and and and and sort of the one of the core underpinnings it's been debated if it should be it straight sort of a trait or as it is now or like one of the pillars you know reactive systems or if it should just be be mentioned that's one way of implementing in my view sort of the best way in perhaps only the only way and that is to build systems that are message driven because you need this asynchronous boundary between components in order to truly scale them to be able to have the runtime actually to to put or Bicks out on multiple machines to scale them out as well you need a sinkers banner that message driven systems give you to be truly for fault tolerant because idea that allows you to actually have systems or your engine that your components fail in isolation without leading to cascading failures and and and and if your components can fail in isolation and have a way of treating failing the rest just a message they can actually notify the world that oh I went I'm going I'm failing here and someone else can then pick that up outside the context of the thing that failed and and act and I really believe that you need is a synchronous boundary between new components and and how fine we'll go coarse-grained they are that's of course up to up to debate but I believe in really fine grain error handling so these are the four core traits like resilience elasticity message driven and this will give you a nice sort of possibility to write really responsive systems and as I said earlier you should start going to read in the reactor manifesto but that's just a very basic walkthrough essentially what I said there are some great articles I don't have anyone in my in my head now but but but but I don't know if you do we have a web site the reactive used to do it we can press post some of the some of them of them of the material there there's also some nice books just on just on the way my friend Roland Kuhn is actually on a sabbatical now he's the tech leader vodka he and he's he's he's on sabbatical now trying to finish a great book on on a reactive design patterns that I'm really looking forward to and there are a whole bunch of other books as well so I don't know if that summed it up yeah I don't know I don't know if there's like a little lit litmus test for four four four react for reacting the question was how can we know if the system is really active or not but IIIi believe if your system is it's really resilient and can scale with and scale up and down with load I believe you probably are on the right track and if you probably I mean doing it all right the the I think the best way is actually just to like start shooting your nose down essentially like I pulled the plug see we'll see we'll see what happens we need to realize really test the crap out of your system that's really the only way in both like adding low like crazy and see and emulate load and see actually scale so the way the way it's supposed to do as well as as I said just pull this pull the plug in and and and and there are there are great tools for this like the Netflix image seaman army for example or the case monkey they and ARCA has the law also knives multi JVM multi node test kit that allows you to put through a test conductor in between the surgeon can fake network drop scandal a messy disc and things like this essentially going through gremlin seam to the system that runs around and do all crazy stuff that's also very very useful but but but it's it's in I think it needs to be said that building reactive system is really about going back to first principles is a way of thinking about how to design software and how to in an art architect how to think about the world essentially it's almost if you feel philosophy and it's nothing new it's it's it's essentially how a lot of people that have done the right have done it's in thus in the seventies and but I think now is the time where we just can't as an industry can't you can't ignore all these all this wisdom and knowledge that a lot of people like Jim Gray and Pat Helen for example you know you know the the the or apply to all this principles in the 70s when they wrote anthem systems so I really believe that they said that it's completely agnostic to to either men are cowards Carla or or Java or whatever you can build it in almost any language okay you mean selectively going and inner and then remove facts essentially yeah yeah I guess I guess when when you have when you have to do you have to do it and and and I mean there of course they look tons of exceptions to everything where I've said I'm just trying to give some guiding principles so I absolutely I but I believe that there is absolutely a place for deleting the history I mean I often wish I could do that actually I mean waking we wake up in the morning sometimes that's not that bad but you know what I'm saying but sure so I mean take what everything I said with with the grain of salt and and but I think I still think doing it just because it feels nice right and out of habit and because because we don't we don't write the one the right system in in in a sane correct way right that's I think that that's wrong okay yeah thanks for that input that's yeah the quit the question was if I think C or D T says has a future and I think is the if it's the way we should if we should make make make use of them essentially and I absolutely believe that I hope it's not the mister the in the end right I think is just a great starting point and they're still great this already great researched extending beyond CRCC are deities for example combining CRD teas with with with dataflow concurrency and and even in a distributed setting like I mean bringing serenity into daily motion which I think I mean Chris McClure John has in has I has started writing some very interesting papers on his latest research on that I think that if you want to Google for it but I really believe this yard adhesive or they work right I mean they depends on the implementation of course but but we we use them in production within types a for our own infrastructure for example as well as as the conductor operational management tool for reactive systems that we build all using CRC our duties under hood and they and they work they work excellent it's a great way to model state in an eventual consistent fashion that you know will always converge eventually so I encourage everyone to look at it the reaction you know Bosch shows databases also I have also added the CR duties that's an extension to their to their key value store the you know they have their links that as far as you could go earlier but now you have Co duties as well unless you model a lot of nice problems and one one thing that it's worth also mentioning when I talk about certainties is that one core property of securities as they compose so you can take one security and have it composed with others you get like a map of map of Seth's or whatever map of the cows or map of counters or whatever they do compose and they are still I've proven to have the same properties and so they are really that gives us a lot of power as developers to model things much more than just key value stores like we a lot of people are used to yeah that's a great question the question was about what I think about spanner you know spanner easy googles like almighty database right spends a lot of data centers and and and and one of the key key traits or key key contributions independent in the paper is it's about it's like it's a time API called true time and and what they do is that they have atomic clocks circling the globe you know that are a service so that they use to have multiple sort of implementational paxos being being in sync but what is interesting so you can say that if that since that obviously worked that might have sort of a counter example to what I said it actually we can be actually be strongly consistent but what it's important if to understand when you when you read the paper they actually define in the true time API time as arranged so they what they do is they model it with with with clock skew as part of the programming model so I think I think I think it's really amazing work and they approved that you can do quite a lot but I also think that it's important that that part of the API they really acknowledge that that even with atomic clocks having fully sort of consistent time is not it's not an option and and most people writing systems don't afford having atomic clocks and running circling the globe yeah okay yeah the comment was by the cockroach DB supports implements the spanner model okay yeah I I'm I'm I have not had time to look into that I've just read briefly about it so I can't comment on that but but as I said I mean this is this is just the way my way of you in the world and I believe that when we need to regardless of how we think about time when you start thinking about time or either if it's like through spanners approach or through about logging so hope I how you you have some food for thought thank you [Applause]