Scale By The Bay 2021 : Konrad Malawski, Distributed Systems with Swift

Scale By The Bay 2021 : Konrad Malawski, Distributed Systems with Swift

Recording: Scale By The Bay 2021 : Konrad Malawski, Distributed Systems with Swift

if you have any questions uh there will be a link in the chat uh where you can ask uh questions on our swift forums afterwards film so please do that okay so with that let's dive in with distributed systems in swift so a quick intro just in case you haven't seen swift all that much i know it's not as popular on the server as it perhaps should be so it's a native language it kind of originated from you know on the devices that apple develops but it's very much a multi-platform language so you can use it on linux uh it's fully supported there and we're still working on windows this is like semi almost here uh and the cool thing about swift is that you get this high performance low level runtime you know native code direct calling into c all that good stuff but also the language is a very very high level you know what you would expect from a modern functional language so you got kind of scatter-like power in the language but with very very little overheads we have some cool things like copy and write data structures so get the benefits of what you would normally have to do you know copy data all the time but we do it much more efficiently we have value types in the language and since recently actors in concurrency which is a part of the talk today so as we dive in uh i always think about distributed systems as being two things really they're hard and they're everywhere and that's a big problem if you combine the two because more and more people have to write distributed systems but it's really hard to get into and keep them you know running and you know scalable and understandable which is a big thing i care about personally so there's a few things we identified that are kind of problems we can address and the primary one is in swift we care about something we call progressive disclosure so it means that as you learn the language as you you know build your project verse maybe more and more things you have to take care of but it kind of comes in phases you don't have to immediately know about distributed systems if you're just writing a hello world app right and this learning curve should be kind of linear there shouldn't be like a huge gap between you know going from concurrency to distribution and suddenly you have to throw out everything you ever built and rebuild everything so we want to address that a little bit another issue is that obviously going distributed was way more uh failure scenarios to think about obviously versus networking involved so there could be both network delays message drops all kinds of issues and as you are kind of coming from you know not thinking distributed all the time it can be difficult to cross that chasm and maybe you'll forget one or two trading scenarios and suddenly you're you know woken up with issues in production not fun obviously people know about that so they store these things but very often it ends up as a lot of boilerplate a lot of serialization networking code kind of swallowing up and entangling your business logic of course you can say well you should have separated out the layers and many many teams do but that again ends up with very much like sometimes ad-hoc solutions we've all seen that like a small startup suddenly has to grow they only know mysql they suddenly use you know mysql as a queue or you know random ad hoc solutions which work good enough but they're not really the end game and then you have to do a big rewrite right so this is again this there's no linear progression never have to reason about test and debug because the tools you use to build them are so different from what you're using in local programming so we want to simplify all of that we want to simplify distributed systems programming and push the state-of-the-art a little bit as we do that so we kind of did that pushing the state-of-the-art with our actual runtime recently which we introduced in swift five to five that's the current stable release and as part of system currency efforts we introduced actors to the language so actors you've probably have seen uh or heard about at least a little bit versus the known implementations like obviously erlang the otp you know orleans or aka the big difference from a lot of other implementations that you may have seen around is that actors are part of the language in swift so you really can't declare actor greeter the same way you would declare a struct a class an enum and that has specific implications on the type system and runtime so this is how you declare an actor an actor's purpose in life is to isolate state and how it does that is free synchronous messaging this messaging we express as functions on these actors and they basically become asynchronous tasks that the actor can execute so uh swift catches isolation issues so we always talk about actual isolation when we speak about that and here's a simple isolation violation that basically you get a greeter passed around you try to access this variable but you know an actor's purpose in life is to prevent you from racy access to such a shared variable so it the compiler will catch that and say well you're trying to access this actual isolated property you can't do that this would have been racy this is a very simple example and you know you could have done that with just putting private fare but this isolation checking is much more contextual than just access control let's dive right in to cross-actor course where this becomes slightly more apparent so actors isolate your state and you have a function greet on this greeter and it returns a string now if you call this greet on another actor so here we created a new actor we call greet on it that is a asynchronous core because that's how actors work for you do asynchronous communication so if you want to call greet you have to corvette actor eventually processes this request processes that returns you the string and we do that by implicitly noticing oh that's an actual core so you'll have to wait on it and that's the classical asynchrony model you may be familiar with from other languages we recently introduced that to swift and it kind of plays very well with actors actually now the other side of it is we can also do so here's an extension so you can add functions to actors like that and we can do one of two things the first example is a asynchronous function you can see the async word there and we call this greet function on self basically because we are already inside the actor that function is not a synchronous right you just call it directly if however you wanted to call an asynchronous function so something that was declared explicitly asynchronous then of course you would have to wait on it right i think a weight but we have this notion of inside and outside factors a little bit and it kind of implicitly adds effects to invocations so um so so far so good you could have maybe done this in different ways um but here's where the language uh the language integration really shines so we're also aware of uh kind of closures and where they execute so here we have this greet function and i want to increment this greeted counter but i want to do it later at some point i would just want to return the hello name as soon as possible and then increment this counter silly example but this is racy right because if it's this much q global that's going to run on a different thread this would have been racing so the compiler can know that and can prevent you from doing such and raise the access and we call this sendability checking so basically we're checking well this closure is sendable it could be executing somewhere else so this isn't safe of course you have ways to express what i just said but oh i want to run this thing later on on this actor and that interrupts very well with the rest of swift's concurrency where we have tasks so task if it's kicked off from an actor it will execute on this actor so this way you can get the safety back uh that's all i will cover about swift concurrency today but because that's all we need for the distribution part but if you want to read more there's really really a ton of really cool stuff to learn especially structured concurrency i think is very cool because we're probably the first language to take it all the way into the language with static checks and like that so it's very exciting but the next step is of course uh distribution so we have an experimental language feature that's available today in likely two chains uh called distributed actors which of course when you think actors you often think distribution and i'll introduce a little bit how to use them how we use how we represent them at runtime and then we'll dive into some examples so the idea is that we want to have this nice learning curve where okay i know how to program vectors i know about state isolation i know that these greed invocations basically get turned into something that that actor then later executes and distribution is very very similar to that right so in order to make your code distributed ready you just need to make the actor distributed so now it's a distributed actor and distributed actors can declare distributed functions um we'll talk a little bit about why that needs to be explicit but i actually think it's very very important and good that we have this explicit and not just magically turn things into remote cores that's not what we want we want to be very explicit about what is remote and what not right so here we have uh basically a ready to go distributed actor and let's talk about what that gave you so the first thing that gave you a distributed actor gains the property of location transparency so location transparency is this way to think about your program and objects where you don't care where they're located so it could be located on the same host you're executing or on a remote host and even more so maybe it's not actively running yet but you can treat it as such and maybe the other side can spawn it on demand that's a very interesting pattern we don't have implemented yet but you can totally do that right so you program against at least statically against the distributed actor model but at runtime of course there's two actual real cases so the actor is either actually actually local in which case it's the exact same execution semantics as the normal actor in the language or it's a remote actor in which case invocations of distributed functions don't actually invoke the code but they become messages that we put on the wire and the remote instance actually gets and executes that we'll talk about how we do that in a second so distributed actors not only have to you know turn things into messages they have to enforce slightly stricter isolation uh so similar to the previous example a distributed actor is an actor so like the var greeted is isolated the same way you would expect but has two additional rules to isolation in our i will call it quote unquote local only actors the access to the let local id13 would be allowed because we have the guarantee that this is a constant property of a sendable type this will never change so we don't even care to hop to the actor for execution if we wanted to reach that local id that's very useful if you think about it because you very often wants to print okay what actor is that right some user id what actor am i interacting with so you need something you wouldn't want to hop to the actual need to print out it and print it out in a debug statement it doesn't make sense so we have this but in distribution you cannot ever access a substate so we prevent that and you cannot access any non-distributed functions of course because we don't guarantee that they can actually be distributed you can pass closures to functions after all and distributed functions would reject passing things that we know we cannot serialize so that's what two additional rules in type checking first more to that i'll have a few more examples and yeah the greet of course you can code because it's a distributed func there's a little bit more to a distributed actor so we add two implicit properties which are non-isolated vars that means you can access them again without hopping to the actor and one of them is the identity as i kind of said with the user a pattern where you want to print out a what actor am i interacting with distributed actors always have an identity the specific identity depends on the transport you're using maybe it's just an integer if you're an ipc system maybe it's a uri if you're kind of you know distributed but you want to point at a specific one or maybe it's just a name if you just want to express the concept of a specific named remote greeter so we inject those two properties for you and we enforce that they're always initialized now the other thing that distributed functions do is as i hinted at they guarantee that we can actually form a message for the for the message this is kind of describing so here the greed takes a full name and that full name is a struct as you can see here but it's not codable so codable is a protocol in swift protocols are similar to interfaces that very briefly and more or less a bit more powerful than that but very briefly they're like interfaces so it doesn't conform to codebar encodable is our way of expressing at compile time we know we can structurally prove we can serialize this and it's detached from any specific encoding scheme so at runtime you can say hey encode this using a binary you know whatever encoder or json if you want but we need to prove to the compiler that we can do that so we very easily just add codable the compiler is happy and we're kind of guaranteed that we can form messages for these things uh certain things like you know uh closures you can't make that codeable uh etc etc and the compiler helps you out with uh doing the right finger now the other effect of distributed functions is on the core sides if you just have a local actor greeter you declare this grid func we saw already that this causes a syncopate across actual boundaries right so we have to await this greed core uh what distributed chords cause is not only do is it asynchronous because there's going to be networking involved it's also throwing so swift's error handling is fairly interesting so errors in swift are values so they're very cheap to construct and carry around you know same as returning a value they're not as heavy they don't have like full stack traces and we kind of mark call sites that may be throwing so you're informed that this call okay this could be throwing it's not as rigid as you may be uh worried now oh my god this is like checked exceptions in java it's not as rigid so you just have to acknowledge that you're either handling or escalating you don't have to be fully typed about fizz and i think that's a very good tradeoff we're making here okay we have uh things in the type system where you can express well i actually know this remote actor is definitely local and the type system is then aware of that and wouldn't cause this implicit effect of throwing because that only comes from the networking side of things if you declare that function to be throwing it's of course always throwing because it's your errors now runtime so the interesting bit comparing a language integrated actual runtime with just a library one is how kind of narrative it feels it's just normal functions just normal course and how efficient it is in very presentation so here we here here i show how you can get a result resolve a remote reference to get an identity which somehow uniquely identifies some worker actor in the cluster you pass some cluster transport and you either get a local reference or a remote reference the important bit is you never always get a remote reference back we kind of uphold this location transparency idea here because you want to write this code once and then run it maybe locally and testing in a single node but when you deploy well these actors would be actually remote right so you code against distribution but runtime is remote or local right now here is the cool thing we can do because this is so well integrated with the runtime and literally how we allocate objects uh remote instances obviously take minimal storage space uh we do allocate storage for a transport and identifier as i showed before but we don't allocate any storage for the actors whatever state it wants to have right obviously um the important bit here is that this is very much ingrained with the rest of the swift's and tools like debuggers or you know anything else can be uh aware of that you know you're trying to in ldb print an actor well we're not going to give you state because it doesn't have stage right so we can really go deep and wide in with these integrations now remote cores chords work like that you literally call the function if it's local you just invoke it if it's remote we call this func that kind of creates a massive representation and calls a specific transport this actor was created with to send a message using it the details are in the middle of evolution so it's still changing but this idea now this is the language feature but just the language feature you know can't really do anything so we still need a cluster runtime for example a cluster on them you could imagine other ones because the language feature is kind of agnostic to what transport you're using but the most typical application of distributed access is really these clustered environments so i'm more than happy to announce that today we're open sourcing a complete clustering library it's available probably right now or in like 10 minutes or so uh should be there after my talk on github apple uh swift distributed actors you can give it a look and what is that so it's an implementation of this actor transport protocol that i mentioned before with the idea of okay you can program as if your cluster is just like a very very big machine right that's the general idea of distributed actors and we implement all of the core building blocks you would expect from such a runtime conceptually again i mentioned all the other runtimes conceptually similar to those listed here and implemented using swift neo which is a kind of veneti of swift actually kicked off by norman the original offer well the main maintainer of nettie has kicked off us with new effort and now it has a full and proper amazing team taking care of that so we have a really great high performance runtime to base for so long so i want to highlight two or three features today uh what this actually gives you um how do you how do you locate your distributed actors how do you react to their failure and how you can interact with the cluster itself a little bit because finding actors is the first thing people ask when they saw the language proposal which we put out there uh earlier but we didn't announce that we have a full implementation yet well today we can share how you can solve these difficult problems that we sold for you so discovering actors basically means that you have to find these identities so you can core resolve on them and get these actors back right but we provide a higher level abstraction we call it the receptionist pattern because like a reception in a hotel actors have to register at it and then check in and check out basically so as you start your actor and you're in it you can say receptionist register self with some key and on another actor system a node you can basically do system receptionist subscribe to workers and this gives you an asynchronous stream that will keep giving you actors as they join the cluster and then you can invoke receive worker how it works and practices this new node joins the receptionist gossiping around about those actors we have two actors on different notes there we eventually get to know about them and we invoke these uh receive worker and the async stream you you see running there uh fairly typical uh just really well integrated in the language i think that's really cool actually so the other thing is obviously if you find actors a big piece of distribution is that everything fails all the time so how do you react to that again we provide this as an abstraction where you only have to think about actors right so you have your worker and you want to watch the greeter and whenever you whenever a greeter terminates you you want to basically remove it from this set right simple logic you can imagine a worker boot or something like that and the interesting bit here is that either the initialization of a remote actor or a local actor for that matter triggers first termination so you know just actors coming up and going down naturally triggers this termination signal so you can remove them from a worker board or if an entire node dies you know then there are no d it hosted a bunch of actors uh terminates then we have failure detectors that trigger and we notify you about specific actors having failed we do that by assuming well the entire node is down so all of the actors that were known to be living on those nodes are terminated and we inform all the other actors that have been watching them so never really have to think about specific cluster events per se you just program against actors so that's really nice we kind of collapsed the failure space into you just have to care about actors not about specific things happening in the cluster you can care if you want to and the lower level of that is implemented using swift cluster membership uh that's the library we open sourced earlier uh last year and it implements the swim failure detection with lifeguard extensions inspired by our you know people at hashicorp we did a really good job with that and how that failure detector works is a bit smarter than just your plain old failure detection so with yes it pings directly a potentially favored node using like a heart beating mechanism but if you fail contacting that we ask other nodes if they also think that node is unhealthy because very often in such systems it can be that i so node a is actually unhealthy and was just not processing the responses in time so we this way can reduce the force uh force uh force force failure detection uh by quite a much so we do that and eventually we mark a note down when it becomes down we issue this terminated signal about the actor if you want to dive in really deep the actor members have a specific life cycle what i just talked about is reachability and you can plug in your own failure detector and the purpose of the failure detector is to move nodes between reachable and unreachable and then we have another plugable mechanism that's a downing strategy and the downing strategy's purpose is to observe these events and eventually issue a down command and again once you issue down we automatically handle all the termination messages for you so you can go low level exchange any of those pieces if you want and this is kind of prepared you can go and deploy this on bare metal you don't necessarily need a full you know kubernetes whatever to manage health for you we have that if you want to you can just uh plug it in here so just it's very extensible and how do you consume these events you've already seen an asic stream but you basically get an async stream of cluster events you can subscribe to that and apply changes to the membership every time you apply a change to the membership you get a change event that tells you well what was the change and you can know oh leadership changed or reachability changed things like that and lastly a real world example building a worker board with all of these things you have a worker it does some work on some work item and then you want to have a worker board so it just has a set of workers and what it does is firstly it subscribes to all the workers in the cluster adds them to the set uh it watches them for termination so when they die when they die over nose die you remove them from the set and your kind of balancing work across the cluster is as simple as just picking a random worker and forwarding the work to it that's all you need to do really obviously you can get much more fancy than that but this is the basic really to get something running and the cool thing is because all of the networking and all of the serialization is kind of encapsulated in the transports uh we only have to instrument it once for distributed tracing and tomorrow there is a talk by uh tim tim is a part of a soar side switch work group about distributed tracing and if you want to see that in action uh he also has a production system using those these things i highly recommend checking that out and again i think it's very important to highlight that tracing is great we know that but it's always so hard to integrate in an ecosystem right that's usually what prevents people from using tracing at all so what we have in swift is again language features that are specifically tailored to make tracing and instrumentation systems easy and robust so let's test local values another feature we introduced this year in swift highly recommended to check out the talk especially telescope values are kind of novel that's not something you have in other concurrency runtimes and they prevent tons of bugs you normally have with fred locals right very much better than that okay with that i want to quickly sum up that's the slide we had before with all of the issues so we said going distributed is too hard to learn so with distributed actors you don't necessarily reuse the code and it's still very different because you have to think about distribution but you reuse some of the skills it's still the actual model you still use the same semantic way of reasoning about things so it's much easier to adopt distributed systems for people versus less failure scenarios there's less specific failure scenarios you have to think about because verse again just normal functions throw in and accurate termination if you want to care about it there's no error prone entangling of business logic and networking codes because we encapsulate it all in the transport which again you can configure but it's far away from your business logic there's no more ad solutions by teams because we can collaborate and write reusable swift code and reusables with algorithms against distribute detector protocols and we can use that in various scenarios even across different transports if you want to and it's easier to reason about because it's you can test it locally distributed actors are super easy to test just in unit tests you can have unit tests for distributed system inject failures anything you want and tracing is trivial to build in so you have a very good production experience as well so this is available today it is an experimental language feature so you do need to use nightly toolchains and a little bit of the build process is slightly annoying right now that's because you know we're kind of bringing up the language features still as we evolve the library but please have a look i'm more than uh interested in hearing all your feedback all the questions and ideas you might have please use the swift forums i'll be more than happy to catch up with you over there that's all i got that's the links to follow to the forums and repository and thank you very much for your time you