Scale By The Bay 2019: Paul Cleary, Re-programming the programmer, from Actors to FP

Recording: Scale By The Bay 2019: Paul Cleary, Re-programming the programmer, from Actors to FP

[Music] all right it's 4/20 everybody have a good long day here everybody ready for a nice talk on DNS I hate a 4:20 in the afternoon thanks everybody for making it out and today's talk I call it reprogramming the programmer and this is an experience journey through building a system and the evolution of that system as well as as hopefully there's a little bit of personal evolution that that comes through in the message as well so the agenda really is we're gonna give a little bit of a background from the Golden Age of hip-hop through 2015 the the building the few versions of the system that we built and then just wrap it up with some lessons learned so so starting out I had spent most of my career building software you know in Java and c-sharp about 16 years of that and there is this there's this characteristic of the systems that I built that became recurrent and and and I refer to it as Mook which is maligned object-oriented programming you know it's very careful not to say are all object oriented but maligned object oriented programming where I say any system characterized by lots of objects imperative programming side effects mock objects and wrapped around some kind of dependency injection framework and so these are this is the typical of the system that I was starting to build and these types of systems are hard to reason about you almost have to download all the code to understand exactly what's going on the system has become brittle over time become difficult to extend an error free way I actually did spend about a year in management because I was convinced that there was no path forward with programming for myself people don't compile I mentioned the other day that there's no Stack Overflow for people so kind of made my way back into engineering after that and within these types of systems when you Moop type system is we like to throw a lot of patterns at these things we have MV P and V Star MVP sorry that's an another buzzword MVC and V star gang of four patterns etc and we like to think that we have this really awesome structure and the data flows within our system are very obvious and very easy to extend the more interfaces the better but actually turns out that in reality it's not quite as obvious there's a lot of mutating of state going on it's hard to track what's going on when in the system and and I tend to think that a lot of the cause for for my headaches at least several years ago was due to dependency injection frameworks in particular auto wiring so how many people here on have actually written code like that awesome so let the record show that we are all guilty in this audience that's 20 dependencies I've seen as many as 40 and I like to say that no matter how many of those you have there's always room for one more so that's say any time you want to add behavior to a system you always have a spot for it and of course the more dependencies that you have in your system the more staging and mock objects and expectations that you have to build as well you can imagine when you want to add a parameter to a method or something like that you have to go and update 30 or 40 unit tests and the expectations that your staging with your mock objects so in 2015 I had an opportunity to change things Comcast runs a program called Summer of Code it was a chance to build something from scratch so that I could avoid all the MU bubbliness and it was DNS as a service so what's DNS this is the most interesting part of the talk how many people here DNS ok we can commiserate later so it is hierarchical and decentralized sort of two key words there and the idea is that you have all these records that are all over the Internet you need a way to manage them in a redundant way and I like to think that I like to think of DNS as a large loosely coherent distributed Cady baton and the idea with DNS is that zones are hierarchical each node in the hierarchy that you see here is its own DNS zone so dead is a DNS zone comcast.net is a DNS own system castanet is a DNS zone and the idea is that these zones can live on separate DNS servers that's how we get the distributed nature of DNS and really all a DNS zone is is a group of records and so what is a DNS record so a DNS record is how we name things in how we answer questions like what is the IP address for scale that by the bay dot IO and we see here an example of a DNS record and just real quickly you know the first part is often referred to as the fqdn fully qualified domain name it's the key in our key value map the first part of that is the record name the last part of that is your DNS zone name we have this TTL which tells you how long you're going to cache your browser to cache this record so you know if you do a dig and you get a look up in 7,200 it's two hours so please don't ask your DNS engineering team like why your change isn't seen yet it's likely cached somewhere the record type is important because that says what's the type of data that's held like the a record is ipv6 and then of course we have or ipv4 and then there's ipv6 all right so the the concepts are pretty easy a zone is what we care about it's just a group of records so that's that's the kind of the crux of DNS so version 1 of the system we had a short time to build it that the primary audience was we're going to support internal engineering teams and managing their DNS we imagine that we might have a thousand DNS owns 100 records per zone records we assume records did not change all that much and that we needed to audit all the change was important if somebody was to delete WW comcast.net we know who did it ok so going into version 1 the approach was let's use scala now I had actually already some experience with Scala naka the team that I rotated into the Summer of Code was already familiar with Scala akka I decided not to use Java and so that we could build it move free so the design that we chose was event sourcing as the architecture pattern and the nice thing about event sourcing is that there is no state that exists in a database you have events that are stored in the event Journal and when an entity is loaded then the events are basically replayed in order to generate current state so what does that look like so this is an example of event sourcing when you have a command come in to add a record to a DNS zone it's going to apply some changes to the backend when that event is complete or that command is complete we're going to persist the event into our journal kind of like a right ahead log the important part here status updated in memory that makes reads hella fast you get like sub millisecond resolution when you're doing queries against this events or system and when you cold load an entity really what you're doing is saying hey give me all the events since the beginning of time for this entity and just replay them and order back in order to generate current state so that's the idea with event sourcing and it really turns out that akka is really well-suited for event sourcing it has it out of the box with akka persistence and the way that you go about and implement this is zone state is just stuff in memory it's just typical case classes we made each DNS zone its own actor because that was the entity that we cared about the persistence comes for free and it turns out that there's other tools like aqua cluster sorting and clustering that'll come into play as well so in memory state what does this look like it's just a case cause I have lists of all the changes that I can query I have indexes that you don't see here you have your record data and you have your zone metadata and that's that's what you have to implement for the your zoom state so to speak the the key fracker persistence is you extend this you get this trait called persistent actor it provides the magic-like persist and that's how you save an event in your journal right it's just it's a method that just comes out of the box you don't have to worry about anything else as far as precision you're persisting your events for the replay when an event dns zone loads you get to receive recover that that you have it's a callback and it sends the events in and then you just kind of regenerate their state so that's that's the the akka side of event sourcing akka clustering is one of the technologies that you use and we use it for a che so aqua clustering uses gossip underneath the covers in order to figure out who is all who are all the nodes that are participating in this cluster and then the other piece of aqua clustering that we you employed was aqua cluster shorting and macaca cluster shorting does is it says you know answers the question like okay I have this persistent zone thing you know how do I make sure that it exists once in the cluster an aqua cluster starting is really part of all that so in cluster starting a shard is really nothing more than a group of zones zone actor instances and the cluster starting system just makes sure that your your zone actor exists once within your aqua cluster so for example when you get a command that comes into the cluster cluster start and will just automatically make sure it forwards it on to the rate so inactive there's no an actor that you care about the last thing that we get with aqua clustering is high availability so if one of the new fails then the shards and that node that failed or automatically redistributed to the remaining nodes in a cluster cool and that's our persistence layer the nice thing about acha acha persistence is that it has these these plugins and all you gotta really do is pull in the right dependency and just configure it and you're done so to recap our event sourcing solution we're able to go from zero to pride in three months the system was super simple I mean there's an angularjs front-end not super simple but the crux of the system was actually really really easy very very little code a shout out to the aqua community you know I was talking to a lot of people and the user groups raqqa during during this time rolling and I actually talked to in the user groups and he was great ok those have a tool for everything out of the box and and the system really wasn't didn't have these characteristics of loop and as at least right now so we were in production for about three months and then the larger DNS organization at Comcast found out about we're doing and said hey how about we use this for all of Comcast's and so the Comcast has one of the largest D foot prints in the world and what does that mean to our system so instead of a thousand DNS uns we have millions of DNS ins instead of a hundred records in a zone we have our largest zone at Comcast is 5 million DNS records records change all the time and they wanted to be able to throttle updates out of the zone level it's a requirement that they gave us and this presented a lot of challenges from an event sourcing standpoint because you can imagine replaying tens of millions of events takes a long time and then also hold an estate in memory would just require gobs of memory so there is challenge fundamental challenges - to the the architecture that we had started with and so we decided to take a different approach and rebuild so that we could meet the scale and so we decided to stay with akka because it had been so good to us we want to replace event sourcing with octopus and octopus isn't with our own persistence layer built on DynamoDB how many people here DynamoDB awesome cool we kept without clustering and cluster starting because it managed the message routing and high availability for us so that was the approach and then there's a new word out here CQRS that's kind of the architecture pattern it means an application with eventual consistency although it sounds a lot fancier to say that and the idea is that you have your command processing aside and then you have your read side and then you assume eventual consistency in terms of your state being updated the nice thing was is that we were able to get our zone throttling because again message delivery made sure that all messages were delivered to the same zone so we're able to use that zone actor as a choke point and then basically just do a wrapper it's a decorator around around handling these these requests and this is kind of what it looks like this is this is some code for creating our timer throttler and all you're doing is when you creep the timer to flounder or you're passing in the actual actor that you want to throttle and in a rate which is five messages per second in this instance so pretty easy and the last thing that we decided to do is actor everything we really did we the read side everything we're actors all the command processing were actors the DNS interface was an actor and so to recap what we did in the version 2 of this system was hims is Delivery largely stayed the same with aqua clustering and cluster charting throttling came out of the box again akka has lots of tools we were able to support our new scale we did have a write a lot more code I mean it was the entire persistence layer as well as the command processing we had to do a lot more heavy lifting than one was done with the event sourcing solution and really we learned that like we really should have known this upfront right you don't want to be like changing your entire application of architecture midstream while the ship is floating in the water so 2017 we had been in production for over a year we had a growing customer base and more importantly a grown codebase and everything was great until it wasn't so a few issues manifested themselves one was networking issues and the other one we're after issues on the networking side after clustering is fantastic at letting you as a programmer and not have to worry about anything about the network and the gossip protocol and the failure detectors and things like that the bad thing about aqua clustering is the network I like to say the network doesn't like you and certainly the network doesn't like any distributed system and so these kind of crapped up as we'll see you now recall that our cluster is multiple nodes managed by a Casa protocol well there's this condition how many people here aqua cluster cool so you know I'm going to set all right called a split rate and so the split brain scenario is where you're can actually in a network partition or prolonged network partition you can kind of form multiple clusters based off the number of machines in your cluster or in or a number of machines that are participating in your cluster and what this means is now I can have my zone represented multiple times and my across in real life and of course this ruins our SLA because now I get more updates per second than we were imagine because I have two instances of my zone instead of one and then the other thing it does is that we we design the back end processing for idempotency you could repeat the same change multiple times but the same change running exactly the same time was a little bit hard to figure out okay the other problem that we had was with actor issues the good things is about actors is actors bring sanity to the world of concurrency he can view everything that happens within the receive block as being processed by one thread at a time it's fantastic the bad things is that actors are non-deterministic by their nature and I'll show that in a second and we started because of all these actors having elements of Moop creeping its way back into our koo base so to talk about actors we typically think about invoking methods by invoking behavior by calling methods and getting a request response type of paradigm but with actors what we're actually doing is putting a message in a mailbox and we're just hoping that the actor will actually process it at some point so it has almost like technically at most one semantics but maybe wants to Mannix is the way I'd like to talk about it the other thing is that actors have two modes of communication all of them have tell which is sending a message is fire-and-forget and then Alka comes with this pattern called ask where you send a message to one actor and then you're going to wait a period of time and expecting to get another message back so what does that look like how many people here are parents how many people here have parents closely I get everybody all right so well kind of so some of the issues that we've had now here's dad and he's has a child Ryan is one of my my children he is a teenager and when Dad receives the to-do list from the wife I might said respond to her and say okay dear but when the trash needs to go out it's not my job I'm gonna go tell my son to take out the trash okay so in this instance when ryan gets the message take out the trash he's gonna say respond well maybe later and then my classic dad responds in real life is I wasn't asking what happens in an actor system is that that message never got to dad because he didn't know how to handle it it goes in dead letters now you can imagine my reaction in the next day when the trash cup the tractor truck was coming down the street there's the trash is not by the curb I never got the message that you said maybe later one of the other issues is with ask and so imagine we're eating dinner around the table and I might ask Ryan in this instance I say how was school and I clip their spawn back sit back to yourself that's the way that actors work now Ryan might be on his phone he might have his favorite twitch feed starts and then there's this kind of concept within acha where you can become an alternative type of behavior so Brian becomes watching his phone when it's twitch feed starts and what happens when Ryan's watching his phone nothing nothing's happening so in the real world when I asked Ryan how was school I get a knock a timeout exception right that was not expected so the state of affairs in versions version two I'm sorry's was version 2 is that aqua clustering cluster starting gave us a che a message delivery however they can comes along with its split brain and network partitions and some of the ugliness of actor or the network while actor is grating Santa seed to the world of concurrency are turns out that our system was largely sequential and actors were kind of overkill for what we're trying to do and in addition to that the lack of the determinism made the system overtime hard to continue to extend and even understand and the last point I would take in reflecting on this was that the persistence layer is getting unwieldy with dynamodb a you dynamo baby folks can ping me at happy hour fun stories all right so we decided we're gonna rebuild this again and 2017 I create the pet store is an example to say hey does FP really work in an application environment and I did find out that FP was indeed the anti-venom to Moop and we want to really simplify everything and the way that we really did this in version 3 was remove aqua clustering replace actors with functions and change our database and hey that we were to recede achieve high availability and and message delivery was using a message cute turns out they're really good at that turns out that the SQS is almost free for our workloads I think we pay like a dollar a month and it does give us at least once message delivery and retries in the event that your new goes down midstream of processing we are able to achieve still achieve rate limiting use and SQS NFS - I have the the naive rate limiting on the side there but we use FS - for command processing and so because FS - is just a simpler way to kind of reason about your flows than all these actors that we're working in the background and to give you an example this is really close to our FS - implementation that we have in production today it just turns out that all these things are just functions okay we pull for a batch of 10 from sqs we process those changes and we may delete messages from sqs as there's no errors remember our example before using actors taking out the trash and Ryan's responding with with maybe later now with functions I can certainly predict what Ryan might say by always returning some value that I could handle you might say maybe later unit assumes that he's going to take out the trash as a side effect and it makes my dad function and my dad life a lot easier as well you know if Ryan were to respond negatively I can respond with I wasn't asking okay also remember the the receive the the receive example Ryan's Watson is twitch feed and he becomes watching phone and doesn't do anything now Ryan has to say how was school and actually return a school status again making my job as a dad a lot easier while I'm eating dinner I could say Ryan how was school and yeah I can say Ryan that was great pal okay then the final thing is changing the database to my soup wall I was weird so the my bad just fell off so this is running a query in dynamo dB so we're trying to get all the records for a zone and this is kind of what it looks like and there's just the dot dot dot actually means that I pull data out right it's not meant for anything else that actually is there's other content there but when querying in my sequel this is what you say that's it before and after okay all right so recap replay soccer cluster starting with sqs NFS - during this time I eliminated thousands of lines of code I think it was about five to six thousand lines of code FS 2 is more declarative we are able to do this whole thing in about three weeks believe it or not and then went the other thing about having sqs and FS - is that we can go active active so our h.a stories actually better we replaced actors with functions that brought back determinism to our system new features from what's easier to implement without breaking everything in strange ways and changing the database sequel is easier and much more Universal if anything everybody that comes out of college no sequel which is great and then also the the power of sequel gives us things that are just not possible within DynamoDB are not easily possible or inexpensively possible so that's my animation all right and so sometimes I don't even know how did we do it so lessons learned overall sometimes premature optimization is ok we really should have asked if this system was suitable for all of Comcast's at the beginning and we would have avoided all the issues that we had I would say a lot about the scylla communities are incredible the akka community was fantastic I forgot to mention the type level community is fantastic as well I worked a lot with people and get ER and they were just awesome and responding to me it helps that I work with Michael pill quest as well that's my cheat sheet functional programming does bring the fun back and into programming I like to say it is the answer to most loop issues so if you're dealing with Moop look make sure you're looking at functional programming actors are great at concurrency they're not suitable for every application out there and then the last thing I learned I'm kind of new but you know it's just kind of in in hindsight is that user Rachel for the job message queues are great at doing message queue things those sequels out is tongue-in-cheek knows who goes to have sequels back in but there are work loads is appropriate for no sequel obviously but sequel is actually good for even very large data when you have billions of things hundreds of millions of things look a sequel it's still a still a really good thing for doing any kind of data processing this is my last slide the system we talked about right now is vinyl DNS it's out there in the open if you're interested in seeing what a code that has evolved multiple times looks like hopefully there are words in there but there's also a fair amount of good in there right now the scalp pet store is something that I worked on that I mentioned earlier you can go out and check it out hopefully we'll have a version for Scala 3 coming up and the last thing is I have I work with some people a second team that I'm really close to and we're doing imagine fs2 is declarative programming for streaming processing imagine being able to do the same things for Network functions in rust which is actually pretty cool so that's the link for mb 2 and that's my twitter handle that's my fault [Applause] would you like to take any questions anybody events the few minutes for questions what what solutions did you approach for split brain and what was the outcome so I had to take that out because we the time was cut so what we actually wound up doing was we built our own static corn-based split brain resolver which static form is basically if there's three news in a cluster and two the number two is forms of quorum then if a one of the members recognize that there is a cluster event and says hey how many people are my cluster and it's less than quorum it automatically shuts itself down so here's the fun part of that story I appreciate you asking we run our own private cloud and can you imagine a scenario where all connectivity is lost between all clusters what would happen everybody shut themselves down we had a production incident and that actually was the impetus for version 3 largely were like we just disabled it and crossed our fingers that was we looked at the logs and we were just like yeah it was working it did exactly was supposed to be we lost connectivity between all the nodes in her cluster was amazing mm-hmm answer question yep yeah yeah you'd have to do a redeploy updated configuration it was a static number you had to know yeah yeah so in your last graph is it looks that you're using the F single no FS to to peel the data and then now we're going to push the result and the TS records to a lot of the services and those are just like really read-only copy of the TS records and server requests is that the case is it the this one or this one yes so yeah the way that we we have F is to running on every single node that's out there and when a command comes in to the to one of these nodes it runs that a little FS to flow and it will actually issue an update via ddns to do an update to the DNS back-end so we actually have configured the authoritative masters in our system so we're actually sending a command saying create this record in DNS and over D DNS and it's given us a response back is answer your question would acha types have resolved part of the issues yeah so that's a good point again the setting was 2015-2016 before before before I think even archetype was maybe was in in prototype land but yeah I think that certainly from a standpoint of determinism it would saw it probably would solve our issues or some of the issues I haven't looked at I could type to be honest with you if some of the timeout things would happen how the ass pattern has evolved and things like that but certainly I think if we had something like archetype it would have definitely been easier yeah and one more okay in this graph you said that if it's to handler runs basically on every node that you have so you could have two messages for the same record come in on two different places do you basically use the transactional qualities of my sequel to make sure that your state doesn't get into an inconsistent state yeah so that's a good question when yet when you're distributed in an active-active standpoint at some point you have to figure out you know where do you stop so you know every time a request comes in we interrogate the message and the current state and the database to figure out hey can this thing actually happened certainly there's race conditions that could go on and you're gonna have that in this architecture we just more or less accept the fact that somebody could be doing so in theory you can't be doing a delete before an ad but you could processing them out of order and a DNS back-end you know truth is we don't the most of the same record updating at the seams at the same exact time doesn't even really happening to let you know scope of our system we handle probably about 50 to the 60,000 updates a week there's millions of requests across the system but 50 to 60,000 per week is still pretty small so the chance of that what you're happening we just kind of live with we have the audit history so we know it happened but we haven't seen that in practice but thank you it's a good question great well thanks so much Paul for that really interesting talk thank you [Applause] you