SF Scala: Grant Gavares, Building Cloud Services with Akka and Scala

SF Scala: Grant Gavares, Building Cloud Services with Akka and Scala

Recording: SF Scala: Grant Gavares, Building Cloud Services with Akka and Scala

[Music] all right so I've just quickly a little about myself I've been here about three years the Oracle cloud group itself is only about four years old so kind of salsa momentum building here with some people I've worked with in the past and decided this was a place I wanted to be in my former life I was one of the very very early people it Amazon Web Services back in the early 2000s got to be a part of that very luckily got to be a part of that learned a ton of stuff and a lot of those very same people that sort of built AWS back in 2003 2004 that group of people is now here building this cloud which is primarily what convinced me to come over get in to work with those fantastic people and getting you know another crack at building the cloud now that we have you know 14 years of hindsight so it's been a lot of fun I've really enjoyed it I never thought I'd be an Oracle employee but you know Here I am three years later we've got this beautiful new office that we built and it's been a lot of fun so I wanted to talk to you guys a little bit today about how we're using Scala naka within our group to build cut you know multi big multi-tenant cloud services that scale out - you know obviously right now tens of thousands hundreds of thousands millions of customers so as I was building this deck I realized I probably had like three hours worth of content and I had to cut it down to 20 minutes so I'm sorry this is gonna be a rush talk I'd love to talk about any of this in far more detail for as long as you guys are willing to talk after the you know after all the talks ago all right so some things really like in the scala community cats akka HTTP 4s and play we use both enumerate em is fantastic we've recently started doing a lot more in searcy we also use play JSON and I really like Li how he stuff as well and then lastly slinky has been a new addition to our toolset I use it personally in my personal projects we're using it here we do a lot of internal admin dashboards with it but our consoles built and react and so there's a potential that we could use slinky as the scala jeaious wrapper around that for a future you know customer facing console development as well so we don't use everything from all these libraries for example in cats we really like some of the monad transformers they you know some of the syntax help was around either's that's great stuff we're just starting to move to use cats effects things like that but but most of these libraries have made developing code in Scala a lot better for us so I'm going to talk a little bit today about why we owe and why we use akka and afford here a little bit how and why we use akka and sort of maybe draw an architectural diagram for you guys and how we build our systems so primarily why do we use akka there's two big things we get from akka one is clustering it's fantastic when you're building distributed systems you either solve that problem yourself or you find a library or something else that can help you solve that problem akka does a fantastic job at this and it's a tool we've come to know really well and come to trust over the years and everything that I personally built here in the distributed system since has akka as as its backbone it's been fantastic the other thing that we really like from arc is have insourcing using akka persistence that's I think all the systems I build here as well have also all been event sourced there's a really nice decomposition that you get when you go down the the event sourcing route and I'll talk about that more in just a moment so for those that aren't familiar with akka very quickly akka solves a number of problems for us one of them is a sort of the model around how you protect state and concurrent access to that state and it's concurrency as in nodejs it's single threaded access to your stated in a given time that makes reasoning about how people access then manipulate their state very very easy to do that's really important because when you're asking people to come in and take up a new language especially a language that maybe has a reputation for being difficult a lot of the problems that we can take off the table they're around how they have to think about concurrency and data access and stuff like that it sort of removes those concerns from and lets them focus on some of the other aspects of learning the language and frankly I think a lot of people are excited to come and do something with actors that you know you've been hearing about are laying actors for years and this is an opportunity to put them into practice message-passing another aspect of it that I think is really valuable I think these two the first two things sort of lead and as well as the third bullet point or what really allow you to go and build distributed systems of vodka message passing and location transfer to big thing location transparency are two really big assets that akka brings to the table it means that the way I write my code in my local development laptop is going to work the same way when I take it into production you know latency concerns aside it doesn't let me cheat it forces me to consider exactly how I'm going to communicate with the other components of my I think that's really valuable we use actors in a hierarchical fashion I think typically when you're using actors a hierarchy if you if you have a hierarchy you're probably doing things right if for no other reason than simply having supervision but they are while actors aren't composable in traditional sense you do get some notion of composability once you have an actor hierarchy and I'll talk about that more a little bit later I mentioned supervision and fault tolerance and then obviously the clustering distributed system primitives they get you cluster Singleton's routers groups things like that these are all really powerful concepts you can go build them yourself as we had to do back in the early 2000s at Amazon it's hard problem to get right and instead we get to focus on building a you know building our products in our software event sourcing I just wanted to say a few words in case people aren't familiar with it unlike a traditional application where you might create a relational schema in a database and put all your data into that event sourcing instead just stores all the state in your system it's just a series of events recorded into a journal when you want to ream aterial eyes the state of your system you simply replay the journal read the events from the journal recreate your state there's some really nice aspects that come out of this there is no relational schema to manage so we don't have things like oh our ends we don't have the object relational mismatch and impedance mismatch that you know the the Vietnam of software I think it's been called so many times and typically it's used with CQRS one really nice thing about having the sort of journal architecture is it's an append-only architecture and it can be very very performant when your database is tuned properly there's no updates there's no deletes it's very very efficient so you can scale this thing out very very easily very high throughput on that single machine or your database tier and then very easy with CQRS to then go scale out your your view side your read side independently this is really important we're building massive distributed systems with lots and lots of concurrent users because you can go to each component of your system independently especially when you have asynchronous systems right sorry asynchronous rest api so this is a great pattern to follow so really really really enjoyed it over the last probably five years that I've been doing that okay so what is what are some of our software look like here so I'm not speaking to necessarily a specific system but we build a lot of systems that look like what I'm going to show you and we're still building systems to look like this so you have your traditional three-tier architecture you've got sort of your API and HTTP concerns up at the top whatever whatever sort of a protocol you're speaking you've got your middle tier that's your business logic and you've got a base tier that's where your database puts its data right this diagram is probably looked the same for the past 25 or 30 years what we insert here is we turn that middle tier we we insert acha into that middle term specifically aqua clustering and that middle tier is composed of in different machines right this is a obviously illogical diagram and across those in machines aqua clustering and cluster starting specifically takes care of understanding how we take our data and spread it out amongst that cluster Anaka does that by creating shards a number of shards that you specify upfront and then it goes in it manages to sort of it conspires with all the members of the cluster to go and spread those shards out even evenly amongst the cluster and once you have those shards out there you can start spreading your data amongst those things and as nodes come come to the cluster and leave the cluster the shards are rebalanced using an algorithm if you're choosing so you can isolate specific customers or if you're more concerned with i/o or memory pressure whatever it is you can sort of tune how akka decides to do that or you can just go in and say let me handle it myself so it's a really powerful way for us to sort of ignore a lot of a lot of the complexities that come from building a system that has distributed replicated or charted data like this in the diagram here I have these little orange boxes these lettered with the letter T that's stands for a tenant that's what we call our customers it's like you know an organization or a group we call it a tenancy what I mentioned that because I'll be talking about that a little bit more going forward so typically the way we structure our applications is a user comes into our API gateway they make a request and the first thing we do is we authenticate the person right there at the gateway so that every message that gets passed within our Aqua cluster always carries user information with it and that means we know who somebody is at every stage of the way now we don't do a thorough zation here because we have a pretty strict separation between a protocol layer and the layer that actually enforces business logic and understands the rules of what it means to allow someone to do something there's a lot of a lot of aspects of that it's not just permission based it's also things like have you exceeded your limits in your quota things like that have you been whitelisted for these features stuff like that so we don't allow that to leak into our HTTP layer and that means our HTTP layer is very dumb about what it's doing and we can substitute in other protocols if we need to G RPC or whatever maybe and we don't have to ship replicate any the business logic that actually matters so once that request comes in the API gateway again we were very careful to hide from it that it's actually talking to an aqua cluster we give it something that looks a little bit like a dowel but we call it a repo just taking the event sourcing terms there and it thinks it's just talking to this this thing with some nice functions that it asked the question it gets a response and it's within that repo that we hide the fact that there's actually an aqua cluster there so the request gets thrown into the aqua cluster and the aqua cluster based on some config that we give it knows how to look at any given message and figure out exactly what shard to send it to and it takes care of that for us transparently from our perspective we just look like we sent a message to some actor and it just ended up at the right place once it lands at the shard the shard might have mini we organized the the atoms within our shards around tenants individual tents and then the shard figures out how to route that request to the specific actor that represents a tenant that lives on that shard and again these these shards can be rebalanced across nodes seamlessly and the communication will continue to work and akka will take care of of handling all that sort of rebalancing and management of nodes for us so from our perspective the application actually a lot of this stuff even-even the notion of shards is largely hidden from us when we're writing our code you don't have to necessarily be concerned with that so if we double-click on one of these tenants here once the request arrives at a tenant one of the things we like to do is I mentioned earlier we like to build actor hierarchies typically we use this this guy at the top here this tenant is the supervisor for all of these actors and his job is to make sure that he restarts them if they crash he understands how to handle specific errors for each of those guys and he can so he understands a lot of the concerns around collecting metrics doing logging things like that so he creates all these actors and wires them up in a specific order the individual actors are unaware of each other so in this model here for example a request comes in to a tenant and he passes that message on to the aussie actor who does things to probably see all their authorization concerns talking to our identity service and that guy ford's a message on to some other actor he doesn't know it's a limits enforcement actor but it happens to be because it was wired up that way and he'll go and he'll check his limits enforcement and when he if the request that's coming in is sort of been satisfies all the the limits enforcement that we need to do he has a little function that we it says if it's this kind of message a query send it to the to this to this actor and if it's a command send it to the sector he doesn't actually know who those actors are and the beauty of this of course is that I can insert new layers into this hierarchy at any given time if next week we have a new concern that we need to enforce before we go and give commands to the AG or ask queries of the view we simply insert the actor into this tree none of the actors or any the wiser only the tenant manager the the tenant supervisor at the top needs to know about that you can imagine how you can use this in testing as well I can test that off see actor in isolation because he has no knowledge about the hierarchy in which he's participating it's very very easy to put probes on both sides of him and test this code very thoroughly as a result of that we've invested a lot in our testing frameworks we have really great tests and typically writing test test test methods for these actors is on the order of four to five lines of code so you got a new feature there's no reason not to go write a test one and that's all you have to do so definitely encourage doing that we have these actors down here that do views views and an aggregate what we call it an act for short is an event sourcing term he's basically the guy who handles incoming commands and he has workers that hang off of him and as well that do sort of asynchronous background tasks stuff like that and again the hierarchy really gives a lot of power in this scenario because this guy is responsible for managing those actors and dealing with failures as he sees fit at the end of the day back to the event sourcing a bit of this the AG if a command comes in update this thing create this thing and if it passes all of his you know his rigor and validations he then generates an event and that event is stored in the database and he responds back to the original person who inserted this request simply saying thumbs up or thumbs down about whether or not this this request was accepted asynchronously we have a view side of the equation who simply is just reading off the journal and as he reads off the journal he updates his view of the world it might be very different than how the AG views the world and in fact there could be many different views that present different projections about the state of the world back out to our query AP is or into our internal services and in fact this view again if you go back to the the note I made earlier about location transparency this view may be on a very different machine in fact there could be hundreds of these views that are sort of serving a very high volume of requests again the nice thing is none of these actors have to be wise about that they simply get a reference to an actor which might actually be collection of actors somewhere else and so it again allows us to scale those things without having to touch anything else in this hierarchy it's a really nice model okay again I was trying to cut this short so I'm gonna up a few things that we really like to do lessons we've learned how we keep things running smoothly as the code base grows ADT is all over the place we think those are extremely important use ADT's everywhere they save you one of the complaints about actors is they they sort of break a lot of a type state to get from a language like Scala because you have this great big receive block here that is you know it's a partial function which means you can set it anything you want and if you don't know how to handle it you will you're a curable crash ADT save us from this so you can see up here what I've done at the top is I've have a fake actor here I've created an ADT here called message and there's exactly two objects and down here we just detect that we detect that object I can hand it off to a total function rather than trying to handle in the partial function and now this code would not compile and I'd be notified by the compiler because I forgot to handle the pong message this is this is what saves you from the you know untyped sameness of your partial function the receipt block we follow this this practice religiously and we we can catch all the messages we don't expect and it's always an error or a warning in this case we do the same thing for our error messages there's no returning strings we return a DTS for error messages and again that forces us to handle all possible errors in the cases where they need to be handled we also if you'll notice here I'm not embedding HTTP status codes in any of my messages that's the concern again of that very outermost layer and my akka layer has no idea that there's HTTP involved anywhere in this we separate those concerns and it's up to the HTTP layer to make sure that it handles all of these again the compiler will compiler will save me if we don't handle them because it is close to type a seal type and it's saved us again you know you won't get exceptions from our api's he'll always get well-formed error messages last thing that we found really is important we used to have a trait that all of our messages would extend and it would have at least all of these fields in it if not more and then all of our messages ended up having the thing they were concerned with with like 12 other fields stacked on the end and it was all this replicated stuff and made testing a little more annoying and frankly made all the case classes very with line wraps and stuff like that and so we switched over to actually using message envelopes everywhere and I think it's a much better practice there's a little bit of annoyance and that you have to unwrap those envelopes before you can actually pattern match on them but it's a small price to pay the other thing is that as we discover new things that we want to add to the envelope it's only one place we have to touch we don't have to go break all those messages and if you're familiar with event sourcing you also know that means versioning your schema versioning your schema because now you have to be able to read the old messages that didn't have the fields and the new messages that do have the fields so this saves us from that as well there's a couple important things here I want to call out and these ideas aren't original to us but we follow them one is I mentioned earlier we always carry the user around with every request and we like embedding a deadline in these messages as well it's great because if I get him if my queues backing up as an actor I can take a message off of that queue I can instantly tell if it's safe to just discard the message completely and I do that just by saying deadlines dot is expired or whatever the function is it's very very quick it's very very easy and I don't have to do date math all over my code as simple as it is people still get it wrong and it clutters every good and then lastly we can better request ID everywhere that's the same request ID that comes into the HTTP gateway and that goes everywhere and we link that up to our tracing tools so that when we need to go and sort of marry our access logs to our service logs to you know open tracing or jäger or whatever it is you like we have a common identifier across the board both at the akka layer at the HTTP layer and everything else and these are the things that have really helped us sort of you know keep the lights on as it were I'd love to talk about this for a long time monad transformers rares these are great they seem scary at first but I decided I meant to delete the slide because I could talk about this for an hour um so they're not scary they're fantastic happy to talk about them later and I think I got done and maybe under time so we're hiring obviously nick mentioned it we have a lot of headcount here and just to extend a little bit of what he said we are the container native group in this in this office so we we own functions here which is like our version of lambda just launched it open world we own manage kubernetes registry we're building out a ton of new Greenfield services which I can't do to talk too much about publicly right now but it's an exciting group we're growing really quickly we have a lot of headcount but we're looking to fill so if you're interested please talk to myself or Nick or any one of the other people there that are here Oracle if you a little bit more about it any questions yeah we use akka test kit it's fantastic probes are great we have big test harnesses we we invested a lot of code in either traits or objects that live outside of the actual test code itself we mix those in and what that means is that the actual amount of code you have to write for any given test is very very small we test from the outside right we want to observe input and output we don't typically try and inspect state within the actor there's a few exceptions to that that are very specific to the services we built but we're you know it really is sort of I essentially this message I expect this message to come back I may go look in the journal to see if the right set of events were omitted or I may try and apply that journal to our views and then go ask the view does the world look like I expect it to but you know very much from the outside using architects kit so oh I'm sorry yes yeah the question was how do we test our actors yep yeah let me see if I can say it properly so it's um it sounds like what you're asking is do we use is there any sort of type safety we can rely on when building out the act or hierarchy the way we compose our actors so yes I said we do it based on configuration and what I should have said is configuration as in code it's all it's all in code that supervisor actor it's a very very small amount of code the first thing he does in his constructor is he creates all the actors passed his references between them so he wires them up in the order he wants them to live in and then he sort of starts talking to the first one and that chain everyone is just doing the forward down that chain which is why the last guy can say you know send her bang and it gets all the way back up to the HCP layer but we get type safety just by virtue of the fact that you know we're calling constructors on these actors and passing in the you know dependency injection if you will passing stuff into the to their constructors they get nude up and we just pass rest around actor restaurants that's a great question do we find that our error file is getting really big that's sealed traits and that means all the errors in our entire application would theoretically have to be in a single file so the answer is you know define big I think I think in one of the bigger services we probably have I would guess that files maybe 100 lines of code there are a lot of error codes there but you know it's pretty manageable and there's not really code in there it's just type declarations there's very little in the way of actual functions I think a lot of times yeah I wouldn't say it's huge but that I'm actually I thought about this problem a little bit I don't I don't have a good solution to it microservice is split mal I don't know anyway else really two questions [Music] yeah so I think the question was looking at the the new sort of typed actors that have come out of vodka how do we think that's going to change our approach to a lot of these problems I'm not sure that I've looked closely at the latest iteration of that I've listened to Roland Kuhn talk about it a little bit I really like a lot of the ideas he has around it but I haven't actually used them I could see that definitely changing the way we approach them these problems there's there's some stuff that I think we were concerned with now that if those if those really do provide what I what I hope and expect they will that we can get rid of a lot of this stuff it's not needed anymore I don't think we'd rewrite our systems I think it'd be more like new systems we would start exploring those things all right [Applause] you