Scale By The Bay 2021 : Nolan Grace, Live Coding: Building Stateful and Scalable Reactive Streams...
thanks so much and uh you know super happy to be speaking again at scale by the bay for the second year in a row and uh with that i will just jump straight into things so uh so quick agenda um yeah we're gonna be doing some live coding always always high risk especially with timing so i'm going to try to get everything in in the 30 minute session so we'll start off with a very kind of quick intro just about me we'll go very very quickly into just some quick thoughts musings i have on stateful versus state lists then we'll introduce for those of you who are not familiar just kind of aca streams and when i say quick all of those things are going to be very very brief just want to make sure you know no one gets kind of left out and then we'll jump into um you know some design patterns and uh the live coding session so with that let's jump right into it so um i actually recently got a nice promotion so i'm actually the engineering manager for our portfolio and data team at m1 finance so um i am also a certified sommelier so um whether you want to talk about reactive things or food and wine i am i am absolutely in shameless plug for m1 finance um we are a investing app so we have a checking account we have an investing app where you can buy and sell securities margin lending and we also just released a credit card very recently so um thank you for indulging my shameless plug but uh yes it is a excellent company to work for and all of our back-end services are built using akka lagum and scala so it's a very cool tech stack to work with so now my musings on stateful versus stateless so simply put i like to think of you know there's two kinds of data that is is super prevalent in building uh distributed applications so you've got data in motion something like we're consuming messages from one place they're moving around our system between you know different functions different services different messaging cues and um you know they're kind of ethereal they'll come through the system and then at some point they're gone and they're done being processed then we also have data at rest so this is typically when people think of state and so state can be in memory or it can be durable there's a lot of different kind of options for state um it can be sharded it could be you know all in one place it can be in a uh you know data lake somewhere but when we think of state full streaming it's the idea that how do we take this moving data and cross-reference it or apply it to this state full data and do something kind of useful and interesting with it so it's been a especially in distributed systems or large scale systems it can sometimes be complicated and uh you know tying these knots through streams and stateful services can can be a little complicated so i just wanted to uh you know do this session so that we can talk about some of these patterns and some of the lessons that i've learned uh over the last couple years you know really using uh akka in production with that what is akka um akka is this actor model designed um you know on scallos kind of with a scala first mentality and um it's it's exceptional for building distributed systems the first piece of it we're going to talk about is akka streams akka streams is this really nice back pressured reactive stream implementation where you're able to kind of take these operators and plug them you know one by one by one and do some really really creative and interesting um message processing and uh asynchronous operations multi-threading with uh akka streams and so when i say back pressured what that means is the idea that downstream operators do do pulls upstream that's this little arrow here so if our sync has uh time or you know has nothing to do it'll pull from the operator before it to say hey give me something to do and same thing with this flow and all of them i really like to think of it as a bunch of people sitting at kind of an assembly line and the person at the end if they have nothing to do will look to the person to the left of them and say hey give me something to do and everyone down the assembly line is kind of operating in that same manner and when someone you know needs a request they'll look upstream and and then work will come down you do your work and then you push it down to the next one the reason why that pattern is really useful is so that no specific operator gets overwhelmed you wouldn't want to be in a situation where your flow right here is a little bit slower than your source and so the source can push data in at 10 messages a second and the flow can only process them at one message per second eventually memory you know there's some buffer here memory fills up and are we have some sort of outage so akka streams is really great and we will be writing some of that today the other big tool that we're going to be using is called akka cluster sharding so akka itself has this idea of actors and so actors can represent state or work and with aqua cluster sharding we're able to spread these actors around the cluster but still have access to them you know as if they were local so we could generate an actor that is collecting data for amazon or tesla and we can guarantee that there will only be one example of the tesla or amazon actor in the cluster and if we're doing over here and we process some message that has to do with tesla we can still forward that message over here and get you know an acknowledgement that it's been processed and then um finish processing it so that's what we're going to be doing today but um cluster sharding and akka streams really great tools to bring into your your design next time you're building a greenfield application so the use case we're going to build an app that is processing real-time financial data so we're processing real-time finance quotes and this is what our case class quote message looks like so we've got an id accompany and then the the field we're really going to be using is this last so what is the last market price for this this share and then we have a timestamp and so we're going to process these messages and so this is our our stream of data so we have a constant stream of quotes coming into our system and then where the stake comes in is we're going to be windowing this information and then trying to make some sort of calculation and for this case what we're going to do is we're going to calculate in this window of 10 seconds what is the max price we've seen what is the min price we've seen and is this a upwards or downwards percent increase or decrease and whether or not that breaks some sort of threshold so the first model we're going to kind of work with is well all the models for today are going to be pulling data from kafka so we've got partition data in kafka the data is partitioned by the company so you know everything for amazon will be on one topic or one partition and then everything for you know gm and apple will potentially be on on other ones but we're partitioning by the company and then we have an akka stream instance that is reading this data um for this example we're running it locally so we've just got one node that's running all of this so the first example stay full map king kat when we're we're building akka streams we you you can see these boxes here and each box is kind of a different aka streams operator and stateful mapkincat is a very cool tool for being able to um pull in a few messages as state and then potentially emit one or more messages downstate when you're downstream when you're done processing that message so what we're going to do is we've got a stream of all sorts of different companies coming into our application we are going to group by the company that way we end up with a sub stream each substream representing only one company and then we're going to window that information in our stateful map concat do our calculations to figure out what the percent increase or decrease is and then push that data downstream merge these sub streams back together so we end up with kind of one outlet and then we'll be able to send our alerts to our users whether or not their their securities are increasing or decreasing in price so now for the fun part let's write some code so jumping over into intellij boom so i really wanted this session to be like completely from scratch writing absolutely everything sadly with 30 minutes i don't think we're going to be able to do all of it so i did a little templating here but we'll walk through each of these lines so this first line here system is just initializing this actor system that we use for initializing actors or active streams we've got our execution context for for futures we've got our logger and then for this example we're not going to be using these two as much but it's where clustering is initially initializing akka clustering so that we can actually have multiple nodes that join this application and are able to collaborate and communicate and then sharding will specifically i'll talk about in the next example more and then i've got this kafka helper that basically will just generate our sources reading the data from kafka and writing the offsets back into kafka and we can actually just look at that real quick so this is our source we are creating this aka streams consumer we're using a committable source and then right here we're actually deserializing the messages uh from just a string or a json serialization into our quote object and then the very beginning of our stream we're going to be pro processing this tuple which is just a our quote message object followed by a committable offset that will be committed back into kafka when we're done processing our message so let's get started writing some code so here we are this this is just some setup things and this kafka runner is just feeding the data into kafka so we're generating a bunch of simulated data right now that we're then going to read out of kafka and then this map can cat stream is our stream that we're our stateful stream that we're actually going to write so step one we need to take our stream and we need to group by we want to make sure that we are each of our stateful map kincat's substreams is only getting a stream of one company at a time so the first thing we do is we are going to use this very simple group by so different from some of your well some other frameworks people think of group buys as kind of bringing things together which technically it is doing but in this case it's actually creating a sub stream for each individual uh company that we're going to partition our data by so here i like to think of group by as more like partitioned by and so with our group by we need a max number of sub streams which we're going to just going to set to 10 and then we're going to set up our kind of partition [Music] key so we're going to go all the way down to this company symbol so for tesla it's tsla i think for microsoft msft and so that's how we're actually going to partition this first data so now with this we've ended up with a sub stream representing each company that we're going to be working with next we're going to actually implement this stateful map kincat that i was talking about so stateful map concat we're going to use our curly braces and we're going to set up the state that we actually want to um to collect so i'm going to call it here uh quote window quote window so this quote window very simple simple is just going to be a list empty of quotes when we get started so this is the state that we're going to be adding to each message that comes into this quote window will be applied to this quote window and then when this each message is completed basically we have a input of quote window we can apply um new messages to it remove messages from it and then um on the next message that comes in that that new quote window state will be will be passed through so now what we have to do is we have to set this up so that we have our kafka message that is coming into this aka streams operator and just to make things simple i will name these quote i will split out our tuple to make things clear so we've got our quote and then we've got our uh kafka offset and with these two things i actually have a little helper function that i wrote that will do some work on this quote window but our new quote window is going to equal there's a in our quote companion object we've got this function called update quote window so we're going to apply the new quote as well as the old quote window and then we're going to use a duration of just 10 seconds for this example oh sorry this needs to be a var so now we've got our every time a message comes in we're going to add a new quote to our quote window and then this update quote window will actually drop anything that is not within our 10 second interval so that's all that's happening is here in here is it's just taking the most recent quote removing 10 seconds from it and then dropping anything that's outside of that window and then next we're going to do our calculate our percentage change another helper function just to calculate the percent change in the window and then we just want to pass these messages forward and then we will pass forward a tuple that will have our percent change message as well as the kafka offset potentially we could pass more messages like a a bunch of messages forward if we wanted to but for this case we we don't need to and then last we will merge our substreams so now we have one outlet that's just getting these percent change messages how much did something change by and then we're just going to um for this case take that message in and then do something like if oh there we go message dot percent change message percentage if that's greater than one [Music] threshold alert and then the last thing we need to do in this map is we need to pass forward our uh kafka offset so that we know that we can commit that back into kafka and it no longer needs to we no longer need to ever process that message again so with all of this i will come out of presentation mode and we will run this real quick so while that's oh there we go oh of course i forgot to uh fix some of these so i will just comment them out for now and we'll try that again real quick so there we go this might be a little small um but boom we are consuming messages and not everyone is causing a threshold alert but every once in a while we are getting these threshold alerts [Music] so with that i will stop this we will do a quick back to slide so that we can go to this next model and one more quick example that i think we can fit in before we run out of time is our uh trying to introduce actually a sharded actor into this pattern and i'm going to throw one twist into it also so what we're gonna do here is we've got messages that are coming through the system and akka streams has this really nice operator called map async and map async will make sure that something that a message that comes in will result in a future and that future will be completed before it's passed downstream so that's what happens here and we can actually send a message to our actor whether it's on our local node or a different node that actor can process the message and return an acknowledgement message and only that that message will only move forward after we get our acknowledgement back so we're going to collect state in this actor and actually do our windowing there but to throw another twist into it that i uh a tool that i really like to use in well it's just a very convenient tool to use in aka streams so it's very easy to pass messages from streams and actors but people can sometimes get confused on well if i just have an actor and i have a running stream somewhere where how can i put a message from that actor back into the front of the stream there's actually a tool called um or a source type called source.q that will allow you to drop messages into the front of a pocket stream and so that's what we're gonna do we're gonna have messages coming in going to an actor windowing and aggregate and then that actor when it hits this threshold will offer a message into the uh the downstream and so jumping into that encoding this one real quick this is that very cool uh stream operator i was telling you about source.q so this is going to create a stream that will um it's a normal aka stream but the result of creating this stream is this thing called a bounded source queue the bounded source queue you can just put objects into like a queue messages and the source will process them and so this actor is actually going to have a reference to that queue and be able to just put things asynchronously into that queue and so we're initializing our active actor sharded system right here we have our sharded actor that i've already built out the security security metrics entity this is going to accept messages of type quote and then it's going to return an acknowledgement that says i've processed this message and then in its internal state it's going to generate this window for us and anytime it has a threshold that it um you know every time any time that the window hits that threshold it's gonna um send a message to this this cue we talked about and sorry i'll also mention each company gets its own actor so each company is gonna have its own actor that's started up by actor charting and so each company will have its own window and so those actors could potentially do more than just windowing this basic information so now we're gonna do our map async we're gonna open up our kafka message again and let's try this again compiler is not picking that guy up for some reason but not important kafka message oh of course i forgot the parallelism so in our map asyncs you can have different amounts of parallelism so how many futures do you want outstanding at any given point so that's what we have there so now we have our kafka message and we've got our quote and then our second one is our kafka offset and here we're just going to get a reference to our uh we'll call it the quote actor we need to tell it specifically which actor we want to send messages to and then we need to and then which specific actor we want to get so for this case we're getting the the actor that has this symbol and so now this is kind of like a a mailbox like a specific place that we're going to send a message to and as a message comes in for tesla we're gonna go get the tesla actor send a message to it and if i were to have multiple nodes in this application these actors could be spread across all you know 10 20 30 nodes and we would be able to still send messages across the network and then get our acknowledgements back in this exact same code and then last we're going to send we're actually going to send the message to the actor using something called an ask so an ask is going to expect a specific message type back so i knew i wrote a couple things real quick right here i'll just add a quick duration how long do we want to wait for this message to come back and talk through this so right here we've got our uh quote actor we're expecting back a uh ack message and so um well yeah we're expecting back an acknowledgement and then we're going to send this message this quote message to the actor and then we're going to kind of sign it and say well here's how you respond back to me and that's what this reference is right here it basically is once you're done processing this send a message of type ack back to this reference and that's how sending a message to an actor works it's very you know very simple and easy to do and actors are great ways to build applications and so that returns a a specific future and we actually down here in this kafka sync we want to give it back the um the kafka offset and so now after this actor can successfully finishes processing this message we're just going to send that offset back to kafka and then we'll be done and so that is all we had to do to send a message to an actor and we will run this one real quick and then do a quick quick three minute wrap up with this one other pattern that i absolutely love but sadly i'm not going to have time to demo today so yep this is our second example and now we're actually sending these messages to the actor the actor is printing out the percentage change and then the actor is calculating the percentage and then it's also saying hey we were able to write it successfully into that source.q that i mentioned and then this is actually the message that was in that source.q so we were successfully able to design this pattern right here so map async sending messages to actors actors sending messages back to uh to um aka streams and so jumping forward into our last kind of model that i i think people should be aware of in akka streams so we had this example of we're reading a message out of kafka that's partitioned by symbol and that message is being read out by node one but the actor for gm is always on node two right now so kafka and akka are sharded in two different ways and that means that every message for gm is always going to need to do a full two network hops in order to be processed successfully there's actually a really great way in built-in akka specifically for kafka to actually co-locate or to to use the kafka sharding to also shard the akka actors so we can basically guarantee that gm will always be on the same node as the one that is assigned to reading that partition and this is called uh kafka cluster sharding and so um well there's a new tool called well i guess it's new from like a year ago but it's a it's a very cool tool it's called the external shard allocation so basically akka can rely on external systems to say how should i be internally sharding my data and the the documentation for this is right here and it makes sure that with these nodes clustered we're always going to have gm reading off onto the same node as that the actor is located and if this if kafka ever assigns this partition to be read by node 2 akko will actually take that q and and also move the the shark so uh this example is in my examples so um if people want to have these to people want to take a look at these examples look at all the codes see how it all works i recommend going to um i've got it pinned on my nolan grace github so i'm nolan grace on github and this is the uh the repo so it's got all the built out examples in it including that final example and for like my uh um you know favorite person in the entire akka streams community for the the content that he generates is colin breck an engineer at tesla his blog is absolutely exceptional it's helped me learn so many things and i know i refer to a lot of people who are also learning aka streams so i highly recommend taking a look at colin breck's blog and with that thank you so much for for having me i really appreciate it and i'm looking forward to hearing from people in the q a you