SBTB 2023: Evan Chan, Lessons Learned: Porting a Streaming Data Pipeline from Scala to Rust.
[Music] so my name is Evan I'm a principal engineer at uh at conviva and uh I'm really excited to be and thankful to be invited back here to uh speak at scale by Bay it's really nice to have uh in person uh conferences again so um good morning everyone and you know let's uh let's get started so uh a little bit of uh word about uh my employer um so uh conva what we do is that we do streaming realtime user experien analytics what does this mean so a lot of our customers are like some of the biggest uh streaming uh companies out there like uh you could be like HBO Disney plus and basically what what we tell our clients are like what uh trouble people have uh when they uh in this case is watch video but you can imagine like you know you know surfing web it's basically tell them like uh what is you know what is happening with users all over the world and um we have massive realtime streaming analytics pipelines it does uh five trillion events processed per day a lot of people talk about billions as being a lot but we're at couple orders of magnitude beyond that so so this is like a really massive Pipeline and we start started with some custom uh Java code and went through a couple iterations and today we have hundreds of PODS serving uh some uh custom AA streaming stuff so it's pretty cool large scale work but today I'm going to be talking to you about our experience uh porting stuff uh to rust and what what we see is that the data world is going native why is that well we can probably think uh python and a rise of AI and ml to you know data science uh to to sum of this it has become really big and like even all the existing uh jvm ecosystems like spark have had to adapt and be able to run like cpython workloads we can probably thank the end of mors law for some of it being that CPU speeds are not increasing what we see is that we have an increasing number of GPU cord and things like that so being able to take advantage of those is becoming really important maybe we can thank Some Cloud stuff uh so why rust in particular well it is safe fast is in like C C++ kind of fast and it has uh good high level abstractions has uh support for functional data patterns or pattern matching and so forth and we see that there's a lot of uh projects that are starting to come out that are starting to like be really interesting like that people are starting to use and um we can map those from the old jbm Frameworks like uh spark And Hive well now we have things like data Fusion how many people have heard of data Fusion like almost nobody so um so data Fusion is a uh modern SQL uh executor and Bist does the distributor version you can think of it as like a new spark sequel there's some other interesting projects on this list there's a bunch of new streaming Frameworks Rising wave or Royal materialized they all do uh streaming SQL or streaming materialized views and even in traditional like uh like for Kafka there's a bunch of new folks that are coming out I didn't list here like uh red panda is and the new one F view is a rust based persistent queue that has uh stream processing so there's a lot of new stuff that's coming out and so it's it's a very very exciting area even in terms of databases there's uh R based databased projects that are coming out even in terms of of graph um terminal DB is a really neat one that not only does graph but does versioning as well so there's a huge amount of momentum and I think there's some other there's some other uh R talks that I saw on the schedule and maybe you know if I went last instead of first what this someone else on my talk so a little bit about our architecture what so what we have is that we have a bunch of sensors that gather data and send it to a massive array of gateways on our back end this goes to I mean this is massively simplified of course but uh this goes into Kafka then it goes into a big streaming data Pipeline and then the pipeline computes metrics which goes into database and gets served to users so nothing really like at least at this picture level it's doesn't look that uh complicated so what what we are uh porting uh to rusts first is this this big box a stream height line wanted to take that and uh one both increased efficiency uh because at our scale at you know trillions of events a day this becomes a you know this becomes pretty important and second we want to upgrade the pipeline to be more uh flexible so that it could be more um dynamically programmable and we'll talk about that in a little bit uh second uh we were having some issues with our uh with our gateways we wanted to increase that make that more efficient and fix a few things so so that's uh so that got upgraded a couple times and and lastly uh there's some work uh with with sensors you have a lot of different small code bases because you could have something that's written in in Swift for iOS something that's written you know in Android Java and you know cc++ a lot of different uh types of framework so um it's kind of nice to be able to try to unify some of that possible so this is what our this is what our journey uh looks like uh to Russ so for the data pipeline we started out with a a hackathon about a year ago and this is just to see hey you know let's let's try this you know new thing out let's see what the ecosystem is like what is it like to develop in it what are some libraries so and then uh we' proceeded to uh we were porting a data pipeline to a new platform initially a scolar prototype was written uh then we spent a month and a half doing a rust Port uh of the new scolar pipeline after a few months we got more comfortable and brought on a lot more people and and that's this bigger bar you know that's us you know bring on more people and and and that's uh kind of our current phase we bringing on a lot of new people now they've gone used to it and and now we have up to 20 25 people across a couple teams uh working on this NE Cod base which is which is pretty exciting there was also an effort to Port the Gateway and the Gateway was ported from jvm code base into go first and then it was ported from go to rust because they and we saw some benefits there so that that's pretty interesting so a little bit about this um rust hackathon basically what we did was we took a really simple part of the data pipeline that just reads data from Kafka dezes and does some really lightweight processing so this this was really an islebound workload it was just to get a feel for hey what is it like to work with uh what is it like to work with async dress code what is it like to work with you know some basic basic really basic data processing oh sorry I realized um I'm speaking their mic um and we were able to get some good good gains out of that uh kind of like two and a half times more CPU um the the memory increase was was more impressive that was like you know 12 times less and um so so that was promising enough to try new things although at that time it wasn't clear if if it was worth the effort to you know to you know like how much effort or time we would be able to put into it or like if it was you know so so so but it was it was the uh the performance results were were impressive right and uh we also learned some things about uh active Frameworks and rusts um we also have some results from porting the Gateway this is pretty interesting the graphs here compare not jvm but uh but golang uh so the the team initially ported uh jbm code uh to go which seems like a you know pretty good choice it's a you know High productivity and it was pretty interesting they also found that even compared to go like rust was able to uh half the CPU consumption and also decreased the memory used by an order of magnitude so that that's that's a really interesting finding so when we do a port we want to ask ourselves uh some key questions I call these the four piece so you can like Take Along with me here for a sec uh one is people like how do we bring people on board another one is uh performance a third one is what I call patterns is is like how do you adopt existing you know what coding patterns you're used to into a new world and the fourth one is is projects which is more about tooling um debugging and so forth although some yeah so let's Dive Right In so so people how do we bring the Vel opers on board and in the end this is like one of the most key questions right it's that if we can't bring the people on board then it's like really really difficult so what what we did was that we did a phased bring up so the initial phase was that we had a very small team of three to five uh very senior folks with at least you know one or two people that was more experienced uh in Rust I guess that person was me um so and and we did a the uh core part of the port we ported a big majority of the code and uh we had a second phase few months later where we brought in a lot more folks you know the 10 to 15 folks that have mixed levels of experience from you know more Junior to more senior folks um and uh pretty much none of them had rust experience and and what what we did there was that because we did this in a second phase we're able to give them more time you know to come up to speed you know to kind of like I think on average uh probably took you know two to three weeks we just gave them some time just to learn just to read but at the same time the the benefit is that we we had a team that already came up to speed and so we were there to help you know answer questions and and help along and and to basically guide them to do hey you know like start here and this kind of thing um and you can see it uh there's a lot of challenges when you when you learn a new language uh lifetimes uh compiler errors how to do you know this or that we'll get into some of this later uh ownership is a big question and I think the tools that we found uh very helpful the uh I mean I think some of it has to do with good uh project organization we split the project into a a streaming a synchronous part and and and an synchronous part and and so it's a little bit easier on ramp you know some folks didn't have to learn certain parts and having good documentation and other things like that is is always like always really helpful but uh specifically to rust there is a really helpful rust book A lot of people found chat GPT actually you know really useful just to help them get started it's not always the most accurate might give you like kind of wrong answers is but it's kind of like helps you uh go get over humburg get started you know and um another thing that uh that we found helpful that started doing a couple weeks ago was to have uh office hours this is just like a say one hour a week kind of thing just to um offer folks hey this is a time when you can ask questions and uh found a lot of folks you know coming by to you know just to chat about what difficult things they're having and I think it's you know you can really replace spending a lot of time on things like uh reviews just just to help uh people with like a lot of detailed things uh and now we come to one of my favorite topics uh performance talk about data structure and so forth one of the quotes from one of our Engineers was that they enjoyed the fact that the default routes in Rust is uh is for performance and and that um if you want to choose a path that is less performant usually you need to type in like extra keyword like um know box or you know something okay so um take these numbers with a grain of salt but basically uh I'm going to be talking a bit about porting our main data Pipeline and um neither of neither of these I say take a grain of salt because neither of these have been uh like heavily productionize or anything like that uh like they were at the time the reported was more early stage but it's the same architect Ure the same inputs and outputs the um the Scola version wasn't uh designed uh for uh for Speed in Rust when we reported it because in Rust one thing is that you pretty much you have to spend a lot of time thinking about data structures so when we ported it we purposely chose to use uh some uh streamlined data structures to minimize allocations so we we got a pretty big uh boost comparing the two like a little more than order of magnitude and so it's it's been very promising so far so so what this data pipeline does is like a whole bunch of um stream and complex event processing which is very stateful and and and the goal of a pipeline of our new pipeline is that we want to be able to let um let people program a directed graph of data operators so you can imagine imagine that data comes in on the left side and we do and we extract some data out we do some transforms you know we bring it back together you know so forth so you should be able to rearrange these and and these data operators connect uh connect time ordered data and we call a breakup of the times spans and so basically an Executor connects data from Operator Operator kind of like an airflow tag you but on a much smaller scale so this is more like maybe like a database query compiler so forth so and the data is broken up into uh multiple spans where each span would have a Time range and some data now if I was to so I'm going to spend a little bit of time on this diagram well this what the left and right s side represent are the different ways that data structures look in uh Scala or jvm versus rests so in Scala we start out with uh something we call timeline which is a bunch of spans or bunch of data organized by time and this is typically something like a seek or buffer iterator or something like that and underneath you would have arrays you know right so uh the seek is typically a separate object the in jvm the the array is its own object so the seek points to the array right and then the array itself you know because the spans underneath it's an array of spans underneath they are not Primitives so again the array would point to more objects each span has its own objects and uh if that span has a somewhat complex type underneath like it has a set of events that would be its own you know again so uh object so again you have like like three or four or five layers of object wrapping in in a rust side the r side is equivalent um but due to a couple techniques we can in most cases we can actually have the entire data structure sitting in line on the stack and that's why you see they are wrapped in boxes instead of arrows and the way that this works is that uh one thing is that having strs helps you inline things so because uh a stru is fields of a known size that are all in line uh they can sit one after another even in something like a collection like a list uh the other thing that we do is that we use data structure that I'll get into a little bit later that allows us to for a small number of elements to inline them on the stack um what this looks like is um the at the top level we have Anam and then underneath we have uh something called a small V that holds you know a ve is a list this holds a list of it items and then underneath that um the list of items are spans which are strs and they can have other things inside and so basically if the number of spans and the number of events are small then they can all fit inside the stock um one so I'll just go a little bit into uh it's not really magic this is something called a small ve and the way this works is this is an enum which is like a suum type and it can be either uh say something that fits on a stack or something that sits on a heap so it's one of those two and then the class manages that so then basically the the the stack one is of a limited capacity n here it holds up to n items and if you exceed that then basically this thing uh becomes a a heap variety so it's it's pretty uh it's pretty simple but pretty effective way of avoiding allocations and there's a whole bunch of other crates that are like that so but this is one of the really neat things you can do is that uh you can control your layout really pretty sely and and uh in RS another important topic is about static versus Dynamic dispatch um in Russ you kind a very precise control over where you want or what kind of dispatch you want but the dynamic is not by default so let's say that you want to work with different kinds of structs that Implement some trait and so this is a pretty common pattern so there's a couple different approaches to to the dispatch the fastest and maybe the easiest is this big word called monomorph isation which is basically that in your function signature you have this trait in this case we have this o uh this type O which can something that implements operator and the compiler will basically whatever you pass at compile time the compiler will make a version of this function execute op for that specific type that you put in right so this works well when you know the types in advance then a compiler can uh put out the different can generate code for each different type that is needed and it's it's direct and static and very very fast right uh sometimes that's not possible in our case because like the uh programmable dag of operators is dynamic so what we do a lot of times is that we use enums because this is a some type that uh has has a type uh there at runtime and um I will go into that a little bit so this is another technique uh and in for dynamic dispatch it's s if you want to go that route there are even two variations uh there's one that does not require boxing which is neat for avoiding allocations but it still requires a vtable lookout and finally you can do function wrapping so a little bit about uh a num dispatch what this is is this is a macro where in this case we have uh two structs the linear knob and the logarithmic knob right and they both implement this trait called knob control and what en num dispatch lets you do is that I can have an En num called knob which has the variance and enom dispatch will generate uh trait implementations and be so that at one time it can depending on which you know variant it is it can pass it to it it's the inner types that already Implement that stuff and and this is still uh significantly faster than doing a dynamic dispatch vtable lookup uh what one reason is because it's especially if the code is small uh the compiler can actually inline all the different variants and so you're only paying the cost of the branch lookup instead of a dynamic jump instruction another thing you can do is this function wrapping and so what this is is that you can just have this function in line in your structs there's two variants in in R there's a lowcase FN which is a type and there's an uppercase FN which is a trait so th those two are different depending on uh your needs uh the uppercase one is uh can be like closures and things like that lowercase FN is a constant it's basically this a pure function you can't have um any uh you can't close over anything I want to talk about patterns a little bit so uh there's different paradigms uh the the predominant async Paradigm in Rust is that there's an async of weight there's a native async of weight Paradigm that that is often used this is basically just a synchronous function cause where basically you can define a function and you say this is an async function and Russ basically turns it into a future and thus the magic underneath um there's also Futures but uh it's it's a lot more economic to just use a sync functions and finally there's other things like actors and async streams uh and reactive as well although uh none of these libraries are quite as well developed as in the the Scala jvm world we ended up we went through some iterations and trying out different actor libraries and so forth we found out uh one of the dominant ones doesn't play well with the dominant async library in in Russ which is called Tokyo and so we ended up writing our own little Library uh which sits on top of Tokyo because Tokyo has exous tasks and so forth so it is is a really lightweight layer that sits on top and gives you an actor abstraction with the ability to send a message and you know and do an ask and so so so on um the there's you know there's a bunch of other interesting patterns which um I think the most interesting is that there are no exceptions in rusts you use functional of result pattern and um the let's see and there's no inheritance so you just have to use composition which is personally fine for me so there's a things to learn uh I'm going to so I'm going to quickly go into the projects and stuff for quick sec uh one of the quotes from one of our Engineers was that our cargo was the best build to ever I I do find it like really easy to use uh because you just write uh tommo and I don't like almost I pretty much never have to worry about uh actual build code in terms of um like say rust code in order to build more rust code which is personally is really nice I think that's one of the best things about the r ecosystem because like I don't really want to be worried about having to write code when I need something to build so yeah it it works really really well and there's a bunch of uh tooling and other things like that one thing um I would give a shout out to is that uh vs code has a really nice it's called Life share it's like a pair programming tool that you can do like especially for remote pair programming each person gets a cursor and like it works really well and uh there's a yeah there's a bunch of other uh there's equivalent tooling for almost almost everything um property testing and so forth uh there's a bunch of links which I will share in the slides uh later that you can see if you want resources for looking up projects and that kind of stuff and finally like I will say like a little bit what do we miss from schola I think there's a few libraries that are a little bit more mature like hdfs support just a few like obvious things like that and I think finally like for tiway it's been a long journey but I think that uh it was uh quite worth it for us has uh you know very promising performance profile as well as uh safety and I think that uh structuring your project and structuring how the project is run can go a long ways towards uh folks coming on board and the other thing is to focus on data structure design early um and finally um uh we are hiring so feel free to come and talk to me um thank you very much um and feel free to reach out to me on different [Applause] platforms