scala.bythebay.io: Tim Delisle, Bryan Russett, I built an actor system in Rust
Recording: scala.bythebay.io: Tim Delisle, Bryan Russett, I built an actor system in Rust
I you um so we kind of want to start off with prefacing this that we did not think of rough stock would get into a skull by the bay kind of conference but it did so and in no way is this kind of in introduction to rust or an introduction to some of the actor systems that we built our failed to build it's much more about kind of the beauty of modern low-level languages and how they can help teams be more efficient solve problems more in a more performant manner as well as kind of what it means in practice for for companies and teams of developers out there the other piece that we want to preface this is that Brian found out he was giving this second piece second part of this talk about 20 minutes ago so we'll see on the range of disaster to great where we exactly stand but to start off with just a little bit about myself and about brian is Brian as well so I'm the CEO and co-founder of data log where we use AI to automate data preparation and we will talk a little bit about the things that we do in the challenges that we face but at the same time I'm also a health tech master's student at Cornell Tech where I study a mainly weak AI and distributed computing and also this is kind of my preferred stack it doesn't tell you much about me other than I'm kind of jack of all trades but a master of none and I just really like to find interesting tools that get that helped me solve problems and interesting problems out there in the world brian is the CTO and co-founder of data log he was previously a software engineer at class pass interestingly he has an MBA and his preferred stack it kind of falls into the infrastructure as code kind of environment and and it's he does some magical things for the engineering team at datalog so let's kind of kick this off and talk about how I built an actor system in Russ kinda and what the lesson learned what the lessons learned there were and as contacts for for the talk not sure if any of you guys know who this guy is at the right here but that's Rajat Manohar and he's one of the Guru's of asynchronous computing he designs asynchronous chips and my buddies and I just happened to take one of his classes on distributed computing last semester at Cornell Tech and Rajat teaches the class by teaching distributed computing using sees open mpi library that's a message passing interface library you guys might be familiar with the paradigm but it's kind of the de facto standard for high-performance computing at least in the academic settings and our goal was to kind of create a cleaner easier to use better abstracted higher performance on average message passing interface then sees library this thing is like decades old so it was kind of a big undertaking and we wanted to do it in r us because russ is an awesome modern programming language that's low level but also has some of the great abstractions that were used to pattern matching zero cost abstractions and things like that and we wanted to do so in about three months while being full-time students and founders so ultimately we were [ __ ] insane and so he went out and we built we built crow which was an actor system that we implemented in rust and then all throughout the semester we were sorry we were using that that system that we built to solve interesting problems anything from finding the dividers of the large integer to kind of doing MapReduce and those kinds of typical problems that one would generally face in a distributed computing class so I want to kind of show you guys a quick little example for anyone who's familiar with rust it's going to look quite dirty you could have built something that was much cleaner had better levels of abstractions like for example we we have these weird bracket e things here we're like for all the Scala folks this looks super dirty but we borrow kind of immutable construct which is our system at actors to it then we can pull those out of the system and then just send messages and kind of in a similar fashion that you would just anaka or something like that you we define these actors a little bit higher in the code but not to dig into details about kind of the rust implementation but here are some of the accomplishments so in this task we were asked to the like find the dividers of a very large integer and what we did throughout the entire semester was pin in the intermediate to advanced c developer versus our team we were complete rookies and ross like had never used it before we had just built this actor system and we had very very little bandwidth to accomplish these tasks and these are some of the performance metrics that we accomplished and really the lesson learned here it's not that we're twice as fast as a see implementation because a perfect c implementation would have probably beat us but that we re we are able to accomplish these tasks on kind of similar resources solving the same problems and building the same similar programs and do so much more efficiently and in this in this graph this was after like a week's worth of refactoring by the sea programmer and he was still only he was sold twice as slow as we were in the first implementation of his program we were six times as fast so what that kind of speaks to is how modern programming languages and kind of their constructs help us build better programs and deploy those programs more quickly so we did really did an outperform see in any way we just show that really ease of use in modern languages matter so as you guys are kind of building out your teams and you're thinking about solving really interesting engineering problems and might need some sort of low level performance then rust is a great alternative to see just because your teams are probably going to get things done much more quickly and in a more performant manner so kind of that boils down to like to drive home the point I guess it's like great Modern Languages make programs more efficient and performant not by just optimizing bits here and there but really by allowing us to reason more effectively about the programs that we write by preventing common mistakes so I'm not sure if you guys have used rust in the past but there's some awesome errors that pop up when you do something stupid Scala's new dot 12 version is also like moving towards these really useful errors and they allow you to easily leverage all the computing resources that you have on a machine of this beautiful Scala example where this is just like trivial to start leveraging all the cores on a machine it's almost as trivial and rust so now you guys might have remembered what the the original title was how we built an actor system in rust but then actually built a company Annika and the reason we did that we decided to make the switch was there's a few wishlist items that you truly need to build a high-quality actor system and those when we were implementing our version of an actor system just didn't really exist like zero cost few shirt futures async I oh that was a library out there it was just tremendously hard to use and very opaque in terms of how it was constructed and we just needed kind of more mature networking libraries so that once we truly started to think about not just scaling on a single machine but then moving to entire clusters we needed kind of these items to to be checked off of the list of things moving forward and rust they just weren't there so that's when we decided to kind of kill crow for now and hopefully as some of these things start maturing in the rust ecosystem will be able to to move some of our high high performance computing needs to back to rust and off the JVM so I'm going to pass it off to Brian now we're going to do an awesome hand off of this mike mike this uh this camera mic so yeah thanks mm-hmm so yeah as Tim said I found out pretty recently I was going to be coming up in doing the second half of stock and I guess the idea of telling everyone here why skull is great when you're all here for a conference on Scala it's a bit silly so if we can just share a little bit of insight into how as a small kind of scrappy startup we find that scholar works really well for us so to mention before but we build systems at leverage AI to automate and scale data processing so data classification ontology mapping deduplication basically all the messy dirty parts of data science we're trying to abstract an automated way so kind of the the scope of things are doing with our first product project was processing 30 terabytes of data for our first client and unlike I guess building a system where you're dealing with streaming data and have the same kind of data coming in every day every time we sit down with a new data source we're dealing with us all the time it's always new and unknown so we wanted to work with the language that had really good existing tried and true systems for data processing and so Scala being a much more mature ecosystem than Russ obviously we can kind of build on the backs of a lot of mature data processing frameworks like spark for example and we're also doing with a lot of crazy weird data formats so just typical stuff like you know JSON CSV and then customers giving us sequel dumps dealing with 807 healthcare data formats so it could have to be ready for everything and we're also highly resource constrained we're a small team we're dealing with a lot of data we've had some help from AWS as far as their infrastructure costs but we wanted to be able to take advantage of a language that was built for distributed in concurrent frameworks so I guess I already talked about scalability reasonability and correctness so for us when we're kind of building a startup in hiring people quickly and try and sort of build these systems very fast obviously some code review practices go the window you're trying to get things done so what's nice is when you when you build something in Scala I think more so than a lot of other languages that are out there right now much more so than Python somebody else sitting down to the code can can read through it and understand it much more and i like that scala encourages but doesn't enforce the use of pure functions again when somebody kind of write something that works in a functional manner you don't have to think too much about that you can kind of just walk through and understand what the steps are doing and stream ability we've got a lot of benefit out of vaca streams for doing some large data processing again being resource constrained we don't want to host and build out all her own intersection we want to leverage existing infrastructures of our clients and so that means as much as possible pulling data in doing our operations and giving it back and so again acha streams combined athaki HP interfaces has been really really helpful so the the transition from russ to scale actually kind of began before I came on board and I like to joke that Tim shows Scala once but I've had to choose it a few times so as somebody that's been coding in Scala for five or six months I I've had some fits and starts I wasn't always completely happy with everything but I think what really solidified it for us it's just that the end result of Scala code by awesome bike you that we have working around the country because we're distributed team at the moment the end result is just really solid and resilient so better scalable faster more productive again the the community and quality of open source code for Scala is just astounding we were on Google Cloud Google cloud platform for the summer and trying to figure out how we're going to interact with all of their services and it just so happened at spotify just transition to google cloud and had built all this really amazing Scala code it's you over to kind of like build on that and leverage a lot of existing quality code and yeah like I said before everything that we put into production is been rock solid so guess that that's it for us we have about seven minutes for questions and answers I just there are questions cool sure yeah any questions that's a good question I think it's more the other way around where there's a lot of scholar features i would like to see in russ so the i guess i have to double mike this right I'm sir the really in terms of inheritance Russ isn't all that great so it's very hard to real Everidge a lot of the code that you're you're writing currently so currently I would say the best uses for Russ are when you have a very kind of small and scope problem that just needs to squeeze every ounce of performance from your machine so I'd like to see it grow a little bit more in terms of how you do inheritance between the different strokes how you start implementing better traits across different kind of code modules that you're building out and Russ just so that the code could be a little bit more reusable and dry but like in terms of just things that we'd like to see is just the performance is outstanding and rust so I think what could be could be pretty cool is to see a better kind of Java interface when you're writing code Russ code so that you can pull it into a Java library and kind of squeeze the ounces of performance that you might need for first go problem right now the interfaces for Java or I think kind of in the works but not very robust know there's great Python ones right now any other kind of interesting questions we're also happy to geek out about a whole bunch of other things as well we've got to you will go here and then go back to left yeah yeah so so it's much more about actors for layers of processing than it is for squeezing ounces of performance so Z we kind of had to do go through a soul search of what was important for us it was it to start being able to build very very large systems that distribute easily or was it to build extremely performance systems that are hard to distribute and so we we moved to scala just for the different layers of processing that we do like for example when a product when a data set comes in we build a series of matrices as to like how it was constructed what types of data are within it are neural networks then process each individual data point and tell you like this is the compound feel that I have and so all of that needs to happen simultaneously and scala allows us to solve those problems very efficiently whereas rust we would have kind of had to build out this entire infrastructure so in space you have you only lose CD you can be hard maybe use like single instruction or data safety instructions or a new instruction set which many of the gym Geralyn support as well yeah not yet um not yet and for most practical uses just the raw rust implementations worked there are efficiency bitings like from russ to see and vice versa so it was kind of a great infrastructure to start thinking about some of the smaller problems that would need to be optimized in the future but kind of being a scrappy startup and if the clients code just like runs for millisecond for my little eye cancer or a couple nanoseconds right now it just doesn't matter as we're thinking about scale russ is always in the back of our mind to start eliminating some processing bottlenecks so and yeah so once we start lot and eating and those ounces of performance will need them yes yeah so definitely the most useful thing for us has been kind of onboarding new people to scala and being able to package their code within an actor it kind of allows us to think very sane Lee in terms of the hierarchical layers of data processing that need to happen for us and so if you're like a brand new intern and you just need to get up and running I can give you a very small subset of a problem that honestly if you start flocking up I really don't care because I could just read very very efficiently like refactor the little module that you work we're working on and so kind of how we we early on started architecting of some of our code is we have hierarchy of different actors that take care of work that is kind of broader to more more granular so we have different actors that'll talk to individual endpoints and then relay some information to higher-level actors that might be doing an inline join for you or might be doing some sort of higher level data processing layer and then eventually like send it back to the client through kind of our highest level actors so that kind of hierarchical system has worked out really really well for us and it's allowed us to onboard many interns who kind of just work on a small subset of problems so and there's another cool part to that as well so reasoning kuber Nettie's for infrastructure and we've had a lot of success with it one of the challenges I guess or one of the canaries lacks the moment is really quality load balancing so the fact that we are using aquifer the core of our back-end systems at a higher level like it will be the request will come in just round-robin concur benetti's but then we can use aqua clustering sort of much more low-level CPU metrics and then at the second level down then we'll actually take advantage of the Aqua clustering and send messages around two machines that are less resource constrained that crew benetti's isn't really aware of at this point so that's been a big help for us yeah so we're doing sort of an auto building aqua cluster in kumaradas so kind of run it in different implementations but one kind of cool example is like if you run your Aaka application as a demon set and then set up an auto scaling group like as like the overall demands of some increase like you more nodes will be added and you now have a new aqua node we're using zookeeper so basically every node that comes online or aqua cluster reports in to do keeper and then everyone else is aware of it joins the join see a cluster and then resources coming in or going to very quickly get floated in at the second level of our actor system based on sort of the gossip protocol on all the great great work that's gone into acha to make that work so that's been a big big help for us you