Scaling out a Rails app with Finagle

scala.bythebay.io: Stephanie Bian, Scaling out a Rails app with Finagle

Recording: scala.bythebay.io: Stephanie Bian, Scaling out a Rails app with Finagle

yeah so my name is Stephanie I work at a startup called brigade I'm an engineer there and um actually I used to work at Twitter so it's really oh really yeah sorry you hear me no okay great so I work at a small start-up called Brigade and I actually worked used to work here at Twitter for a couple years before I went there and today I'm going to be talking about how we use Scala and finagle to scale out our rails application at Brigade so for some background and I promise that this will be a pretty short brigade of social network for voters and so what we do is we try to provide a platform where users can have discussions about politics and become more informed citizens overall our application was originally built using Ruby on Rails and that's still where the majority of our functionality is still alive so obviously rails has a lot going for it it's it allows you to do really quick developments and it provides lots of really nice to use libraries like active record which is Justin or m4 for rails and it also has really great community but there are some drawbacks namely performance is just not quite as good as the JVM there's really a lack of concurrency in rails and other libraries that it supports and it of course uses Ruby which is a dynamically type language I probably don't need to explain to a bunch of Scala people why using a Scala a statically type language might be a nice thing sometimes so on Brigade Wolf's so in order to explain kind of why we decided to scale out our micro services architecture in the end I'm gonna have to explain a little bit more about the platform and so one of the core components of Brigade is this thing called a position and this this position or a debate you can see an example right here it reads the democratic establishment has in thus understood this election by heavily favoring and Bushman candidate so that's that's the position and as a user you can either agree or just disagree with this position we also provide something as a core feature called alignment and what alignment is it represents how aligned you are ideologically with someone else on the platform and that's what's circled on the on the right there and so what when you agree with the position and someone else also agrees with that position that counts towards your alignment with that person and so that's represented by this percentage score so alignment is shown a bunch of different places on the platform including including on the profile which you were just looking at but also on this ballot guide which we were offering during the election season and so in this case instead of comparing two profiles you're we're comparing a profile with a group of profiles namely people who pledged forge you'll sign in this case or Hillary below and so this measurement of alignment is sort of shown everywhere in the product and for product reasons that really needs to be a real-time measurements however this was basically how we were calculating at that measurement we were using an SQL query and you can see basically what this does is it does a join on position responses and sort of aggregates how many position responses you overlap with a person on and so this can be really expensive considering that this computation needs to happen multiple times per page view per person and so at its worst I think the p99 latency was something like 10 seconds which is pretty unreasonable um so this is the performance bottleneck that really inspired us to actually build out our first micro service however as you might know already moving towards a micro service architecture introduces a lot of new problems namely debug ability becomes more more difficult because your competition doesn't lie and one place that lies in multiple places and you need to figure out how you're gonna build systems that are easy to debug also you have complexity with data access that you need to manage so previously you had a rails app which managed its own data and now you have several services that potentially need access to that data so how do you enable that and then finally you need to integrate these services with your app across different environments and namely and testing and dev since those are sort of environment side you don't really think about when you're kind of designing your architecture and so in this talk I'm gonna explain some of the decisions well we made while building out our micro services and how we address those issues that I sort of outlined here while moving towards that architecture so first of all why did we use Scala there are a few reasons we wanted the performance of the JVM and we also wanted the support for asynchronous computation that it provides for Scala specifically we we really wanted and appreciated the static typing that it provided us and possibly most importantly we were already using Scala in our data pipeline and since my team the data team was charged with implementing these micro services it was a pretty easy choice for us to make so as you may have guessed from the title of this talk we decided to use finagle as the framework for all of our micro services and one of the reasons for this was we really appreciated the simplicity of futures twitter futures are essentially the same as scala futures and that they just represent the result of a computation that hasn't necessarily completed yet and it allows us to do some really simple composition of futures both concurrently and sequentially and so in this example we're we're using this patch your L and this example I kind of just stole from Finn yet the finagle documentation so you can if I explain it poorly or you get confits too quick you can just take a look at that there but basically if that URL returns a future of response and then we use that that result to fetch more URLs concurrently and then we can collect them using this future dot collect to return the final result and so that all happens in three lines it's very simple and nice to use so finagle also one thing that we also really enjoyed both Nagle was how it represents a service as a function and what I mean by this is that it abstracts a service as basically a function that takes a request type and returns a future of a response type and it also defines a construct called a filter which can be composed with a service in the same way that two functions can be composed so it can basically wrap that service to create a new service and so that can be used to do things like doing time outs or authorization and in our case we use it to do things like translating requests and responses into a more debug friendly format and I'll I'll elaborate on that a little bit more in a bit and so an eagle also supports a bunch of different protocols including thrift HTTP my sequel and Redis which are all things that we use and so this means that we can we have the flexibility to implement services that speak both thrift and HTTP and we can use the same framework to talk to other services as well as our databases however for the majority of our microservices we decided to default to using thrifts and the reason for this was Swift's compared to HTTP provides a much stricter agreement between the client and the service and that means that there's no ambiguity over what parameters a request takes or what the response lottery looks like and so it does all that validation and parsing for you one nice thing about using thrift with finagle is that Twitter actually wrote a a thrift Scala code generator called Scrooge which works really well with finagle and so in this example I've defined a thrift service with one endpoint profile alignment and then using Scrooge it generates an interface for that service and you can see that it has this profile alignment endpoint which is just a service taking the requests type in the and the response type also by using Scrooge you get to avoid using the Java generated thrift code which can be a lot clunkier especially when you're using it in Scala and so on the top I've created a thrift object using the Java generated code and you can see that I first have to instantiate an empty object and then I set the profile ID and the second line which kind of violates this immutability thing that would usually follow in Scala and then if I want to actually use that profile ID I first have to check if it's set and to use it safely because it arises lots I it's null so instead using this Scrooge generated code I can just use I can just instantiate the profile line it with a profile ID and one line and all the other fields are just declared or defaulted to none and then if I want to use that profile ID I can interact with it as an option so I can check if it's defined or if I can that or I can map over that value so if you're paying attention there there are three things that I wanted to talk about which are potentially problems when moving towards the microservices architecture and debug ability was the first one so if you're if you're working in a rails app debug debugging your app can be pretty easy generally all you have to do is open a rails console on your on your running service and then run your commands there and it's pretty easy because you have a really nice ruffle and console to do that in when I was preparing this talk I I was going to say that you can't really do that with a running Scala service but I found out that you can't actually do that I think you can use something like a like Scala life and actually binds to a running service on by binding to that process but I actually haven't had time to go over how to use that so I'll talk about how we a Brigade decided to approach enabling our services to be a more debuggable so one way that we decided to address this was by implementing a thrift to http filter which we can compose with a thrift service and by composing it with that thrift service it basically converts that thrift service into an HTTP service which takes a readable JSON request and returns readable JSON response and so we expose an HTTP port on all of our thrift services that basically exposes all the endpoints that that service implements so that developers can just send a curl request to see what kind of response we'll get for a certain turn request however we found that the the to JSON protocol that Thresh provides is extremely difficult to read and write by hand and so for those of you who aren't really familiar with rifts thrift provides several different protocols both binary and JSON to serialize direct objects however it only has one JSON protocol that's both readable and writable but it's really hard to do so as a human and so what I mean by that is this so I've created a thrift object here and then I've sterilized that exact thrift object as JSON using the T JSON protocol and you can see that it doesn't really include any of the field names it just uses the the number of those attributes as defined in the thrift definitions which makes it really hard to read unless you're sort of okay with cross referencing your thrift definitions so for this reason we decided to implement a more readable and writable JSON protocol /dc réaliser and so this is this is our readable protocol this is the same sort object as before and here it is utilized as JSON using our protocol and for those of you who are familiar with thrift you'll recognize this to be the same protocol as T the same format as T simple JSON protocol however the the steriliser provided by thrift it makes this only a rice only protocol so we had to write our own D sterilizer to turn it from a JSON object into an actual thrift object so another problem with moving towards a microservices architecture is how you manage access to your data so rails manages its own database by using active record which manages things like doing schema migrations and also doing some nice things like tying your application objects to two objects in your database and so the problem is like when you start moving your micro services out from your main application is how do you allow those services to actually get access to the data that they might need to operate properly so initially we decided to do this the wrong way we had our alignment service periodically pull our my sequel database for the data that it needed and this is extremely fragile and I wouldn't recommend it because there's nothing stopping us from there's nothing stopping rails from changing the schema of my sequel underneath the feet of the alignment service so instead what we decided to do was we move towards a venture of an architecture and so by this I mean we we had rails push updates to the models into Kafka as events and then we had our new alignment service read those events from Kafka in order to get the state of the world and so this decouples are a new service from a potentially volatile schema which is much better than the previous solution that we had so finally moving towards the micro-services architecture means that you need to figure out how to integrate your new components together with your main application in different environments namely how do your engine how do your engine areas who work on your rails app develop and how do they test when all the when there are certain interfaces that now exist between the rails app and your new micro services so the way that we decided to address this problems we created two different strategies for our alignment services we we created a my school strategy or a local strategy and a service strategies the my sequel strategy was basically calculating alignment the same way that we had always been using that ugly align the ugly SQL query and so we use that in in our testing environment and then we use the service strategy and in our dove our production and staging environments and so that service strategy was basically having our main rails app actually make requests to some instance of the alignment service running somewhere in addition we in order to support a Deb environment when we started implementing a lot of different micro services we created this thing called an all server which is basically just all of our micro services running in a single JVM exposing those services on different ports and so in that way our rails application running locally could just connect to a local instance of this all server in order to make requests to it and then in order to run all these dependencies together in our development environment and also yeah in our in our development environment we use docker and also a tool called dock which we used to help define build and run a docker environment with all the dependencies that are realz application needs in our case our rails app needs to talk to my sequel Redis Kaka and like this also everything that I'm talking about and in order to run them since we've just committed all the configuration into our repo all you need to do is just call the command doc and then everything gets spun up with all the dependencies wired in for you so if you're interested in using this tool if you use docker at your own company we just open-source stock and so you can check it out on github so to summarize in order to move towards a micro services architecture from a rails monolith we decided to use finagle because of the simple RPC framework that it provides we also use zip to enforce a strict requests and response structure we address issues of debug ability by providing HTTP endpoints that our developers can call easily from from the command line and we also address issues of data access by moving towards an event-driven architecture and then finally we we enabled integration with our main app and our micro services and different environments by offering different strategies and then easy to use easy to use tools so that's all I have here are some links to the first is sort of like a link to a blog post that this talk was sort of inspired by if you're interested in using doc you can check it out on github and if you're interested in Brigade itself please do check it out so thank you so much you