Scale by the Bay 2018: Phani Srikar Ganti, Erin Kavanaugh: Reactive Microservice framework for...
you micro-services and moral executions in in real time quite simply we call it quantum realtime in capital one there's a homegrown project also for the rest of the slides I'm gonna be referring it as quantum real-time do note that this is not quantum computing we wanted to sound like half way cool so we just took like part of the name basically quantum real-time is an art world provides is like it's not it provides out-of-the-box capabilities for real-time micro services and model executions at scale some of the other goals that that we focus on as is the fact that we wanted to provide like a simpler interface to develop and deploy real-time applications a little bit about myself I'm Shrieker I'm a software engineer at Capital One currently I'm the tech lead of a couple of teens which one of which is developing for in real time the other is a model heavy use case which which also uses quantum real time so I'm kind of like the customer of myself I'm passionate about distributed systems and big data and real time applications I've been with Capital One for a couple of years now I'm Erin Cavanaugh and I work on a used case of this framework so both use it to run applications and models in real time but also contribute to the framework before that I was working on internal marketing tools at Capital One and have worked on some developer tooling as well okay so we're gonna kind of give you an overview of why we built this and what it is before we dive into the architecture so when we were thinking about what it was that we needed to do we had several use cases across the company where we needed to run models in real time and we wanted to focus on what was the common issues between them and how could we solve for it in a way that was reusable so the first thing we thought about was obviously whatever we needed to build had to be focus on performance first we need to get results back very quickly scalability needed to be thought about early on as we bring in more teams we need them to know that we can scale to their needs and their users the next thing was configuration driven so we wanted it to really simple for business teams to get started with the business logic and make it really easy for them to explain exactly what they need to do in a way that's easy to read easy to reason about and reusable across teams so not everyone is having to redo the same code the next thing is about the model of consumption we wanted our model developers to use the tools that they wanted to use so whatever our framework needed to do it needed to make sure that models wouldn't need to be recoded and we could do it in a model in a language agnostic way and lastly we wanted to think about fault tolerance early on and ensure that we have logging tools in place so that they wouldn't have to contribute or to configure all of this upfront so as you may have observed by now we're doing a little bit of real-time context switching here slide by slide so why did we build this there was a recent shift within cap one few years ago we wanted to be on cloud we want to be developing all the real-time applications in terms of micro services we be really fast and scalable for all of this there is a way concrete need from use case teams that they want to have the capability for micro services and also be able to execute models in real time so this product concrete profession you know there are multiple teams doing this what is an easy way of like enabling and making it easy for all of them so that's where we came in we try to solve for this problem and one of the things that we immediately identified is that every use case requirement is unique every data science need is unique they're all solving for different problems and for a company as big as Capital One standardization is a very tough problem to solve and so instead of solving for that what we decided to do is go the other way around how about just relaxing everything so that we can we can facilitate all of those requirements and like keep keep things flexible as such and this also has brings in the question of like great we created a framework but for all the people who are trying to use it you know it has to be simple in US they would want to like just take it plug it and start using it really fast the time to delivery a delivery of any product should be really really fast so we also started approaching the problem likewise so when we were thinking about this we knew it two main users the first was the data scientists so in the past the data scientists would write their models in Python or R or whatever language they chose and those models would need to be recoded into whatever application they were getting deployed on so sometimes that was Java or Scala or whatever that was and in that process they couldn't use important libraries that they wanted to use in the back-and-forth time between the data scientists and the engineers as they did that recoding could be error-prone and involved a lot of interactions there so when we thought about that we wanted to ensure that the data scientists were able to use the tools that they wanted to do the training the deploying and the building the second team or the second user that we thought about was the engineers so we wanted engineers to be able to really quickly start up a new application and we wanted to put the deployment pipelines in place where they could iterate really quickly and make improvements and so when we thought about this working model we wanted to ensure that we would be able to provide collaboration and flexibility between the two of them by creating a really loosely coupled system both of them are able to work to focus on the things that they're really good at and the working model between the two works really well all right coming back to technology choices we we have a problem to solve we we know what we wanted to solve and now the bigger question is how do you want to solve it we did evaluate a bunch of technologies to figure out what what best fits for this particular problem along the way we we ended up with few choices Scala we use Scala pretty extensively with and with an ad within the framework primarily because it allows for functional programming and it also gives that flexibility of object-oriented programming wherever it is required to and most importantly it integrates really well with Java libraries so we don't have to reinvent the whole wheel if we wanted to use like any community-developed support one of the important components of this entire framework is the utilization of akka akka is a distributed messaging framework which which provides like an excellent out-of-the-box concurrency models using actor supports so I explained the first slides how we end up like using akka actor system which really helps us an infinity in the solution finally docker we want to enable this particular framework for easily shippable content so docker enables that lightweight and easily shippable containers so we're gonna walk through the framework starting off with a really high-level overview of what it does and then kind of getting closer down to the details so somewhat obviously what quantum real time does is it starts up a real-time application and there are two main interfaces that we're running currently one is a restful api micro service or you can run an off of an event stream you can run both of those interfaces at the same time if you have a workflow that needs to be run in multiple contexts so that provides a lot of flexibility the micro orchestration is kind of core to what the framework does so that's the akka that trigger I mentioned and that connects the services to the business logic we maintain a shared library of plugins in each plug-in is a component of business logic that can be either shared across teams or customized to needs in the future library are the rules and policies that the application developers are creating so to high-level this is kind of the major pieces that you'll see throughout the rest of the slides we did speak about like have have the framework provides like an art of the blog's application interface and once we start drilling down further we the primary goal was to provide those business functionalities which would have pretty much language agnostic and feasible to be plugged in by any user need so let's dive into those plugins plugins primarily constitute of the models or feature execution requirements or any business logic as such so the way we approach this problem is these these plugins could be written in Java or Scala Python and we needed a way to easily plug them in aside from that you know there are always like external data sources that we would like to talk to in order to procure the data and like process in further in order to enable the business logic for further processing and so this is the the framework is pretty well built and in a way to communicate with the restaurant stuff to on top of everything for any application as such logging at the end of the day is bottom-line important whether you'd like to debug whether you want like understand how the flow works or you won't even like support monitoring requirements logging at the end of the day is like a solid thing and we would want to reduce the burden on application developers as such by providing a whole lot of out-of-the-box logging to cool so this gets in a little deeper to what a quantum real-time application looks like so it's made of two main configuration files one is the JSON workflow configuration and the other one is the application configuration the JSON workflow configuration without all the routes all the workflows and all the communication between them so that includes the dag which you'll see here so all of the business logic which are which is what we call plugins are those red circles there and quantum real-time is handling the communication between them with the executors so those are the akka actors that provide the seamless communication between all those plugins the application configuration file is where we store constants environment variables and those kind of things so we can write really reusable and generic business logic plug-ins that can be shared across teams all right here comes the media part of corn in real time this this particular slide actually shows one of the fundamental blocks offered and by now I'm assuming that most of you actually know water Daggers or in general bag is literally directed acyclic graphs so remembering about like how we want to approach this use case requirement with the way we imagine the whole stuff is that for any given requirement or a use case with a set of requirements they can be broken down into simple tags where the graph nodes can represent the business functionality and the edges that are connecting them are simply the connectors to those logical subdivision subdivisions of the world now the only thing that's different between a use case you only use case is that the the core functionality of that business plug-in is changing but those connections which formulate those edges are basically the same they they just they can just be switched around so in a way we just wanted to create that orchestration layer just abstracts away the burden for the application developer to keep on repeating the same orchestration job like over and over again among these cases let's walk through this particular this particular diagram those red and blue circles as you see form from the business functionality where the red is the act of the red part of it is actually the core business logic for individual components and then the way we made use of ARCA is that we are wrapping wrapping each one of those business logics around by an actor so they're literally getting in encapsulated by actors so at the end of the day they don't really have within the system they don't really have to know what is the other plug-in that is actually existing within that application all they have know is that you know the system comprises of a bunch of factors and they know how to send and receive messages because it's a very standard interface and we all just work off like how these plugins can be language agnostic they they can they can be written in Scala or even Java we also have a way of like integrating Python data science models directly some other aspects of this is that if there are capabilities that that can be run in spirals if they don't have to really wait on each other they can they can literally be configured to do the same cue few things that data scientists usually run into is that say you're running a model in production you have a model version 1.0 and that's actually affecting your customer now you would also want to test you out like some other versions of the same model to figure out how how are the other model versions performing so in a way you would want to do shadow scoring without affecting the end customer there are certain patterns that we came up with where we can do those tasks synchronously I'll talk a little bit more about those essentialness faster and how we came about but one of the important things over here that enables that functionality is that we also have like streamed capabilities where there are certain generic plugins which are available within Kuantan real time which would allow want to publish data or events you a certain set of streams like Kafka streams and the rest of rest of the flow just follows a core act accordingly where you can combine multiple models you can just do for the processing on them or insert into sinks as as required let's look at the higher level architecture now this diagram may look messy because application development in the end is a Naxals messy process as he let's walk through from right to left right to left where in the previous slide we looked at a workflow which is a simple dag now that particular dag or a business functionality in general is fitting all the way on the work workflow processing layer which is to the right side of it that's all just about everything else the common set of things that are required for any application as such as you can see we have like three layers in this one as a workflow processing layer where the business functionality actually sits and then there is an orchestration the orchestration layer which is also provided out of the box which enables processing of those workflows which which also like receives the required request and then knows who where to forward this request in order to process them and then we create a logical logical separation for the services themselves so that any number of services added currently there are a couple of them which is streaming and handling with aware wherein it's constantly listening to events that are occurring on the streams and then doing the simple fire-and-forget so they are all commonly interacting with the orchestration layer which would then know how to process those particular requests similarly we have a restful service where you can stage like any and n number of endpoints as required and it would forward the request the same way in the previous slide I also talked about like shadow scoring the reason why it was important is that we run multiple models in production where only some of them are affecting the customers we wouldn't want to affect like make use of every single version of it to affect a customer as there are certain compliance and standards where a model can only be affecting the customer only if it is mro approved and this is a very important step to us let's go back to the previous slide once for the task that we wanted to do a synchronous processing we just publish those events with the moment we receive a request to the stream and given the fact that we have enabled stream and handling where it's listening to events all the time the moment it it realizes that there's a new event that is happening and it just receives it and knows that it has to for it to the orchestrator in basically knows where should that particular event end up or which particular workflow to assign actors so that it can do its own corresponding job so in a way if if you are receiving a request on one particular workflow where you just have to separate out the work working tasks we have one one and of it is like affecting the customers and the other end of it is not this is one way to just completely separate things out all right so we've talked a lot about these workflows in terms of circles and lines and stuff so we wanted to show you what that actually looks like so this is a really simplified version of what these look like in our applications but pretty much to the left here we have the different routes that we're running so you can run as many API routes in this application as you want and each one is associated with a workflow so for instance this one is associated with workflow a you can see workflow a in here the classpath is all of the generic plug-ins that we're maintaining and you can run as many models as many business logic is you want so we have two models running and some sort of DB interaction one as well and then over to the right are the edges so this describes the data flow between all of the actors so this makes it really easy to reason about and understand what's going on in your application and kind of consolidates all of that into one really easy to understand place so our obligatory programming joke futures and promises so when we're looking forward we have a bunch of things that we'd like to add the biggest things we're thinking about right now is looking into plugins state level management especially through akka persistence so if a if a request fails we want to know at what point it failed and to be able to restart it from there so that's something that we're actively exploring now the next is akka clustering and kubernetes we have some proof of concepts in place for this but we want to be sure that as we continue to scale out that we have the right technology to do that so we're using we're looking at using a combination of akka clustering with kubernetes for scale the G RPC protocol is interesting to us because then we'd be able to pass data between our data science models in our application much faster so we're exploring looking at that as well and then the last is probably the most futuristic for us which is looking at the framework as a platform how do we deploy micro services in a way that makes it really easy for people to put these business logic units together and to deploy them without having developers be interacting with them and setting up all the infrastructure so thinking about how to make this much more self-service in the future and we'd also love to hear your questions but also if anybody has any experience working with similar things any ideas of frameworks we should look into any feedback we definitely love to hear thank you [Applause] that's that's a great question so something that we don't sorry okay so the question was how does this compare to air flow definitely both both of them are like a good way to solve orchestration needs but one thing that we were facing was how can we integrate the data science models as required with into the application without having an issue like we want to just remove the barrier of like the languages where we can integrate anything as we wanted the current way we're doing it is we have like a custom solution for for the data science models wherein they can be plugged in and just provided to the application so it's easy easily pluggable one big issues is that even though there are other app other solutions out there not not everything is fitting into our particular needs we have a lot of custom solutions on streaming a lot of custom solutions on how do we want to do perform our data science models we found few issues with using other ones like a flow and so we decided to create our own stuff mmm no they are written in Python most job then we did have like few a few of them which which were also in Scala primarily when they were trying to use suo so the way they just integrated was like they created like a mojo around it and it was easily typical and it was easy to basically plug it in but most most often in like the primary uses that we saw goes that it's extra written in Python so the question was I read running multiple runtime executions in parallel like if Python Scala Java everything in paddle so the answer is yes let's take an example of this particular let's say let's take this particular example where you know there is a small box around three particular plugins and let's say all these three are models each one of them could be in first one could be in part on the second one could be in Scala the third one could be in Java and they can all run in parallel without without an issue the question is in production is it not too difficult to manage there are some standard set of challenges that that we do face in terms of how do you want to manage every use case is slightly depends different in terms of how heavy eaters some use cases require help there like highly dependent on like just models themselves the others are not entirely based on models but they do use a mix of it most often what we saw is that we have to take care of the compute power that is assigned to the application if if you are spending up way too many models and other than that we have we usually fully test test our frameworks like before taking them to production and typically we haven't seen much issues at all does that answer your question sir yeah so we're not doing model training over here per se it's only the aftermath of model training where the model training environment is in isn't the development which the data scientist used but once they have trained the model they give us the finalized version which basically is used for affecting the customers in in real time but at the same time to your question what if you have the other versions what does the bike drift I'm assuming that you're asking about any kind of additional overhead that may involve right absolutely and and yeah yeah so we have other monitor monitoring in place with this constant retrofitting in terms of like how the older models behaving how its and your models behaving and it's precisely why we also run the shadow scoring models which are not impacting the customers so the moment you know we determine that a newer version of the model is much better than the one that is currently running we just swap it out and the swap out process again involves redeployment but the redeployment is kind of painless because we are not really messing around with the code anymore all we are doing is just going back to the configuration file and just saying hey instead of using model ABC just use ABC which version 1.1 and that's all it just knows how to get it fetch it and like spin it up [Music] there's the students to that particular right so so the question was how do we keep track of those models started running for a long time where does the maintenance approach to it as such I mean I think it has like twofold answer to it one if if the if the decisions that we're making in real time is impacting real-time customer customers and since we also have like a whole lot of audit around it we have to be constantly checking and monitoring production systems so we are always we always try to keep it up-to-date so we know what is running - given that we also provide extensive logging and then like enable the use case teams to provision their own monitoring systems to monitor what is happening within the system they know if everything is running as as expected or if something is down in which case there is some kind of an interval intervention to bring it bring things back up and it's also like kind of related to the previous question as to what if you have modeled becomes stale and it's not really applicable anymore so it that that is when like that there is definitely a human intervention which says ok this this model is not as performant anymore or like a certain model version is not working anymore so we basically try to intervene and like figure out either we have to swap it out or if something is down like what is the better way to do it but yeah it's not completely autonomous like be the monitoring aspect is still there where there would be a certain amount of human intervention it's currently in a phase called inner sourced which is more global to Capital One as such there's a term that we use where it does not open service to the public but primarily just within Capital One but there are certain plans but you know these have to be properly evaluated as to how we can make it open source there's there's a lot of compliance things that we need to go through so the bottom line is there is no clear date to that as such but something under consideration and and for anyone who may have like listed the Netflix session today you may have seen like few similarities among matter flow and the same surprises he the same talk like years ago any other questions cool thank you all [Applause]