Building a metadata store with Akka

scala.bythebay.io: Jean-Marc Soumet, Building a metadata store with Akka

Recording: scala.bythebay.io: Jean-Marc Soumet, Building a metadata store with Akka

how everyone my name is jean-marc and I'm gonna I work for Salesforce I'm going to talk to you today about how we built made a data store using HTTP so here's my agenda I'm gonna first introduce akka and the actual model and I will talk a little bit about what we do at Salesforce especially in the field of machine learning and data science and then I'll go a little deeper into a case study of how we built a data store service to store all metadata and then I'll share our experience and our learnings from using HTTP Annika so this is probably not the first introduction to like I you've had this week but forgive me I'll do another one I can is a JVM implementation of the actual model it's built with Scala and provides a Java API and the actor model basically is a mathematical model for during for doing concurrent work and basically every the system is working down into actors that are entities that do the work and actors talk to each other via messages they have a state and a behavior and they can only change each other's states using by sending messages to each other actress systems have a year a key parent system parent-child type of jerky and a parent actor manages children so now I'm going to talk about the specific implementation of the actor model using akka akka provides an actor system class that you instantiate and it will manage your actor your lifecycle and the communication between actors and it will also manage its own thread pool and it will allocate threads as needed when the actors actually have message to process and then you have all the actors each actor is roughly 200 bytes and they are pretty lightweight that means you can run on one gigabyte of memory you can run over three million actors so it's much more efficient than using threads called it's much more you can fit much more actors in your memory then if you had to use four threads actuarial donaka have mailbox supervisor which is the parent and possibly child actors and so all messaging is asynchronous so a curiae can send a message to actual B and sometime later in the future actually we can respond and do send a message to actor a Akai comes with extensions there is the Akai remoting extension for for a casa actor system to talk to a remote actor system there is at the Rochester extension for splitting your actor system over spreading your actual system over multiple machines and I can stream for doing stream processing using actors I kept persistence for during actor backup actor state persistence in recovery and so of course there's a lot more extensions this is just not an exhaustive list and you can also write your own pretty easily and then the extension we're going to look at today is i KH TTP I catch TTP is the tool kit that lets our actors use HTTP to communicate with the outside world HTTP is the most common protocol to exchange data today and so our cache CP comes with the client and server API it uses a cache streams to handle HTTP connections using actors and it has a very expressive functional and iterative DSL that actually originated from the spray aisle projects the people that were we're working on this prayer project are now working fully on their cash project it's great for api's and RPC in a micro service environment it's not so great for rendering things like typically dhtml a quick summary to recap the baraka akka is very good for high concurrency it lets you maximize your the use of your computing power it's good for fault tolerance and resilience you can basically decide what you what you want to do if the nectar a child actor fails you can restart it or you can spin up a new one it's easy to extend with the extensions that I just mentioned and easy to scale out using a cluster it's used by or being used by popular projects it was used in spark up to version 1.4 and then they switch to their own custom RPC layer is used by flink for doing stream processing flink actually uses the Java API in fact and used by play and spray now I'm going to talk to you a little bit about what we do at Salesforce surface is a cloud CRM company CRM stands for customer relationship management and we basically help our customers manage the relationship with their customers I'll see you Marc Benioff is very is a very big advocate of using data science and machine learning to make this year a more intelligent and he's been very clear that this is the direction we're going in the future and so in the past couple years we've acquired a few companies such as meta mind SAS was a cue and prediction area meta mind does NLP and image classification using neural networks sus casas q does a smart CRM and prediction Lu is a open source machine learning platform that actually uses this prairial project and so recently at Dreamforce we announced this initiative initiative that's that pretty much drives the whole effort towards more data science and more machine learning into the product and that initiative has been labeled sales for each time so what it says for Einstein satsang is AI for everyone it's for our customers who are using our product is for data scientists when they want to create new intelligent features in their products and it's for XML developers service is pretty big on the developer community we have something called for zircon which allows any developer to sign up and come and develop their own applications on top of the sauce post platform and so we're going to continue in that trajectory the goals for for our engineers for the platform is that we don't want our customers to do ETL so we have to handle that for them a big pain today of doing machine learning and data science is you have to prepare the data ETL and clean it we want to take that effort and make it easier for our users to use the platform we don't want necessarily to restrict the platform to people with advanced degrees in machine learning and statistics so we said no PhD required and we want the platform to be scalable because we have a lot of customers a lot of data and we want it to be reliable so now I'm gonna dive a little deeper into what our platform looks like in the technical side we are basically performing three functions we during ETL so we transfer data from Salesforce the Salesforce API into our environment and we compared it to from CSV to Aero and we prepare it for machine learning in the future we'll also be dealing from other sources we're doing scoring on the data and push back we are pushing this car back to the SAS fourscore platform we're also periodically retraining models because ruin the models to be accurate and so these are the three tasks that we will focus on we have a strong partnership with AWS and a lot of experience with using things like Amazon EMR and s3 and so we're going to leverage that and all the the whole platform is going to be on AWS and so we heard about this cool technology called micro services and we're definitely going to at that so you can see on the right side basic diagram of some of our micro services and then they're executing the executive jobs against Amazon the email so we want to pick the for a platform we want wanted to pick the right tools for the job so our engineers have a lot of experience with Scala and in spark and so we're gonna we're using that for data processing running spot jobs on EMR writing the data to s/3 s/3 s/3 is great because we it allows us to not have to have a Hadoop cluster permanently up and we can save on operation costs one of the critical pieces of the architecture is or executors micro-service it basically its function is to monitor spark jobs from Amazon EMR and it's also running or ATL's from the surface API and that piece had to be very resilient scalable and fault tolerant so we decided to use akka actors and I Callister to implement that macro service and so for the rest of our micro service Scala micro services we decided to use a cash TTP to connect them together we've had a lot some of our engineers had a lot of experience with pre I oh and they were very adamant about the DSL and so we decided to leverage that - for the communication between our microservices so now sorry going back I'm going to go into more detail about this micro service called del Estero I'll stake the the problem that we were trying to solve so when we ETL data from Salesforce we know a lot of things about the data we know where it comes from which API which customer what time we pulled it what it is how many records what format and what schema it's in and this information can be very useful for machine learning and in subsequent operations down the pipeline and so we wanted to be able to keep track of that metadata and so what we've been doing in the past before in previous years we used in s3 the the path to store all this information not all this information but some information so for example we would store in a path that would look like this we will store the information that it the data is in CSV format and it's to belongs this company and and it was saved on this date and we will use that to be able to be able to identify what this file is when we want to use it so that has the presents a couple issues because first of all there's a limited amount of information you can store in the path and also it requires you to do to query the information in the order of the paths X segments so it forces you to use a certain sequence of predicates when you want to find information so we decided this was not a good solution so we built a micro service called the datastore service using a chi HTTP and we connected it to a post grades data database and basically what this the restaurant service allows us to do is whenever we want to write save a file to s3 we are also writing a record in the Postgres database sending a record to that datastore about that contains all the information about this file and so by keeping these two things in synchronization we're able to do a nice abstraction over the s3 file system and we're able to answer very complex queries about the data so for example we can say can you give me all the unprocessed CSV files for this customer or can can this file of scores can this model is called this file that has schema is e and it also allows us to do very advanced features such as daylight in age so if we have the files the file with scores and we want to know the history of this file we can trace back that it was generating generated from this a profile this our profile was converted from this CSV file and this CSV file was pulled from this ETL job so it's very useful for troubleshooting and debugging all this data this is an example of what data metadata we store about each file so we have an IDE and the company the format the schema ID because some files can share schema and we have even more metadata and here we have the location of all the files it could be one file or more files in s3 and the good thing is we're not necessarily stuck to s3 if we ever switch to something like Alex you or Hadoop then we can also switch to that so now a little bit more details relevant to our HTTP I wanted to show you give you an idea of how nice the DSL is for writing HTTP routes so here's an example of two routes the post route and a get route and you can see that I'm basically creating a routes value and I'm binding it to this IP address and this port and let's analyze the route so each of these keywords it's just these function calls are called directives and they basically allow you to filter HTTP requests coming in to in that's how that's how you implement the route so path posts and get our directives that filter out a cache TP comes with a lot of with pretty much everything you can find in HTTP such as boot patch those predefined routes majority sorry and so when you want to chain write routes together you use the shield operator and if the first route is rejected then this only then the second route will be evaluated the complete function call actually closes the route and returns the HTTP response to the server this HTTP function call is the akai HTTP extension and that's how you instantiate it so again very simple very easy very functional in the structure and when you have this in your code you can basically this becomes a value and then you can also concatenate two values together and then you have a lot a very nice way to represent your route directives can match on any HTTP attribute you can also write your own directives that basically look at the HTTP requests for things like headers and the that's how you can do things like authentication now I'll share or what we learned while using akka and NACA HTTP so using akka and the actual model for ITIL was very very nice because all our actors are lightweight and they're very good for infrequent activities such as polling for the status of Spock job so what they do is they wake up and they query the EMR server and then if the Spock job is not completed then they go back to sleep and we don't use any thread we don't have any colonization at all Akash GP has a very nice DSL we're very impressed and has all the HTTP constructs that are in the RFC and you can also do very complex more complex things than rallying you can do requests proxying and routing and you can do things like WebSockets and like I said routes are very easy to maintain and troubleshoot if she if you have an IDE that lets you click through to the value definition a bit of warning actors are maybe lightweight but they do need an actual system so I guess if he might not be suited for simple clients so if you have a simple lightweight you want to make your client lightweight and only want something I can talk HTTP it might not be the best solution because you need to spin up an actual system if you want to use a cache DP the actual model has a learning curve you have to think about how your application can be modeled using actors and that's not necessarily is something you have to learn a little bit more about the actual model to be able to do I cashed it becomes bare-bones by design it's it's very clear in the docs that is not a web framework so things like corals and CSRF need to be implemented or you can use third-party libraries so to finish I want to give you based on our experience these are the libraries that we used we use CLC it's very very good for doing Jason theorization encoding/decoding typesafe config is a very nice library for environment and configuration it's actually the one it's from light band and it's the one that akka is using by default we used slick for doing or RM on top of database Hickory CP for connection pooling Flyway for database migration I cashed EP session for things like authentication and durability tokens and I cache DP tasks it for testing or routes and that's it for my presentation [Applause] anybody asks questions so the actual system itself spins up it spins up I mean is it's also lightweight but it spins up a lot of things you might not necessarily want when you just want to do HTTP client it also has if you don't configure it properly you can also have a lot of things you don't might not want but it's so not so we we tried it and we liked it but there are other there are other libraries there's a spray JSON library but the good thing with chassis is you can use implicit encoders in your code and so we really like that but yeah so hey I don't have a better answer yep all right thank you [Applause]