Scale By The Bay 2020: Conquering All Stores with Gimel – A Unified Data Processing Platform
[Music] great man thank you uh hello everybody welcome to our talk on uh conquering all stars um my name is vladimir beckwenski i'm principal architect with software architecture and i'm here with my colleagues deepak and anisha who will introduce themselves when they get to their section so let's go to the next slide and let's see what is that we are going to talk about uh so we are going to start giving a little bit of the background and then figure out why do we need this system that is providing uniform access to a variety of data stores and then uh deepak and anisha are going to show you uh some lower level details including some interesting scala code and then we'll discuss the future so let's go to the next slide so thinking about paypal uh paypal as you know is in payments and we have a large number of customers and also um paypal is growing both organically but there are also acquisitions and uh you all know venmo but there are other groups and companies that we have acquired and what is interesting um when you look at the numbers that all of these things contribute to 360 million customers there is significant payment value and there is a large number of transactions that are happening so when you look into the ecosystem that we have the ecosystem consists of the core paypal but then there are a number of how we call them adjacencies venmo zooms and and so on and they create very large amounts of data that we want to analyze and gain insights so people was running um a large number of hadoop clusters we have over 20 on-premise clusters and now we are migrating to google cloud 200 000 yarn jobs a day over 160 petabytes of data and rapidly growing and some of the largest installations of oracle hortonworks teradata and aerospike which is the key value store to add to that there is such a variety of processing that is working over these data stores from conventional hadoop and hive to spark and so on so here is the uh a big picture that is showing the landscape in the lore area uh you have uh sections that are showing the uh data centers and in fact we are running across multiple uh data centers in different geographic regions and we are now in the process of migrating more and more of our activities in the public cloud and the interesting thing here is that our infrastructure is uh kind of a smart and it is using various mechanisms to predict the required allocation so we can move and shift the processing where it is needed uh there is also strong emphasis on redundancy uh because when you are in a financial world reliability is is uh very important then on top of that um a raw infrastructure layer there is a variety of data stores from your uh traditional big data stores like um age base and processing with files uh with hive but also there are uh streaming systems uh most notably kafka very large installation of kafka with with some uh very high numbers over six million kafka messages per second uh there is also significant text search with uh elastic but also there are stores like teradata for the data warehousing and variety of nosql stores like cassandra and there are also a druid for our analytic workloads and the the question is you have a variety of roles that would like to access this data and process so in the top part of the slide you see different roles developers scientists analysts operators and because we are heavily regulated on the right hand side you see um elements of logging and monitoring alerted and particularly security is really infused in every little thing that we do so all of our systems have either native or some added capabilities that would provide such great level of services now when it comes to processing this data you want to create um approach that makes this user roles more comfortable and more productive in the use so um in this area of the slide where you see the gears towards the middle left you see a variety of compute frameworks and apis they pre-can some of the developer experiences and the ways of common access but one of the really interesting and common things this is really essential is this symbol that looks like a book this is udc universal data catalog this is the meta store for our data that captures every little thing that you have in a variety of data stores in hdfs files it has the explanation of the data the columns if this is a privacy related information or highly sensitive information and the interesting thing is that you can go to the catalog you can browse you can search and this is also the vehicle that drives our compliance efforts in security and privacy because by inspecting the catalog you know where are the things and then the data is accessed through the uh apis for many of the uh stores you don't want to allow everybody to run wild uh queries over them but you want to have a more controlled access now a little bit over that in the upper left part there is a gimmel sdk this is the sdk that provides the uniform access to all of these data stores and all of these stores will appear as the sql data stores and this is actually really powerful for our analysts data scientists who would otherwise need to know and understand all the details and intercourses of accessing cassandra versus druid versus terra data and so on which will not be feasible with this they have a one way of interacting the system and one of the really exciting things in this space is uh trying to create a unified experience for our users through notebooks so we have the version of jupiter notebook that is heavily customized very much infused with security and that provides a really nice platform for productive development and experimentation and with that i will um pass it to deepak thank you thanks vladimir hi everyone this is deepak i'm a tech lead in data platform services organization in paypal and i would like to thank you all for taking the time to be here with us so why gimmel it's uh the most important question before we get into the details as vladimir mentioned the payments industry has accelerated along with that paypal's business has grown tremendously what this meant is the data has exploded in paypal and we've uh as you may have seen we've deployed a plethora of technologies and data is everywhere and being a data driven organization we would like to make sure the power of data is really leveraged by all kinds of personas not restricting to specific types of personas for this we need to democratize data so when you look at accessing data that resides in multiple different systems the access is really cumbersome and fragile this is a simple boilerplate code to access data in hbase back in 2018 and 2019 when we were writing spark applications to read and write data from different systems the code used to be something like this this is a simple example but it's complicated when you try to mature this and scale it not to really go into the depth of this code but if you expand on this pattern the code is really very complex when you look at multiple stores that may be in an ecosystem now even if you take a simple example of working within the realms of spark as a compute engine spark provides multiple form modes to execute your business logic we could write spark sql spark batch application or a spark streaming application or even expose the data in the form of apis by integrating thrift servers for the data developers there is a plethora of options to write their business logic one could use spark sql or choose a language as python or scala or java and then express their logic and then write a spark application so you can see the the permutation combination keeps getting more and more complicated for somebody who is going to begin working with the data and this creates a lot of challenges especially when you look at the data application life cycle let's say we take an example of streaming data from kafka into hbase maybe i'm working with payments data i need to create a data application that needs to stream live payments data or activities into hbase and then i do some analytics on top of it and expose the data as a developer we tend to go through this life cycle we learn how to access kafka how to access hbase how to write code in spark in one version we then optimize the code we deploy it and we run it but what if the compute version changed we have evolved from spark one dot x to two dot x now in three dot x just within the spark real so when you look at that migration the code breaks again the developer needs to learn how to adapt to the new version of spark and optimize the code if required and then deploy and run it but what if the storage api changed let's say i'm using a hortonworks connector for hbase and then it happened to change an api that requires me to actually go through the same cycle and the story is very similar for various other dimensions if the connector upgraded it's the same if the host migrated paypal as vladimir mentioned works in the multi-data center hybrid cloud uh realms so there is always something changing under the hood the hosts are being migrated there is a migration of a particular data stored to a new uh data center and so on all of these causes the underlying host names or fquds or words to change these again result in the breakage in code which requires us to go through this lifecycle and what if the story change let's say i really realize after running the workloads maybe after a year that hbase is not the solution any longer maybe i need to move to elasticsearch or another store i need to rewrite the code that requires to go through the entire life cycle but i can't be sorry to interrupt it seems somebody else is speaking if you are not presenting can you please mute yourself thank you okay seems quiet all right so the story is same regardless of what vector or dimension changes the application life cycle is very fragile what what this means is as a data developer or a data scientist who's working with data or even an analyst if you write something to work with data you might have spent five to ten percent building a business logic the remaining ninety percent of the time is really being spent on managing the life cycle of a data application and when we look at this across the board across multiple organizations in multiple teams the time to market and the power to the potential to leverage data is really reduced because of all the time that we are spending on this life cycle so we wanted to get to a stage where as a developer you should be able to write a code regardless of the changes that happen under the hood the life cycle should not really affect the application it should be able to run seamlessly and we should march on with new data driven initiatives so with that we ended up creating the data api but before we look into that let's look at many other factors that play in our organization at paypal scale we created a unified data api and a sql api which we will be looking at in the coming slides but once we created these apis the other things that started evolving as we deployed our data lake and then started developing big data applications is we needed to gain visibility across all the applications that were running so each team our product had their own way of instrumenting things and then having a visibility at their level but as a platform provider as a or as a frameworks team the visibility of the entire ecosystem was missing and as vladimir mentioned in the beginning you may remember there is something called the unified data catalog which is the enterprise catalog for paypal which provides a view of the entire landscape now our applications required to tap into this catalog apis in the runtime so that they can do many other things like redaction or classification of data to apply security controls now this again posed a problem because each team or application had its own way of integrating with the catalog and security is ubiquitous so it's not just within the realms of a store that the security protocols need to work at scale and strength it's also in a uh in an infused way manner in every touch point in paypal so we needed to have uber security control that we could apply if required let's say the gdpr has to be turned on in the entire hadoop ecosystem and we need to apply a switch of reduction across all the applications that are accessing some kind of sensitive data it is going to be a real nightmare because it has to become a program we have to work with multiple applications and the other problem that platform providers usually face is every platform provided whether it is kafka or hadoop or a database admin they always have the way of instrumenting certain interceptors or libraries that will provide them operable visibilities so if you think about all these different touch points which are very common to all the applications that are running we not only require a unified api to access data but also we need a unification of instrumenting all these different things that slowly got added over into giman which we will be looking at just to double click on one example that i mentioned if you just take any database or a storage system and if you want to audit the queries or the application access there are different means teradata has the forms of dbq logs you could instrument eagle for hive or there are a bunch of other tools in the industry right now oracle has audit tables i can keep going but the point is this way of logging and auditing access is very fragmented and we don't have a holistic view similarly if you look at the platform providers problem area and double click into it all these different administrators need to instrument interceptors and they have to work with the applications to make sure the applications inherit these interceptors and they get the monitoring logs and so on but from a user perspective the user has to work with all these different store admins to understand their interceptors instrument them again the core problem of just working with business logic versus putting all these other uh it is on the application is actually in the ratio of 5 is to 95 where 5 percent is spent on business logic the 95 percent goes into all of these other factors so what if we had a common layer in the form of gimel where all these operators can work with gimel while we inherit all the burden of instrumenting the right interceptors and we provide a one-stop shop to all the application developers this works uh in in the benefit of multiple organizations the platform owners need not work with hundreds of applications while the hundreds of application developers need not work with every platform owner to make sure they are instrumenting the right things while we form that partnership or the bridging layer between the two so putting it all together if you look from the left side a data application or a user submits some form of code in the form of a query or in the form of a data api and that query now when it goes through the gimbal application which is a spark application not only provides a unified way of running a code for example reading from kafka joining with mysql writing to hive applying transformation can be expressed in the form of a sql but also out of the box we are able to make sure we connect to the classification apis see if there are any sensitive elements if required to turn on redaction we are able to do that at the same time we are able to monitor and alert on the access patterns and provide the interceptors and monitoring capabilities that the platform operators require we can go beyond this but the idea is having this unification of api access plus instrumenting all the different requirements that are on the touch points of the ecosystem makes it a one-stop shop while the data development can hack accelerate in ecosystem let me just take a quick overview of how the gimbal api looks like what is the experience of a developer in our ecosystem when they use the gmail apis and then followed by that i will hand over to anisha so i am not going to run a code here in the interest of time but what i'm doing is i'm on a date of google's data proc cluster and i'm running uh a spark application with spark 1.2.7 and scala 2.12 basically you can observe here that we are going to uh start a spark application and we are going to add the gimmel api if you see here the gimbal api comes in the form of scala api there is a dataset class or there is a python api which can be a simple reference to the scala class and the pattern is same there is a dataset dot read api which provides access to any dataset in the ecosystem the parameter is you pass a string which is the logical represent representation of an object that resides somewhere in the paypal's ecosystem and there is an option that you can provide which allows you to control the behavior of the data api behind the scenes similarly there is a write api which you can use to pass on a data frame through which you can push the data into any sync it could be bigquery it could be elasticsearch or it could be kafka for that matter but the pattern remains the same and because we want to make sure our sql communities are taken care we have added a sql abstraction on top of these so that somebody working with sql could just express the same select clause instead of writing a java or a scholar or python code and also express the same right apis in the form of insert into as you may see here so the quick thing to observe is there is the notion of a data set there is the notion of sparks data frame which is the main object in the contract and there are options so the user now has a simple uh prescribed pattern to work with any data set or any object in the ecosystem they could do that through the apis which we expose in the form of scalar python or they could actually work through sql and the options that the user has in the programming uh real can also be exposed to the sql community by providing them the set options so that they can pass the key value pairs and then work with the data store to tweak the behavior now you might wonder how a string translates to something behind the scene it works very similar in the way the sparks unified api works the only difference here is we have the concept of a catalog provider where a catalog provider can be unified data catalog in paypal's ecosystem or it could be it could be a hive data catalog or it could be a user driven data catalog basically the job of a data catalog or a catalog provider is to supply the configurations at runtime which will help gimmel generate the code behind the scene which connects to any data store we'll look at the details of how data api works when we get to anisha's section so with that uh i would like to hand it over to anisha who can walk us through the details of uh gimmel i just want to quickly show you the code base after the call we can go to gmail.io and that should take us to the github repo and there is also a notebook attached here which i was walking us through over to you anisha thank you back okay so hope you are able to see my screen uh okay hi everyone this is anisha i am a senior software developer in data platform services in paypal okay now uh let's do a deep dive into gamer implementation [Music] okay uh so as explained by deepak before like in order to read or write from different data storages a developer has to do some research about different connectors and how to use them in a most efficient way so in order to solve this problem gamma provides an abstraction by providing a unified api which is dataset.read and dataset.write and also sql access over these data storages so that not only the application developers but also the ddbas and product managers who are not aware of the scala or java code can also access this data pretty easily so basically uh to summarize with all the boilerplate code for connecting to different storages is abstracted out here and developers just need to uh focus on their business logic okay now uh let's take a look at an example of reading from two different storages uh here like kafka and elasticsearch so as you can see here there is a unified connector config which is passed through the dataset.read api so these configurations so in order to in order to change any storage specific property or if you want to change the data storage itself we just need to be tweaked these parameters and the underlying implementation of dataset.read remains the same so this gives a flexibility to user to just care about these configurations but not the underlying implementation of the dataset.read api okay these configurations can be provided to these apis in different ways through catalog provider in gamer today we have different options for a catalog provider uh first is user so in this one in this one the data set properties are passed as json but as you can see here multiple users accessing the same data set will have to mention the configuration in their applications so there is no shared repository where these configurations can be stored and that's why we came up with a hive catalog provider with this all the configurations for these data sets are stored in a shared uh hype catalog uh so that like it's it's it's being shared by different users who want to access the same data set but still there can be different hive clusters right in our ecosystem so in order to maintain these configurations between the different clusters can be challenging that's why we came up with an external metadata service option here a user can provide plug-in their own implementation of fetching of their metadata services in order to fetch the data set from any metadata service if they have so in a paypal as mentioned by lab and deepak we have a unified data catalog which scans all the storages within paypal and stores the metadata for all the data sets within these storages in a centralized repository so this this is how like you can provide your own implementation uh of like you can plug in your own implementation of the metadata service itself okay now let's look at how a gimbal sequel works so a sequel can have can possess a select clause and insert clause and also bunch of set statements to set the runtime properties now through our code gen the select clause is converted into dataset.readcall insert is converted into dataset.write and all the runtime properties which were set before the sequel are passed on as arguments to these aps once we have the read and write apis we call the catalog provider to get the data set properties for this data set now and based on that storage type attribute of this data set the appropriate data set object is fetched from the data set factory and as you can see here the data set factory has the storage specific data set objects and with each of these storage specific data set objects have their own implementation of read and write method and these read and write methods have the core connector implementation like for example uh for kafka the read implementation has a kafka consumer code and for write implementation it will have kafka producer code so in gimel we have used a combination of open source connectors and some of our in-house implementations for the storage connectors so this is how a gimmel sql is parsed and converted into a scala code behind the scenes now let's look at a gimmel thrift server now as most of you might be aware of hive server 2 which is a long running service used to run low latency queries on hive this is built on apache thrift concept spark thrift server is an extension to hype server 2. the difference being spark as a compute engine instead of hype now with both of these services we get a power of running low latency queries but still we don't get an abstraction to read or write from different storages that's why we added a support of gimel into spark thrift server so that their combination with their combination we get a power of low latency running low latency queries with from thrift server and the power of reading from any data storage from gamer in this in in gamma thrift server we also added some paypal specific features and um like integrated it with our unified data catalog okay now let's look at a scenario where a user wants to read data from hive through sql now in a normal scenario user requests for a spark session from the cluster and once it like it will wait for the user will wait for the spark session to be created on the cluster and based on the cluster capacity and traffic they might have to wait for a couple of minutes and once they get the session the query gets submitted and they are able to read from different storages so overall this process has some latency in running the queries and also some external applications like node.js are not able to access these data storages because they need to connect to the spark cluster in order to get a spark session okay now uh let's look at uh how it looks like with gamma thrift server so here user can just submit a query to a gimbal thrift server and with this with the long running spark session maintained by the thrift server user doesn't have to wait for creating a spark session now instead of that the query is query gets submitted to the already started spark session and they are able to read from different storages right away with this the app external applications like node.js can also connect via jdbc to gamma thrift server and are able to read or access these big data storages very easily so this is how by combining the power of thrift server and gimmel we get we get to run low latency queries and we are able to read or write from these data storages using sql okay uh let now let's look at the next steps uh over to you deepak sure can we just move over to the next slide cool thank you so uh paypal is on the path to cloud so we are looking to add uh support to all these different stores that are in in the public cloud and as we expand support to these cloud storages we want to make sure we still keep the sql community uh completely taken care so we'll need to make sure the sql support that anisha walked us through is available for these different uh stores that reside in the public cloud and moving on uh as we continue to abstract the access one of the problems that we have started facing is a lot of users don't know exactly what happens when you fire a sequel on a terabyte system or a petabyte system so we need to start adding query guides and preempt runaway queries so we'll need to spend a lot of time on tuning the queries and applying guards behind the scene and on a wishful side we would like to expand our catalog connector support to google's data catalog so that the community beyond paypal can make use of either hive catalog or google's catalog or maybe aws blue or something else in the future so that it's not restricted to specific set of catalogs that we are supporting in paypal and as we move on to support these different kinds of stores one of the big realms missing in gimel is graph we want to make sure that we simplify the axis for graph as well with the emergence of neo4j in many different realms and spark having a good support for neo4j we are looking to hopefully add neo4j access via giml that way the graph access is also simplified and maybe hopefully in the future we can add sql support to it and uh last but not least as uh things are really heating up on the ml and lp side we want to make sure the integration with ml and nlp is tight in gimmel and there is a rich support in the gimbal apis so that the data access that is really simplified can be very easily integrated with some of the ml workloads with that uh we would like to uh end this session and uh for uh anyone who is interested in uh chatting with us we would like to take it forward in our uh gitter chat or we will be available in the channels in this uh conference and please visit gmail.co if you are interested in either using it or contributing to gimmel and we can chat outside looking forward to it thank you i'll just add a special thanks to our awesome gimmel team that is implementing all of these features and we are really on the path to democratize the data access in a safe secure and very flexible way to all of our users i'll see you then in our community room thank you