Scale By The Bay 2021 : Lei Gao, A Journey of Migrating Data Platform from Data Center to Cloud
okay awesome and uh yeah and thanks a lot for attending the my session and uh um actually yes last year i presented our data platform internal data platform and the name the the platform of the project name calling the goku a lot of people are interested in this naming and but uh this year we i want to present some new change or new effort and based on this data platform and we are migrating or moving to the public cloud uh i think we started this journey from middle of last year and the leadership made the decision and until now you can think about one year period but uh during this one year and we did a lot of research we did a lot of poc or technical design i think for this conference i really want to take this opportunity to share you know what a benefit or motivation for us to move to the public cloud what the technical problem we try to resolve and also very important that i want to share some practice or some experience with how we have we have gotten and hopefully next year and when we finish the migration i have the opportunity to present again and share more more best practice okay uh yeah i know san diego has introduced me but i will introduce myself again and my name is ray gao and i joined the the the workday actually as a data scientist or machine learning engineer so in my early stage of my career i really working on the consumer side of the data platform and working building the machine learning models of data etl pipelines afterwards and i moved to the data flow data platform of foundation or tooling size and now it's leading the team build a internal data science analytics platform we provide you know different analytics solutions for the internal users and we build some data access layer or something and really like make sure at work day every engineer every people collaborate data and access to the data very easily so here is uh today's agenda and i first i will do a quick and go through the current data platform in the data center and last year i presented something just in case someone potentially have never seen it and i will go through the platform and the technique now we have and then i will list several challenges or problems and wait for it definitely this motivation we move to the public cloud and afterwards i will talk about uh when we move into public cloud what are our expectations what our technical goal or even the business goals we set up and to to to evaluate the migration and the last thing i want to share you know the journey and we are just several months you can think about really the technical part is like six months or eight months and we learned a lot of things but definitely i know maybe it's not a best practice but i think it's worth sharing and eventually we can discuss and if you have a vital idea or you have like a bicycle practice definitely and talk with me and i'll be happy to learn uh yeah before talking about the the technical part i have a quick introduction to the company and the work and is a leading provider of enterprise cloud application for finance hr and planning yeah and we are the cloud provision for the enterprise company softwares so for our team our organization and what's a business problem we try to resolve and even uh not only our organization because we provide the tool and but we are really providing the data platform for the internal users and most of the data are operational data you can think about the logs and the stats or metrics and this really operational data but the benefits of the the business value of this data side that we use is really like optimize our cloud service you can think about we use our data to improve the performance and we use our data to improve the user engagement a lot of things we try to make sure to deliver in the world class and basic works class and cloud service for for our customers definitely data is very essential components and during this journey and definitely we i believe that most of the data scientists are given the pm and they will feed into three pillars and either they do some descriptive analytics you know building the dashboards and the real queries answer the business question or even some people do some diagnostic analytics and they find the performance issue and they dip in and find okay what's a good cause and how to improve it in the future and even we do some predictive analytics it's really like the like very typical quality planning and the volume prediction a lot of things and this is generally how the company used these logs and the operational data to optimize our cloud services and then talk about our data platform and uh we are a team a really big team and our neighbor team and working on the data platform and really on the data lake we are working on the data serving and part so i have a quick introduction to the you know data lake and since now uh we have like 19 petabytes data in over the last several years and accumulated but i believe that the number should be much higher than 19 potentially 20 or 20 something and we have the data velocity is like we have the 2 trillion message and i ingest it through kafka per month and it generates the volume a lot of message or data and ingest it to our data lake and we have like we use the tables we have over 740 tables i believe that now it should be 8 800 tables and really a lot of tables and in the in the in the data platform if you look at this really the data the serious data lake and how we can make sure every uh you know most of the engineers or employees at workday either from the engineering department or from the business department that can very easily to leverage this data make the business decision it's a really a big challenge how we serve the data to the user this is actually our team's pillar and to serve our users using the data this one is uh our architecture in the data center now and you can think about that we are really the analytics tooling sites and serving serving parts um if you look at our our in-house tool i will have a quick demo show how it works together and you know desires one use case but if you can see really uh in the in the in our in our in our platform the core uh component that we call the flow and goku flow engine you can think about it is really like the orchestration and the coordination and even some workflow integration engine and it will be in charge of building the etl or even machine learning pipelines and also be used to integrate with workday stuff when i talk to workday stack is not a workday code services it's really integration with workday tooling sites for example integrating with jiras or confluence slack email a lot of these internal services and we integrate them together so user can really build a workflow automated and the flow engine we built it on top of the missiles and i know measles potentially is a little bit out of the picture in the tech world now but considering that we made the decision five years ago and finally we pick up or choose the measles yeah definitely since now we are still using it very well but in the public cloud potentially we will consider kubernetes or other you know technologies but we build our distributed orchestrate an engine based on the missile's cluster and the missiles will be have the different agents talk with either the running environment and the query and in flink streaming framework on top of it we build the software it has a you know job scheduler it has a dsl pattern it's our own programming language to build a pipeline and people can very easily write several lines of code and build useful data pipelines and the last one we have the flow of code editors i will show the detail and definitely there is a ui on top of it and across and this goku flow and we have a different tools and we elaborate open source and community a lot and for example the notebook we use javelin and the nimbus we built it based on the uh supersite and rco connect h2o and the data catalog we built it based on the amazon from the lead but the very important thing for this one is that all these tools are built around this flow engine as i mentioned it's not a orchestration it's really like a workflow engine so we build all these open source tool and they integrate with the flow so all the tool can be connected i will show one example how it is how to use uh a quick demo and for the for one of the scenario use case so people maybe can very easily understand how the flow engine and other open cells tool are connected with each other so here is a a use case and you can think about this use case is very typical and happen every day and in each organization a lot of pmo engineers they want to build the dashboards and also they want to automate the dashboard refresh weekly or monthly very typical of all simple requirements see how the goku can can meet the requirements if you come here this is our in-house tool of law engine and it has a project concept so it has a naming space concept people can create a project organize your flow and the code into different workspace and in for example in this demo working space and firstly people can go to the code builder and and write the scripts and the online version and you can create the file for example you can either support multiple language and uh for example you can create a python or some some scripts and you know using the code builder and afterwards and i have for this demo i have two scripts one is the sequel and another one is a place you know simple enough so this sequel uh you know it's a very typical sql query pull the data grab the data do some aggregation but for this python ones and it's really like the integration it can be used to further transformation or cleaning or some work but most important thing that if you look at the code and the right report this is our api it can automatically invest with the results to the to the dashboard for the realization so this is the thing i really mentioned how we can integrate with a different tool across the flooring so here yeah i will not go through one by one line by by line about the code but it really highlighted the the goku's context it's really like a lot of abstraction and layer and provided by the api and also very important that we integrate with our gita and repo at day and people can easily work to the question control and i think that is a very important requirement by exteriors and machine learning projects and now you will see there are two uh slow engine flow engines one is the dsl it's probably building the the pipeline and another one is ui based flow it's dragging the job flow engine so for this demo and i will pick up the ui one because it's very you know based on our experience it's well definitely the most popular more popular and the flow engine and here you can see people can choose what's a running environment for your for your step and you can pick up the scripts and we just we just just write in the in the code builder and specify this is the spark and for the second step we use you know the same and you pick up the running environments you pick up the scripts and the very important thing that you just click them and it will be connected it will support the sequential of work or join any complex and flow engine and also it has the environmental variables and you can set up the global flow variables you can set up a stack variables and also very important that because we use measures and you can allocate the results for each step and you can configure how much results actually used to run the flow render stack yeah it's a then it's really some other functionality you can rename or edit your flow again and you can schedule your flow to run a different frequency and yeah so this is a weekly engine scheduler and also you can directly and trigger the float right yeah definitely i will right now uh there will be a progress bar to show the status but most important thing that you can debug your flow online and you can check the status and debug what's happening and behind the scene okay and afterwards when the flow completed and you will get a email and say your flow is running successfully just check i have either how much time i have yeah you when you get the flow and running successfully you will get a notification then you can directly jump to the supersite and build the dashboard and based on the data automatically interesting i don't know that i will skip a lot of things about the the nimbus because about the super size because i feel that a lot of people are interested uh familiar with very popular open source visualization engine but the the beautiful the beauty of this integration is that uh when you integrate when you use build a flow use the flow to build the etl pipeline to drive your dashboards and all the things is automatically and when you build the dashboards and you schedule your float well automatically and every time or every week your dashboard will be refreshed automatically so this is really what we mean the integration we really highly integrate with the open source project with our flow engine making sure all those or things can be automated sorry and yeah but i know sometimes and it looks great but the challenge is that always we have the challenges and then i will talk about what challenge we should we try to resolve and when we build this platform and we have three different challenges we try to resolve one is the scaling and reliabilities and if you look at our uh tooling side of the platform is not so big but we still have a lot of scaling capability or reliability issue and one of the reason is that we need to deal with a lot of infrastructure we need to deal with a lot of running environments and allocating results a lot of these things so it makes the running environment of the team and spend a lot of time on the operational work and second the challenge and is the cost and um um you know the cost is made for how to optimize the results and when we have the peak time and how we can allocate enough results and for the for the uh for this work and the last one is user usabilities we try our best to integrate with the tools different tools either but the still is fragment and a lot of people need to jump around the tools and they give the suggestion of a pinpointed can we have a single stop and we can do everything and at least 80 percent of the world can be covered in the single tool to address this challenge and we start to think about the journey and moving to the public cloud and um for our team and we it's not only our team but the whole organization has some uh observing the trend in the industry and the data match i believe that yesterday the workshop and there's a really good workshop talking about the deep match a lot of companies i'm talking about this thing and this is our uh solution and how we leverage the data match and in our new cloud and the data platform and here the blue one is really one of our pillar and the service we try to provide it and we try to provide some domain agnostic service data service like the micro services for the rest of the team the domain experts they will build a dashboard and reporting application or even publishing some data size we try to serve this one and provide these services before we really provide the analytics solutions if you look at the flow the super side a lot of things we call it analytical solution it's a ui and people can do the work there but the we got a lot of requirements said oh yeah we have our own tooling and we have our own requirements and can you provide the you know the query service and metadata as a rights api so we can very easily integrate or grab and access to the data and from our tooling site so this reason we provide also very uh try to our advisor to provide the rest api as another a very important feature ui sorry the user a interface and some people can variously to access to different micro service or data services and through the rest api they are not highly titled by these solutions we provided and going a little deeper and that's why it's really high level you know see the data match and in the public cloud what our belief on it and here is really like a deeper and look at how we can implement it right and we have the data lake and we have a different data side we have stats log metrics aggregated data and based on this data lake we try to provide different services and in our backend we call the data micro service or any any name for example we have the data access service or data access layer and people can access the data using a unified view and we still have the flow service and our flow engine and can provide the etl or orchestration capabilities and also we try to provide the metadata service and it's really like the data itself the schema the lineage and also very important is that the data assets and around the data the dashboard and popular queries the data assigns and definitely the authentication service is a security model on top of it that the people can get access through the rest api and also we have a bunch of the energy analytics solution and out of the box and for example we have the data catalog still realization engine the flow console notebook and even people can bring their own computing engine and they can use the right api to access all the micro services on the backend so this is i think super important that can meet the requirements either some people and pm they just want to get the question answered immediately they can use the visualization engine or notebook but for some domain experts they really want to build the the their application or their their tool based on the data and they can use the rest api to access to the data then uh yeah we set up this goal just really we try to build the data manager you know the micro service domain agnostic service different words describe the goal but what the journey and during this journey and we can think about yeah several things when we move to the public cloud and so for my team or on my organization the first one is definitely we need to migrate the existing data assigns or data to the public cloud you know very essential and the second one is that we need to fully customize or integrate with open source even some cloud services because we need to put them together and become a tooling for workday and we are not building a generic tool for for the for the industry we are doing really something work they needed and the last but not least is modernization and we build a lot of tools and stuff and based on our infrastructure people and we are limited by the you know security and and the infrastructure and the platform and by the moving to the public cloud we have more toys and we have more freedom to pick up the latest technology definitely the platform modernization will be a very important dimension so i will i will share several examples about on each dimension or each category for example one modernization thing and i share is with building data access services and this one is really interesting and because before we have multiple datasets we have logs and the metrics and even some more business data each data has its own metadata and the view or access interface and some through the jdbc thanks to the rest api and the people the pinpoints is that firstly they need a job across a different tool to understand how to get access to the data and the second way is that they spend a lot of time trying to merge or join the different data sites together so this data access service is really important and essential is that we try to provide a unified and view to other users and they can very secured and access to the other data sets join query and running some stuff and it's super complex because it across the different datasets a lot of things but we list several requirements of from our users or customers and they really try to see what the data access layer can achieve and firstly yeah we we have two personas one is we call the data viewers or data analysts and they build the query and build the dashboard and be answer the business question distribution requirements and when they look at the this data access service firstly they need a standard sql interface and they most of the data analysis of engineer can write sql query they really love it and they really want using the single sql query to access to all the data sites and yeah this i mean another requirement is really like multiple data cells and including the logs stats and operational data even debating this data and they can use a single interface to access other data source and then the next one is interactive performance because for this uh persona and they really want access to the data and building the dashboard so they are preferred to some interactive performance and the last one is the query and report a lot of requirements they can they want to build a single stop and they can query and report and another persona and we call the data engineers although experts and their requirements is a little bit different but very interesting and for example they prefer to start accessing to the data anywhere because they have potentially accessing the data from their vpc in the public cloud or even they want access to the data from their ide for the development purpose yeah definitely access anywhere and it's very important and the second is remove a lot of complexity this is really the api uh come from and we need to provide a lot of api to hide the complexity of the different data source or different locations and query something a self-service model and a lot of like the etl or something service and there should be the self-service they don't need to depend on our team or other team and become a bottleneck and the interoperable is really like the other data size of or or domain silo can be interact with each other this is also very important thing and look at another example what we are proposing is a customization and i as i mentioned earlier and the customization the goal is really provide uh deep integration make sure people have a smooth experience and using the single tool size one example is the super size supersite has been used uh dramatically as workday as a virtualization engine and building the dashboard and look for the performance of other purpose but the challenge when people elaborate the super side is that firstly they they really want to have someone optimize the flow engine or data pipeline engine behind the scene and user they don't want to build the etf etl by themselves they want to write the query and do some transformation and they can someone can help them optimize their their query and do some data appending or accumulating make sure the dashboards can be refreshed automatically so what we did is that we will uh integrate and super side with the flow engine now you have to we have integrated it will automatically ingest data for the localization but the requirements is that people want to really build from the ui side they can create the flow they can schedule the flow they can build the dashboard and from there they don't want to jump to the flow engine and write a bank for code so definitely one of the requirements of customization is that we will build a ui from the super site and really and trigger the the data pipeline or etl and behind the scene but under whose the flow will be a backhand but the flow uh backend will be integrated with this ui and people can very easily to schedule a trigger or you can't pipeline from the ui side and another require oh yeah this one this is another requirement also in the super size if you have to use the supersite there is a component calling the sqlite and you can think about it's really the sql item and it has some basic metadata and for example if you click you can see the partition you can see the schema you can see some basic uh metadata but a lot of metadata and people really want to see like who is the data owner and what the most popular query for this table were the most popular dashboards and for the square for this table and we have the amazon as our data catalog and mine provides hub so what we want to provide is also the deep integration and between the supersite and amazon and people can using the sql lab from the ui we will provide another ui interface or even the form and people can very easily pull and the other metadata and in the same ui so they guided the enrich the metadata from the sql lab they can grab all the metadata information from the from the same ui but before it's really they need to jump to the data catalog and to look at it more detail and then jump back to the super side build the red query so it's really a challenge and a pimple point and another one is about migration migration is a pretty straightforward and we but we need the migration is really challenging it's a long tail and effort and we definitely we need to migrate some data to the public cloud that's pretty straightforward and simple but the more important thing is that we need to migrate the methods and because for example the data when we do the migration actually we do some cleaning and do some transformation or even change the data format for the users and they really want the cable the consistency so how can keep the mic data consistent and migrating the metadata is actually more challenge and comparing to the my data migration and also running out of time oh okay uh it's nine plenty on nineteen nine ten oh sorry yeah yeah i will quickly go through it and uh yeah the last slide lasting hurry yeah my team is hiring the one principal engineer and one uh product manager and definitely if you are interested in this product and the fantastic opportunity you can contact them with my uh recruiter mike or you can directly ping me and we have we're happy to talk and discuss and potentially you're you'll find a fantastic opportunity there yeah the last one is my linkedin any questions you