Scale By The Bay 2020: Lei Gao, Goku Flow: A Self-Service Data Pipeline Builder

Scale By The Bay 2020: Lei Gao, Goku Flow: A Self-Service Data Pipeline Builder

Recording: Scale By The Bay 2020: Lei Gao, Goku Flow: A Self-Service Data Pipeline Builder

[Music] thanks hey and uh thanks everyone to attend my session so uh today and i will talk a very interesting uh in-house uh project called the goku and the goku flow and we define it as as a self-service and the data pipeline builder yeah firstly introduce myself and i'm a senior machine learning engineer and at workday and now i'm leading a team to build a data science analytics platform but before i was a data scientist and purely focusing on the model tuning and some analytics work okay here is today's agenda and firstly as david mentioned and i will talk about the problem what's the workday and we're trying to resolve using the data and we'll try to optimize uh our cloud services and i'll generally talk about that what problem we try to address and then i will introduce the goku platform and the google platform you can think about is our internal uh data analytics tool and for the operational data and there are there are a lot of components and tool side i will give a really high level uh overview and what it is and what what way we use to resolve different uh problems and finally i will focus on the goku flow and because it is our in-house tool and also is a centralized uh component and we build all the other components across this this one yeah firstly uh introduced workday and the workday is really a leading provider of enterprise cloud application and we provide the application for finance hr and planning and if you are potentially some people and use your uh to the time off or to request some time off or do some uh uh receipt to expenses you will use workday and also we have a lot of applications for the financial but one of the most important that we are based on the cloud so this is the reason we can provide a better service for our customers and using the cloud and also we try to optimize our service and to provide a brighter customer service satisfaction i will make one of the example and our team or the whole data analytics team uh is working on and we provide the service of the cloud service for different uh uh side of customers different size customers for example we have a really jumbo customer and they need a lot of results for their transaction for their daily work and also we have some relatively small customers and they potentially multiple small customers will uh resident in the in the in the single cluster and for their service so if you can think about this really a very traditional optimal optimization problem like this water bucket problem and how to optimize the the service and the and also reduce the cost and it's very interesting sorry problem and uh sorry so this is just one of the problem we try to resolve and also there are a lot of very other interesting problems analytics problem we try to reduce for example we how can we do the capacity planning and how can we do the anomaly detection and about several things and but to summarize it and it will feed into the traditional three pillars of uh the analytics questions or problems and the first one is a descriptive analytics and it's really like we have engineers and we have the pms and try to leverage this data or metrics to understand what the performance looks like and whether our surveys meet the sla and whether there is really the request the peak and for some special uh uh window and we try to analyze the data understand our surveys understand our customer whether they're happy to use our service and also uh when we find some challenge and of performance concern definitely we won't understand what the root cause and it will go to the next stage i would call the diagnostic analytics and really like people can dig in and zoom in and find what kind of performance issue and finally people will be interested to do some prediction analytics analytics it's really like okay can we focus the the customer uh growth or results for research resource growth and can we do some optimization and based on the prediction and so we can provide a better service and before the customer sees some performance concern so this traditional three uh analytics problem and has been used as workday allowed and we use this different people use different tool technology just try to optimize our cloud service make sure our customer is really happy to use it so going back to the goku and we position at the goku and as a self service analytics one of the reason is that um we have in in the workday and we have a different persona and the groups and we have the engineering group we have the pm group we have the application group it's really hard to provide a centralized team and to serve all these people's requirements and and so we want to provide some tool internally and make sure the tool or the user can use it by themselves and they can resolve the question or answer the question by themselves so to to to provide this kind of self-service analytics and we define five dimensions and to me the requirements and the first one is data discovery and it's really like some people okay just join the work day old and he has some questions on the new services and how can they find the data by themselves very easily how they can not only find the data and also the context for example who is the service owner who is the uh the expert in this data site and a lot of information we want to provide a centralized data discovery a place and for people to start their journey and when they find the data i think they're very important that they can do the data exploration by themselves right and they can use the c code and do some basic analysis they can use python or r to some other also analysis really like they can play with the data by themselves using different technology and go to the next phase is that when the people people love charts and when they find any potential issue of defining very interesting insights and how they can share this insight using the realization and across the company this is a data realization problem and the last one is very interesting with causa data sharing and because we have a lot of data experts and they have built a lot of really value-added aggregation data and how can they share this data with other people either we use or share the domain knowledge that's very interesting question how we can provide this facility so people can focus on to share their expertise not a lot of data infrastructure problems but if you look at the whole uh surface and people will be interested okay if i can go through once and the next time i want to automate it and i don't want to you know do the ecal every week by myself so this will be go to the centralized component we call the data pipeline and it is really designed to automate the whole lifecycle for the for the data work okay and then i will talk about a little bit of word problem and what the data big data looked like at workday and we have currently we have a like this number and is i think now almost 20 petabytes and we collected our logs and from the different surveys and to centralize the uh data lake and just try to under analyze understand what the performance of the surveys look like and for every month and we are using kafka and for the daily ingestion and we have a two trillion message every month and we invested into our data lake so it's a lot and also we have over 740 tables and has been has been created by different team and this table actually represents a single service and you can think about that there are around 700 different micro surveys or some other services running there people try to understand it and it from different perspective okay uh and then uh i will talk about what's it what's goku and i i have talked about the problem we have the big data and we have some analytics problem and we want to resolve and so how can we resolve it so this is the question and goku tried to answer it uh by the way and if you have seen um the cartoon and animation and you will know what the cockpit it's very acute a very good japanese cartoon and for the for the data we uh we provide actually at workday and there are three layers and for the data internal operational data and firstly we have the data layer of data storage and the computing we are using the spark primarily and for the photo query and but on top of it is our user group and it's really like our domain experts and they are running some ad hoc queries and so they build the dashboard and reports and even they build some data application either machine learning or some optimization algorithm and also they publish or share some data and this is really like the requirement from a user group and actually goku sit me in the middle of this two layers and we are generally provide the serving layer for user group and you can think look at this function chart and we are using the amazon and for the data catalog it's amazing it's very interesting project really good and we like it and we provide a centralized place and for the for the data discovery using the amazon so people can find the relevant data data by themselves and going forward and they can use the japan and for the data exploration and it's really nice notebook and people can use sql python r different language and to explore the data and on the right side is the analytics function component and we use supersite and just build the dashboard and and it's a drag on the job open source tool and it's really interesting and how uh how popular it is at workday and use it to build the dashboards for the operational data and also we also provide are shiny or are still connect it's really like some data science uh analytics and because of its uh flexibility it can provide a really great uh drilling drill down capabilities and also we leverage h2o and to provide some prediction and uh capability because one of the reasons is that we really like a h2o flow and is a a notebook and people can very easily to do some basic model experiments or even model tuning so we provided not only for the data scientists we provided also to the pm or engineer so they can start to do some model uh experiments by themselves and also uh they are our in-house tool is google flow and we really design it as a data pipeline builder but literally we expand the the scope not only for to build a data pipeline and it's also a workflow engine and it has been integrated into our workday internal technology and for example we we integrate with a git some data git repository and we we integrate with the uh jenkins and for the csid and also uh conference and the jira it's really like for the business automation purpose and very interesting that we are building or building some service based on the delta technique and from this from the data break and i will talk about more details about how we leverage delta data lake and it can currently provide a really nice facility and for the people to share their data or to publish their data okay and then i will focus on the goku flow and is our in-house technology and we we're really proud of it and we really and we got also some patent based on this technology so we're really excited to share what it is and how it works but actually before talking about goku flow and we do a bunch of evaluation and before developing this in-house technology and but this evaluation is like a it's a journey and isn't and with along with the environment that we keep this evaluation and for example uh we look at the cooper flow it's really nice machine learning pipeline tool and we look at woody as a really star to this journey and are in airflow and it's really nice tool and also we look at the piper and from uber and it's it's a blog and it's really actually it shares a lot of uh uh design principle with goku flow and a lot of a problem actually uh we are facing we face the similar problems and and also a bunch of the open source sorry the commercial tools and for the data pipeline build and after evaluation and we list several requirements and for us it's really like the business problem we try to resolve and the first i will not go through all the other requirements but i will highlight several ways for example we want to provide a self-service data pipeline build and either people can probably build a data pipeline or they can use the dragon job from ui because art is not used by the data scientist and it's a lot it has been used a lot by the pm and some business uh users and the reason is that they really use it to build some dashboards and reports so definitely we want to provide a tool they can do the same by themselves and also we want to provide a really high availability we want we don't want to go through the downtime a lot and also the horizontal scalability because our data pipeline or data flow and it's not only used for the uh for the for the orchestration and actually it's for the for the data pipeline or data flow uh uh management and uh we want to integrate with workday technology stack so it can become a centralized workflow engine not a only a data pipeline engine and also i want to highlight is that uh we want to provide the seamless uh data pipeline and across different the precision ngo tool and the people can work on this on the on the data plug on the flow builder and they don't need to care too much about okay how my infrastructure look like and where i should store my data and how i can uh access to some uh spark of link service and i we don't want we don't we don't want to expose this information to the to the final user because um we want them to focus on the business problems they try to resolve and here is the very high level architecture design and for the gopro flow and uh major we have actually a three and four components and for the flow uh application and we have the flow builder and it's really designed to how you can build a data pipeline of flow and we have the scheduler and we can provide some scheduling and for the flow to run and also we have the running engine and make sure how the flow can be distributed on the different components or different servers to run and in parallel and very efficiently and also we based on our flow runner on the missiles and so measures can help us to distribute the work and to different servers and to either precising the job using the python scholar and r or even it can become a agent and it can it will be responsible to connect or talk with other big data processing engine for example link and spark ohio so this really uh the goku flow architecture and we build a data pipeline builder and the people can write it to build a data pipeline and also it has it provides a horizontal scalability and also isolate a lot of infrastructure or details and i will give a demo later and but before uh going to the demo session i will uh provide and highlight several features and we really i think they're very useful and the first one is that the flow builder and people can build the data pipeline using either the our dsl language and ou from the ui and the first one is a language the reason why people want to use the promoter to build a flow is that they can they want to generate this flaw on the fly and potentially they are using other tools or other language and to primarily generate this flaw on the fly and it's very important the future for the people for example if they want to do some simulation and also and we have we have seen some people they are using are shamey and they build the iso console but uh under whose they use this uh dsl language and to trigger a flow of build of load right on the fly but for majority of the our business user and they really want to use the ui and the dragon job to build a flow of pipeline and and they so this reason we provide another interface for the user and the people can drag on the job builder data pipeline and you will see more detail on the in the table and also uh we we want to highlight and very important or interesting feature we call the google contacts is really an abstraction layer and the people can the user can don't need to care about the two modules about the infrastructure especially the io how i grab data from the previous step how and how can i move the data to the next step i will show the uh give the example from this uh this slide and for example and we we created a google contacts and you can use the cdx dot read it means that this step you try to grab the data from the previous step you don't need to understand where the data it is and how the data transfers from the from the preset to this time it's just a call one line for code you can grab the data from the previous step and also you won't move the data or push the data to the next step for the further precision and definitely you just use write csv you can push your data to the next step and for the further processing and also we have a lot of uh integral we have integrated a bunch of realization of different components and uh from the google flow one of the examples we call the right report and this actually is pushing the data and the final result and to our realization engine automatically and so people can view the dashboards and the last feature i want to highlight is the data sharing actually this is this feature is based on the data data lake and but we don't simply provide the uh data data lake facility and just people using the native api and to you know invest data or read the data actually we provide a facility of service and we provide the google contacts and to abstract and or reduce the complexity and people understand how the data lake work or the data lake work and if you can see here and the people from the goku flow and they can grab the data from our centralized data lake and then they use the python or something to do the processing and they just use one life code and the data published and they can choose a different mode either appending or updating and they can automatically publish the data to the data lake and the data lake will register the data as a table to always to the latest version and people can readily grab the data for the realization of other purpose but the chat so the interesting thing is that uh based on the data lake and we can provide two uh functionality definitely one is the doubt of sharing that out of data injection and the second one is that we can track the the history and the version of the table of the data so we can provide a full lifecycle of the data and combining through combining the goku flow and and also the data lake might date okay uh yeah and five minutes and for the uh for the demo so and here is a vocal follow i will go through one end to end and data pipeline or life cycle and just to give a quick example how the people use it but the scenario the context is pretty simple you can think about some people want to publish or build a dashboard or reports and using the goku flow and the supersite so and here is the goku flow and we provide the workspace and you can have a multiple workspace and some workspace you can use it as your personal development environment and some space you can use your team or group development environments so uh i i go to our my yeah this one you can create a workspace all the projects and here i go to the uh demo and projects and uh and here actually is the code builder and it's for the you can view the different script here for example uh we can create a python you can see a multiple language provision so i have this data and the script ready and it's pretty simple data pipeline and i have the sequel as a practical just to grab the data and i ingest yeah this data is from a very typical machine learning area and data side very public data site and the next one i want to publish this result uh to uh to uh to to super size and for the realization you can see i i use the right report just to push the result to the to the location and here is a highlight and from the ui you can you can you can do the workshop or library some reporter and for the for the collaboration development and after uh i have the scripts i can build a data pipeline you can use the dsl but not use ui because simple networks update and the first one i want to create a spark sequel and this is a really the running engine and as you can see here i just pick up the screen that i just created and several minutes ago and i i choose the version and afterwards and i can use uh the next type i use one to use python as a running engine i pick up the python stack and i want to connect them definitely it all account this one is a sequential uh data pipeline but definitely it can support the 4k and join really complex pipeline logic so the next one is a bunch of for configuration is you can have set up as type of variables and you can set up the flow variables and and also you can have the advanced sighting and you can allocate the different results for your stats when i finish my flow and the next step is either uh i can yeah here is you can add it literally so the the next step is that i want to schedule my float right and here is about the hourly of the weekly flow running this is a scheduler but i decided to trigger it right immediately and when the flow is triggered immediately and actually you can monitoring and the flow status and also you can do the debugging and using the flow monitoring and you can find the standard odds and errors and to debug your flow very easily okay and the next step is that uh when the flow is finished or complete and you will see the status is green that's also and so you can also find the historical uh record records about your flow and here you will see uh maybe you cannot see it very clearly uh when the data uh is a result and flow complete and i want to ingest my data to the super side for the realization and all this work has been integrated together when the data ingested successfully you will got an email notification that okay your data has been ingested successfully and what the table name and in the super size so i will go to the super side and internally we call it the nimbus if you are familiar with this cartoon you know what it means uh in the super side and uh the data has been uh the table has been subject uh registered uh uh automatically so then you will go to the super side and i believe that the majority of the people will be familiar with supersize it's really a mature and popular visualization tool and open source so yeah here is very typical use case and i can create a several metrics and based on the data and uh i create a lot of metrics okay that's great and then uh i run the query i got a chart and i save it and save as a dashboard i just get the quickly because that's what it's very typical use case and build a dashboard of all charts and finally when i got to the realization i got the dashboard and the nice thing for the google flow is that when you schedule to run your dashboard can automatically refresh every time so you don't need to go through this lifecycle again this is the key to automate it so uh i think yeah that's all the information i have and yeah this is my linkedin and if your question definitely approach me and ask you