Scale By The Bay 2021 : Adi Polak, Rethinking scalable machine learning with Spark ecosystem
perfect um so hi everyone today we're going to talk about rethinking scalable machine learning uh with spark ecosystem uh we're going to talk about how we can leverage gi why we need to think about massive amounts of data why distribute machine learning is on the rise you probably heard about uh different tools today or during uh there is this wonderful conference around rey and other technologies uh and overall why we see such a huge demand for machine learning productionizing machine learning ml ops and the different tools that exist in the industry so i hope you're excited i am definitely excited and let's get started so like we mentioned earlier my name is adi pollock i'm managing a team at microsoft i co-created the data engineers community you're most welcome to join us we talk about different challenges we see in the industry from full supporting the data scientists to the analytics team to how to build um the data infrastructure data ops privacy security a lot of great conversations there so you are most welcome uh i'm currently writing a book for riley about machine learning with apache spark how to build the pipelines and the different tools that exist in the industry around the apache spark tools i'm a data bricks beacon and i have more than decades of hands-on experience in the machine learning and skill bill data infra so let's get started today the goals for you is to understand why machine learning is slightly different than how we do code what are the different steps that we're going to take to productionize machine learning why data is key when we look at small versus large data set there's always a conversation of do i need more of them one gigabyte data set to train my machine learning or can i use just 100 you know small files like csv and stuff what spark gives us out of the box and how we can craft the model at scale looking at emily and how we can bridge spark and deep learning as well so let's start from the beginning machine learning is different when we think about it often when we write code we deploy it to production it runs it takes some data all is good but here actually we need to produce the machine learning model first right we know we take the data we ingest it into some algorithm it can be algorithm from different libraries pi torch tensorflow emily there's a lot of libraries out there today it produces a model then we take that model when you wrap it up and we add more data into it and we put it into code and that code gives us answers right so we inject the code with with data and then we get back answers so it's a very high level simplified idea of what we do in machine learning but how the data scientist process works we get the data ingestion we get data in our system we do the cleaning transformation some feature engineering domain expertise and then we train the model we test it we validate it maybe we change a little bit of parameters there high parameters we change the algorithm that we're using and then we're pushing it forward right from a data scientist perspective we're pushing it forward to staging and deployment and this is about the end it for data scientists but production like saying machine learning it's slightly different conversation right because we have to think about the different stages that we have the different stages which we go with code before we push it into production any software we do go through those stages so we have the development stage the staging stage and then the production stage and we're going to talk about each stage a little bit deeper to understand where are the challenges and what are we facing today first thing first development stage and in development stage often with data scientists in machine learning workload we're going to look into two different stages within the development stage so the first one is going to be when we're working on offline data we're trying we're trying to understand what the domain problem is what the data that is available for us how we can train the models which algorithms we want to use a little bit of understanding of the feature engineering process if you want to take and all of that something that we do often on a small data and this is great however it imposed some challenges code is not built for the large scale data right we use tools that we can do we can work with on our local machines and code is definitely not production ready we work on our local machines it's offline small sample data and this is great because it allows us to try multiple algorithms features hyperparams like really get into the unique video of the data and what is available for us in there really get the domain expertise try and really understand the domain problem as well and this is great until the data science often called the engineers or the rest of their team and say hey you know i just discovered that the accuracy of my model it's it's not so good i mean it's fine it's a good start i know what the features that i want maybe but i need more data i need more xyz data streams into our system because i need to develop more features it's not enough right i need to bring in and enrich my data my csv strings somehow are not loading properly something has happened along the way with the formatting and the biggest ask that we see all the time that data is not diverse enough it doesn't reflect uh it doesn't help me solve the problem business problems that i'm trying to solve and this brings us to the question around data why data is so important in our machine learning workload and we can look at small data versus large data set and each one of them brings has benefits and also brings challenges to our system as well we know with small data set it's a little bit hard to appraise the real worth of the representative of a particular sample like it's really hard to uh define a set of rules to solve a specific challenge however from a compute perspective it's easier for me to run it on my local computer i don't need an extensive skills to write it on my local computer the tools exist also there's a lot of open source tools that exist out there right so i can start working right away however with a larger data set i can get a better chance to encapsulate and represent the domain problem or to solve the problem that i'm trying uh to solve the problem basically the business problem however it requires a cluster multiple machines it requires or a very strong machine it can be pricey it also requires sometimes software that is proprietary close close software or some tools that i'm not aware of or i don't have access to or not part of my skills skill set so after we went through the first stage and we realized kind of okay we understand the overall flow of how we want to build the data scientist part now we're going into the second stage of my machine learning experiment and that's leveraging tools that are available to uh to operate on top of large data set so in the second stage i have a larger offline data i'm still working in a batch processing manner i have a larger offline data set but that requires specific expertise right not all of us are expert on you know apache spark or ray or all those great distributed systems tools uh frameworks and that requires a certain set of skills it also requires to have a running distributed environment which we don't often have access to however if we work with the different clouds we can easily get one running for the first time price so after we went through the first stages of development and let's assume we're happy with the result we kind of understand that we need to learn a little bit of new tools and leverage some tools that exist in the system maybe work with the data engineers to get it into from the first stage to second stage of development then we want to push it into the staging right we got a model we're happy with the model and now we want to test it again test the integration around it wrap it in code and try it and maybe even try to build that model again on a fresh data if we need to or uh estimate the accuracy and the quality of our model in staging so in this phase we're actually translating the code to a production environment and we're testing it again and again again to understand if it stands with the standards that we expected to stand in when we push into production so this is the staging phase after we're happy with that we get you know go no go if we decide to go with it we push it to production and inside production we still have a similar process of data ingestion transformation however now with the things we're pushing we're pushing the model right we're not put we're not pushing the training phase we're pushing the model and in production we're using the model data score predict analyze or do anything capture or do anything related that get an answer about the user what they're going to do and then we also need to capture the user real action right because we need to compare those two results from what the model gave me as a prediction and also what actually happened in reality so we can monitor it and give and refresh the model when we need to so we use the model for scoring we capture user reaction we sync the data for monitoring and then we always ask ourselves do we need to trigger a new ml workflow do i need is my um is there a data drift is there a model drift do i need to change something so this is kind of like the high level stages that we see in productionizing machine learning and now that we understand it better let's understand what specifically apache spark gives us first thing first it gives us the ability to work with data it allows us to ingest data from different sources it allows us to run cleaning transformation feature engineering and all of that leveraging the very rich capabilities that spark has out of the box and if you're not familiar with apache spark it's a unified analytics engine for big data and machine learning it was originally developed in uc berkeley in 2009 and since then it was the go-to uh tool for big data analytics and anything related to distributed so like i mentioned it has a reach api you can use sql like capabilities you can use uh structure streaming um different capabilities on top of data frames it also has ml library which is the one we're going to touch deeper into today so we say we're still in the first phase we need to run the futurization the future engineering so in sparc mle it gives us the ability to extract inference more features because there's a lot of capabilities already out of the box it also gives us utilities because in machine learning we work with statistics right we work with math so it gives us linear algebra and statistics and kind of gives us a tools like summary tools to get a feel of what's going on in their data so this is great we cover the data now we need to craft the model right so we are in the second step and the third and fourth steps are we need to do the model training model testing and then model validation so this is something we do always uh in the loop and spark emily gives us out of the box those capabilities as well it gives us algorithms that are already built in classification regression clustering very known algorithm in the industry they also give us the ability to construct them and put them together through a concept named pipeline and pipeline is not invented by apache spark it's a known concept in the data science pro so they're actually leveraging the same concept that exists to make it easier for data scientists to use a participant and those the pipelines it helps us construct and put everything together in a nice code so we can work collaboratively within our team it tells us to chew on the machine learning algorithm and also to persist it to disk because they don't want to use it and leverage it for iran but sometimes as the data if we talk to data scientists they actually need more and sometimes spark is just not enough and this is where we want to branch out we want to bridge deep learning clusters with apache spark this is something that we see very common in the industry we see it across different image recognition video analyzing um also a little bit with nlp world et cetera you want to bridge and leverage the capabilities of the new uh algorithms that exist today in the industry when we want to reach out we want to find a way to find a tool that can help us translate those uh data types or different uh data frames that we work with into the data data frames or data formats on the other side and this is where we're going to use uh uber patent storm so uber developed the tool named path storm it's an open source data access library that helps us uh we can use it in a single node or a distributor training uh and basically helps us bridge those uh different data types and different data formats so if i'm trying if i'm starting working on monday my data with pi spark for example the python api for apache spark i can later on even if i save it in some special format that spark has i can later on transit into the pedestrian dataset and pull it in using tensorflow dataset or pytorch data loader for example so it helps me bridge between those two worlds pretty easily so if i have the spark data frame i can use the pedo sporm spark conventor and later on pull that information back into a tensorflow data set pi torch data loader and leverage their capabilities there it also gives me the ability to do cash management i can define if i have specific cash somewhere i can do that it allows me to handle and believe specific types like vectors uh sparks for example the different type of vectors spark has in it and it also helps me migrate from single node to distributed computation if i need to because i'm later on able to for example work with per data frame and if i'm filtering the data out all the way and later on i only need one small machine one node one node to uh to actually do the training i'm capable of averaging that as well so this is a high level architecture of how the workflow works we're doing the etl entity transformer load around us using spark and data frame if there's a cache file that we already defined in the system and we'll see it in a second how we define it we're leveraging it if not we're creating one and later on we can pull the information back into tensorflow for example or the pi torch data loader so how does it work within pedestrian there's a library dedicated for spark and we can leverage the functionality of make spark conventor we do need to set the configuration uh into and give it a specific uh parquet cache layer like and you can see here in the code sample and then we just give it a data frame and call the functions we can also use the different python apis like the with um definition here and bring it run actually make the data frame into the data set that we want to work with and start running some machine learning training on it as well so let's take a look at the very short demo uh where we're going to do image processing so we're going to see basic etl handling small files uh petastorm and then distributed tensorflow cool so i'm actually running this locally on a jupiter notebook but you can write it in your cloud as well or any other environment that you're working with so i'm leveraging a dataset named uh call tech to 256. uh it's relatively a big data set for uh image recognition uh image classification and i'm starting my spark cluster right here and i have two ways of loading uh the data binary or images i always recommend you using the binary file format it's easier to load it's more efficient you can see it's loaded into uh the structure where i have the path modification time length and the actual content running forward there's a lot some operations i'm doing um extracting the label right because we always have to extract the labels the labels are part of the path i'm extracting the labels and i'm starting to run with the feature engineers for example i want to gray out my images so i'm leveraging the different tools that spark gives me for example um you can see it uh i'm i'm using udf functionality uh and you can see it right there applying pandas also the new pandas udf that is faster and more efficient so i'm turning it into a grayscale uh you can see it right there i'm doing it first on an example like we've seen a bit like we saw in the beginning very short uh very small chunk of files first to make sure it works and later on i'm doing it on a larger scale also again we talked about a little bit about the file system the small file system uh the small file syndrome so at the end of making sure to avoid the small files problem by uh saving it uh into a parquet but removing the compressed format because apache spark has a compressed codec out of the box so we're changing it to uncompressed because we don't need it anymore because those are images actually so we don't need to compress it again it doesn't make sense and saving it back to this so this is the first part of working and wrangling the data and extracting those the second part is actually training my machine learning algorithm right so once i have that data i actually want to start working with my pedestrian and i can leverage you can see here i'm importing the peta store and i'm also important to prince for inspect it helps me uh give specific definitions to the data that i'm working with and what i want you to see i want you to see here how i'm setting the cache directory for the data and later on how i'm defining the convent the converter train so remember the beginning i use spark to do a lot of data manipulation uh feature engineering and all those good things but later on i want to leverage an existing machine learning model from the tensorflow world so we're going to leverage a mobile net v2 and since it's a neural network we can always take it leverage the existing model and just change the last layer so we're getting the base layers that already exist we don't have to train everything from scratch we can always leverage it and add another layer for the classification in the end so this is what i'm doing here i'm taking the mobilenet v2 from tensorflow and i am and i'm basically um getting the model itself the base model all this code by the way is available for you on a github public repo so you can look into it i'm taking the base model and then i'm training and fitting it into my distributed system later on so i have the pedestrian pre-process because i want to adjust some of the data to fit my machine learning algorithm as well and then i'm training the distributed model right here so i have the model i'm compiling it defining the different optimizer i want for that specifically um and this is where the peta storm thing happens i'm converting make tensorflow data set and here inside you can already see that i'm leveraging that data set that i converted from apache from spark data frame into a tensorflow data set and i'm training it running it only one epoch you know some accuracy some value lost some you know some more information about the machine learning specifically but what i want you to take away is exactly the fact that you can leverage both worlds and you don't have to uh only be uh you know reduct restricted to spark emily so wrap up uh now you understand a little bit better at the spark and believe component and what it gives you uh you understand the spark ml pipelines how to bridge spark with tensorflow and pi torch you understand a little bit of how peta storm works and also a nice cool demo of how those things connect together all of that exists in my github repo you can check my twitter it's already there on the top and just to give you a heads up about how the book around this is going to work there's uh the book chapter flows uh distributed into three three chapter three parts one of uh first of all working with the data the feature engineering the data processing second thing is the b part how we're training with emily how we can train and bridge into pi towards how we can train and bridge into tensorflow and c is going to be about deployment triggering and new machine learning uh workflow uh so thank you so much for your attention i hope you learned something new today um if you have any questions uh i'll be here later on in the q a and also feel free to reach out on twitter my dms are always in forever open i'm happy to get your feedback and thank you so much you