SBTB 2019: Prashant Sharma & Nick Pentreath, Deploy end to end ML pipeline using Apache spark ...
[Music] okay hello good afternoon everyone it's really great to be here let's go by the bay our first time at this conference today we're going to talk about deploying into in machine learning pipelines with apache spark kubernetes kafka and my last exchange first a little bit about us is my colleague Prashant IBM code a group and data in AI technologies we are spark emitters and BMC members my interest area is park structured streaming and spark and kubernetes and i'm nick country's ml neck on twitter and github was at the center for open source states and AI technologies focusing on machine learning and AI frameworks and open source projects author of machine learning with spark book a little bit out of date now but hopefully one of these days it'll be it'll be updated so before we get going a little bit about code a we're an open source team within IBM over 30 open source developers and our mission is to enhance the state of the end-to-end enterprise AI lifecycle hard an open source so when I joined the team a few years ago it was mostly focused on Apache spark and since then the focus has really expanded to include the rest of the Python data science stack deep learning model and data exchanges which would be talking about a little bit today as well as projects for AI fairness and expandability and model deployment open standards and frameworks for serving so we will start today talking about the machine learning workflow and the common perception is you know you just throw data at machine running and you get out profit right you you you you're done but this workflow is a lot more complex right so it spans and silos starts with data which is sitting around in various locations some of it is historical some of its arriving in real time streaming is kind of be talking about today and that's the province of your data engineering teams then the data scientists and researchers are taking that data ingesting it into their workflows doing data visualization exploration taking the data that is in RAW format and transforming it into a form that can be consumed by machine learning model that's data transformation feature extraction feature pre-processing and engineering and then that little piece in the middle of model training which is what all the data scientists and researchers really want to do that's a quite a small piece but important selecting the best models testing it on data doing the evaluation and figuring out what is the best approach for a particular data set and problem and that's where a lot of the discussion often stops but there's no point in having machine learning model that's trained on data and is actually going to make predictions in the real world so the next phase is deployment and actually throwing it over the wall from these data science research teams into your machine learning and production engineering teams and getting it to deploy it into a live system actually predicting on new data where it has to be monitored and most of the time you have a feedback loop so the model itself is generating new data it's it's performance is changing over time and you have this kind of feedback loop back to the beginning that data data and ingest phase and that's again the province of your machine learning production engineers the workflow versus spans many many different tools a whole zoo of tools and it's a small but really important piece of the whole puzzle so this is by now quite an old diagram from an old paper by Google the hidden technical debt and machine learning systems but I really like it because it illustrates that entire workflow is only you know perhaps 5% of the entire problem and really what we have to worry about is interfacing this machine learning piece with all of the other different pieces you know in particular for deployment of machinery models the serving infrastructure the monitoring the analysis tools streaming systems and so on so before we talk about your streaming side you just talk about this model deployment phase and when you talk about model deployment we have to answer three questions the what's aware and the house so what are we deploying what do we mean by a model where are we deploying it is it is a batch environment streaming real-time are we making real-time predictions in support of a web service that matches your scoring a bunch of data and how we're deploying it what is the actual DevOps mechanism how are we actually getting it into into production what is the software deployment mechanism so starting with the first one what do we mean by a model now we saw that machine learning workflow most of the time we think okay once you've got a trained model that is the model right it's an algorithm it's a bunch of numbers which represent the weights of that train model and that's that's it we're done but if we're going to look at that workflow that comes before the model training we we know that to take raw data we can't just feed it into a machine learning model it has to be transformed into a bunch of vectors bunch of numbers and we need to take the raw data and plot feature transformations extraction pre-processing now that's what we do a training phase and if we do not do exactly the same steps a prediction phase we get garbage in garbage out so that's that's known as train test time skew and we need to ensure that the entire pipeline is the same at prediction time and what you've always heard about deep learning is that it does away with this need for feature engineering right we just throw our picture our image at our fancy neural network and out comes the correct prediction right but in reality again a deep learning pipeline is a lot more complex so yes at the core inference phase is using our favorite framework and that's our fancy to know deep deep neural network model the latest and greatest from research but before we can feed it into that framework that requires a tensor or a set of vectors we need to pre-process the image and then once we get to the prediction out of that framework we need to do post-processing so the image pre-processing is typically done in the deep learning land it's in Python OpenCV tensorflow hello and then even once we get the prediction outs we got to do a lot of post-processing which is typically custom code so there's a lot going on here which is you know a lot more than simply the core algorithm so we need to think about deploying pipelines and not models so one way of doing this is to think about using containers and containers have become rightfully so really popular way of deploying software gives you repeatability ease of configuration in particular for this application we have the separation of concerns so data scientists can worry about the what they can use their favorite framework they can train intensive or whatever they want to do and they don't have to worry about how that's going to get deployed they're just packaged it up as a container throw it over the wall and okay you know UPS teams and machine learning engineering teams running production infrastructure can just deploy it and you get some stuff for free monitoring and fault tolerance and so on but that question about the what is still really important so what goes in the container does matter because performance matters so if you just throw arbitrary Python code in there you don't know what performance may or may not be and you had SLS to worry about and it doesn't solve this problem of how you interface with that model so one said microservice let's say that model has been deployed as a docker container how do you actually call it how do you actually incorporate it into the rest of your the rest of your application stack so this is one problem that we're trying to solve within our team with a project called the model acid exchange so the steps that one needs to go through to actually find a machine learning model that works is you've gotta go out there and see is there an existing model that does what I need so if I'm doing image classification what is out there is it performance enough it doesn't meet my use my use case you can actually use it from a licensing perspective if a model exists out and researched and I need to go and find the code for either training or deploying that model so is it out there an opensource a lot of it is research code is it actually production-ready some of it may work may not checking licensing if I'm training the model then obviously I've got to use all the different frameworks worry about data licensing and what we're really worried about today is deplane and consuming so I've got to make any adjustments to that code packaged it up deploy it and again this requires a you know a lot of time expertise and resources and it doesn't necessarily solve that a standardization problem how do I know how to actually call this model so in the model acid exchange it's a free and open source collection of deep learning models or state of the art deep learning models across various domains so it's one place to come and find all of these models wide variety of domains we couldn't test the code in the IP and ensure that it's open source licensed and that it all works for production use cases and we have pre trained versions off the state of the art models that can be deployed as well as you can you can train them yourself so we go through all of this all of these steps we are eliminating this these barriers to entry that existed that we saw before we've gone through all the steps already we found the model validated license terms wrapped it up in our framework and probably provide a standardized REST API publish it as a docker image on docker hub so that can be you know one-click deployed into communities clustered for example so deployable asset takes this data model compute and expertise that puts it together with the processing and the complete power of course and you get a pre trained model but what's important here is that the deplorable as it on the on the exchange is both the pre-processing and post-processing code that we that we saw before as well as the model and standardized REST API so you get something that is easy to use and it speaks JSON and you can easily incorporate it into application so as much as possible across different types of applications we use cases for for the models we standardize the way that it works so you can you can kind of drop in replace different different max models and they will sort of speak the same API so going back to at work flow we saw that there's a lot of different steps and spark streaming and yes spark ecosystem can really solve a lot of these problems so for ingest data processing the feedback loop we can use spark streaming and message queues like Kafka for model training we can use whatever framework we want and for deployment we've seen that we we can have a standardized container based micro service that exposes a standards-based API with the model asset exchange so combining all of these different components we get a really good solution so Prashant will talk us through some of why we want to use some of the other components spark streaming Kafka and communities yeah Thank You Nick for explaining how mortal acid exchange makes it easier to deploy machine learning models moving to why we need Cuban eddie's spark structure swimming and Kafka so Cuba nowadays is an open source system for container orchestration you know it offers scaling automatic deployment management of containers of applications and it provides a lot of convenience features on top of that not going to the all the details of it why we chose cuban at is because we can scale dynamically we can just say change the replica count from X to Y and kubernetes steers the ship to move and meet my requirements I don't have to worry about load balancing all services are load balanced decouples you know infrastructures and scaling makes things easier for your developers to deploy above all spark you know you can use cuba Nettie's as a resource manager so thanks to the spark on cuban artists effort it makes deployment very easy spark can more intelligently scale itself it can make a decision on how to elastic scale based on the amount of profit asked pending and so on and there is another advantage that since the entire system is running inside Cuba nowadays there is no need to expose any service to external this makes things both easier in terms of logistics and also in terms of isolation and security so this is one downside Cuban IT system is pretty complex and this you this challenge comes when we are actually maintaining kubernetes ourselves but most likely if I need a kubernetes cluster I'll go to Google cloud or Amazon or somewhere so I don't hit this problem in that case but if I were to operate it myself like while developing this demo for this talk I was trying to set up my own Cuban Aris and I got to understand this next is spark structure streaming given the popularity of spark it's already well known it's a scalable fault-tolerant stream processing engine built-ins spark sequel lets us write a very expressive code and concise functional and above all it makes it as I said writing streaming job without having to reason about it so the same code that can run a batch job can also runs almost the same code Canada on the streaming analytics job makes it very easy to work with so the entire data set in the batch case can be viewed as a table and in a streaming case it can be viewed as an unbounded table so kafka is well known for you know processing large volumes of data it's distributed fault-tolerant and spark streaming plays really well with Kafka it has a built in connector for Kafka and on top of it offers performance and reliability guarantees so going forward we built this demo for twitch stream sentiment analysis we are using a Kafka message queue and we are this the stream itself is of popular airlines in America and there is a machine learning model to process the sentiment and spark streaming does both trying to simulate a stream for the demo and also perform analytics we will go go into that a bit about the data set its source from CagA it's a one month of tweets on travellers there's a sample you know United and American and so on and we are using the model asset exchange text sentiment classifier we can briefly talk about that thanks so yeah given that we're using an L and sentiment data set we'll probably expect that it's going to be called negative sentiment hopefully there'll be some positive stuff in there so we saw before how the model asset exchange gives us a one-stop shop for various different use cases and as it happens on Macs we have a tech center pin classifier and for anyone who knows a little bit about the latest advances in natural language processing and deep learning they'll know that a class of models called the transformer and pre-training very large language models with with these architectures is kind of the state of the art in terms of what you can do in you know P domain so the tech sentiment classifiers based on Google's Bert which is effectively taking these state-of-the-art pre trend so instead of our transformer networks and pre-training them on a huge amount of data and the idea is that you take all of this english-language data you know Wikipedia and kind of the whole internet pretty much and you you train this model on tasks such as predicting the next character predicting the next sentence based on what's gone before and in a completely unsupervised manner this results in a really big model that's really good at doing a bunch of stuff in in natural language without necessarily having to tune it to a specific specific use case but what you do tend to do is then fine tune it a little bit more so for text sentiments for example it's a classification problem it's tuned on and IBM data set from the IBM debates a project where they actually created a and that could debate against a human and this is the claim stance data set so it's a bunch of small text annotated with whether it's a kind of positive or negative sentiment and what's interesting here is that even though the model has been trained on something that's not tweets and tweets and social media have a very specific distribution of words and lengths and and the kind of language in there and characters for example even though it hasn't been trained on that particular data set it actually transfers really well to doing to performing pretty well on on small data sets pretenses that may not be the case if you look at trying to do sentiment on very large tell us it's like long movie reviews for example but in this case it works pretty well so the idea here is we spoke before is that max creates this really nice API we just feed a JSON with the text and we get back a JSON that specifies what is the predicted probability of being positive or negative so here we see you know a spark streaming is great well that's a positive statement you know and I think it's not my favorite framework is a negative statement so it does a pretty decent job at that kind of arbitrary text now one point to note is if we're in a streaming environment you know this kind of setup is is good for real-time right I mean with you you give one input and you get one output it's it's it's behind an API to microservice so it could support a real-time application pretty nicely if we were moving on to streaming unless we kind of record at a time we're in a mini-batch environment typically that's the case of spark screaming instruction screaming so in this case what we really we may not care so much about latency I mean we still might be in the few hundred milliseconds latency or even a few seconds but typically we're not we're not you know hard real-time so we don't care that much about latency but what we care about is throughput so what we really want is to be able to patch things up a little bit so fortunately for us machine learning models in general do pretty well with batching they they're kind of designed to be trained on many batches of data and prediction time you can actually speed things up and get a lot more throughput by doing batching so if we put together a whole bunch of documents and feed that through to the model fortunately we get a batch response so this fits really nicely with our use case of doing predictions on mini batches within streaming so we can now use this max text text sentiment classifier it's already packaged up as a docker container since we're running everything else in communities so we can just drop it in Thank You Nick for explaining how we can do model scoring or predictions using max text sentiment classifier the best part I like about it is that it exposes the REST API I don't have to worry about how to deploy the model or how to do you know a lot of stuff that goes around inside it so getting on to the steps that goes through in the entire streaming pipeline first one since this is a demo and I needed a stream to work with so I have a spark job which uses the rate rate source that lets me specify you know how many rows per second I need to generate so it's just a generating job it randomly chooses from the data set we spoke about it randomly chooses tweets and sends them to the Kafka service next this second spark streaming jobs job which does the analytics and model scoring using max text sentiment classifier as we talked about so since the streams these tweets are coming in real-time so as they come for each for batch of tweets or each tweet I make a service call to the tech sentiment classifier service gets the center and perform the analytics and displays it so this is the entire cycle that goes around and the whole thing runs inside the kubernetes so the best part about that is it makes scaling this very easy so if I were to suppose increase the rate to you know a million rows per second then all that cuban artists has to do is increase the replica counts for the MEC micro-service in question and the number of executors for spark and whole thing can scale seamlessly we offer a single click deployment you know you can to try this demo out it's working demo you can try it out clone the repo and just run this command I have a brief video of the Tamil yeah so after running that command I check the parts that are created so the first item happens to be the executor for data generating job the SPARC data generating job and then there are three instances of the microservice that does model scoring from IBM that max text sentiment classifier service and then there is Kafka and zookeeper to do the message queue and the SPARC ml pipeline is the actual analytics job this one so you see the results since it's printing to the console we just need to see the logs here we see yeah so here we see tweets and their corresponding sentiments and there is also an overall aggregate of you know so-and-so airline did got so many positive reviews and so many negative reviews so not going into the details of those next if you if you like to try this out I have this open sourced your contribution of you in the form of feedback is very much welcomed a bit about what goes behind the scenes for the further and for this demo and this is a simplest spark job which is like spark by example so if I were to do this I and using the same demo repository em I worked with I would just specify the cluster configuration for spark and say you know deploy more this cluster and this is the class I wanted to run and so on and rest I leave it to the defaults then I say spark job deploy deploy by iPod this is very helpful because if you're working in a client mode and your system your local laptop is behind NAT or firewall the the the workers cannot connect back to the left on the local machine and that leads to all kinds of problem so we deploy why iPod I spin a part that has a spark image and that part deploys for the this does a spark submit which spawns further executors and I'm starting the job or cluster is as simple as saying thoughts start and then we can wait on it for like like in this case I'm waiting for one 120 seconds thank you very much for your audience thank you everyone I would thank you push on for very good demo I'll just encourage you if you if you want to know more about Cote and what we do and all the projects that were involved in co.org obviously follow on github on Twitter and if you're interested in the model acid exchange spark on cube any of the topics either come and speak to us afterwards I don't if you've got time for questions now we might have time for one question as I yeah I think just take it offline and in a way if you want to follow up on anything thanks very much thanks [Music]