Scale By The Bay 2021 : Roksolana Diachuk, Marianna Diachuk, Alice and Mad Hatter....

Scale By The Bay 2021 : Roksolana Diachuk, Marianna Diachuk, Alice and Mad Hatter....

Recording: Scale By The Bay 2021 : Roksolana Diachuk, Marianna Diachuk, Alice and Mad Hatter....

so um today we are going to talk about a new alice adventure and uh this time she will be joined with the mad header and uh we're trying to resolve uh the news problem and we'll try to figure out whether we need to predict or not to predict [Music] mariana is the slides visible because uh i don't see whether they're shared it is not yet now it is not yet comfortable now i'm just going to share that maybe it's just got broken okay so basically as rexella said it's going to be another adventure of alice uh which is connected to the previous talks which roxanna gave before and this time alice won't be traveling alone she will be with a mad hatter and they will try to solve yet another riddle with the power predictions i'm marianna i'm nice to meet you all i'm data scientist at restream and uh the main product of uh restream is a multi-streaming platform and we are also hiring so quite actively so you can uh search for our careers uh later after the talk and uh also i'm a data science leader a local branch of women who code community and in my free time i speak at conferences and also write short stories i'm rexella i work as a big data developer at captify captify is the largest independent holder of user search data outside of google and the engineering team is based in both keith and london and i'm in cave and we are also hiring in both locations and remotely as well and there was infusion ambassador there also i'm part of women who call kiev community as uh data engineering lead and i often speak at conferences mostly around scala and big data so as it was mentioned i did a series of talks about ellis adventures um totally different ones and um this time i'm compresenting it with mariana and she will bring more of the machinery in perspective to my more kind of data engineering and backhand side and uh before we are going to just dive into the story i will make a short recap of the previous adventures or previous talks as well just so we would understand better where alice is right now in terms of the plot because they are all connected and also the kind of knowledge she gathered during those adventures so on her first adventure she traveled to the world of plots and higher order functions was a world of functional programming and kubernetes and her first place was functional forest where she met the functors and the monads and also she visited the default namespace city where she met the parts the deployments and learned how to create simple objects in kubernetes and in the process she created this magic database which was a custom object and one of the instances was the pot and also there was a storage volume of all of her knowledge about this world which she decided to kind of steal and take with herself so she wouldn't forget about this adventure and this world and the pot also tagged along with her so then we skip to two years later um ellie's became a successful scholar developer she totally forgot about this trip and uh once when she was walking in the forest just the same spot from which she got to that magical world she found the cave and there was this safe lock which is in the form of the persistent volume and she remembered what happened before that this amazing adventure and also she met the pot which was leaving all of this time in this cave in the forest and was longing to get back home so her new task was to combine her knowledge of scala and kubernetes and to build some custom objects in this case it was a launcher rocket which helped to just move the pot from there from her world to his home and she succeeded at that and uh three months later she became a big data engineer because she was just boarding her back and role and one of her new tasks was to help the data science team to build um the pipeline around machine learning model uh mostly on the serving side but um alice decided to build a whole pipeline kind of ml ops style and uh it was an entity recognition model and she built the pipeline got the predictions so it worked great she learned how to use kubeflow for that and uh one of the times when she was launching this uh pipeline she got this mysterious message that parts and high order functions are in danger and now alice has a new task she needs to figure out what happened to the pots and higher order functions and how to get back to their world and help them to avoid this danger which is to solve the problem that they have right now so now five days passed uh since alice learned about this uh new message elise found herself in a mysterious forest and she didn't know how she got there or what happened there and then she suddenly heard someone calling her and she couldn't understand whose voice was that but it sounded so familiar and it was calling her name and then uh before she even had time to realize who was that that the mad hatter appeared in front of her and they were happy to see each other but they didn't have much time to reconnect although they saw each other many many years ago and uh ellis just ran along with uh the mad hatter to find out what kind of middle he he just got and he wants her help with so they just passed through the forest and found themselves on the medal and there was a caterpillar in front of them who was just waiting for them but uh ellis was not really in use because her previous encounters with the caterpillar were not really helpful in terms of her adventures and he usually was talking in some kind of riddles and which was mostly kind of misleading instead of helping that's why she wasn't really happy about that although the mad hatter was definitely leading her to that place so uh the caterpillar was not offended uh with this kind of reaction from alice uh he on the country said that he was only going to entertain them with a riddle and probably that was her purpose in this world now so it didn't explain much in terms of the world itself and how she got there but she decided just to listen and check it out so somehow the caterpillar just guessed that it was exactly what she was feeling and just said mysteriously that soon it will all make sense and i was really hoped so as well so the riddle was like that what was lost can be found only in the anomalies of this world and um ali started to wonder um what exactly was lost and whether it's connected to the message that she received lately about uh the plots and higher order functions being in danger and uh also she was wondering maybe it can give her some kind of hint in terms of the world itself and the understanding better the nature of this world and some kind of weirdness around that might help her find a way back home as well so she just asked the mad hatter straight away whether he had any ideas on that and somehow the mat header was even had pretty clear understanding of the riddle itself and the way they are going to solve it and that's why he just proclaimed that they need to find anomalies in the data and alice didn't really understand what he meant by anomalies since alice didn't know what anomalies are the mad hatter tried to explain to her in a simple in a simple way that basically anomaly is just some kind of a deviation from the standard and depending on the problem we have on the case we have this standard can really uh differ from one situation to another like for he tried to explain anomalies and on fishes to alice and uh sometimes we have like really many parameters which uh i didn't which help us to identify whether it's anomaly or it's not and for trying for detecting anomalies we can use unsupervised learning machine learning algorithms and as you might remember there is two groups actually a bit more but here we will focus more on supervising unsupervised learning and if in supervised learning we already have a label data like for example here we have a cat and another cat then with unsupervised learning we may have data for multiple classes and the model itself tries to identify how to differentiate them and whether it's possible by the features and parameters that we have so basically uh we can use unsupervised learning algorithm for our anomaly detection problem and in our case it can be kamens which is a clustering algorithm and it basically works in such a way that we identify the number of clusters that our data should be divided into and we can do it through experiments in order to find the best division of data into clusters and uh then we basically have uh the coordinates of of the centroids the center points of clusters and we try to find how close our data points to each of the centers in order to identify whether this point belongs to one cluster or another so for alice it sounded quite familiar and it reminded her of the previous work that she did for the data science team so she figured that the mad hatter was talking about machine learning models and uh he approved that because he was talking about machine learning models and specifically anomaly detection models and for alice it was familiar more on the side of um kind of abstracting away from the model itself so remembering what she learned during her work with the data science team with that there is this whole kind of pipeline around building the model and actually building machine learning model is only one of the steps of this pipeline which has an iterative nature so there are these multiple stages uh also in a very kind of generalized view they can be interchanged and uh just flexibly change with one another depending on the task so where is data ingestion it can be bad streaming data doesn't really matter any kind of framework to use for that the next step would be data analysis we need to understand um the types of data that we have the data sets in general we can have two different streams of data for example stringing and batch data or we can have two different uh formats of the data therefore we need to explore all the types that we have understand it from the business point of view and uh also to figure out the values themselves like for example there can be some zero values which will point out to some records that might need to be deleted in the future and after that we are able to build some transformations on top of this data it can be cleaning some kind of pre-processing of the data like removing the records uh changing the data types if it's needed which often happens uh with the values of uh timestamp format and also in the same stage we are able to build some features like for the future of building the machine learning model and after that we can start building the model training it um as soon as we build the model itself we are evaluated validate the results and we can get to the serving of the model but again it's uh the process itself is quite flexible because when we are working on the model for example we can figure out that actually we need to build more features and we can get back to the previous step or we can figure out that we need to analyze the data better and add more features or add one more source of data so basically we can get back to one two or three steps before and it's totally normal to actually repeat the cycle as well and uh to rebuild the model or enhance it in some way so if we abstract away again from the model itself from the algorithm within the model um the solution to this problem would look like this we have some source data given by the caterpillar some kind of numbers and ranges and we need to build the machinery and model we abstract away again from the fact that we are pre-processing the data transforming it we just going to put it into this one stage of building the model and the result of it will be detected anomalies in any format that you wanted to see so the math header asked how exactly are they going to solve it together considering that they had different perspectives and different experiences and alice was thinking of the way she collaborated with the data science team and what she figured in the process is that it's okay for data engineers to take some part of the work and collaborate with the data scientists on their part of the work so in this pipeline image again we can see that there are some green squares for the data engineer work for example data engineers would mostly work on ingesting the data uh building some transformations like cleanup pre-processing and actually serving the model which is more on the infrastructure side of things while data scientists work on analyzing the data which would help them to build the features as you can see data transformation is kind of divided between the two so they are building the features usually and then building the model itself and evaluated in this case um as i said the data engineers communicate with data scientists without getting into the algorithm itself kind of viewing it as a black box while data scientists get some pre-processed data that they need and also they perform some work on the model return it and don't really care whether it's served in some way which is more again on the infrastructure side of things obviously it can work differently but in this case it was a perfect solution to combine their perspectives and also alice proposed to solve it in scala and not only in scala specifically using spark because spark helps for people with pyson background to kind of move easier into building the model and combining both data engineering data science work and at the same time spark sometimes is kind of similar to the way we are writing in python just same as in scala so therefore it's kind of easier to transition to another language so um ellis decided to take first the part with ingestion of the data and she decided to work on the rdds just to explore how it works on the lower level so she just took up the file that was given by the caterpillar removed some unnecessary values and as a result got the vector which is basically an array of floating point numbers and the labels of this array or columns and also she cached some data for better performance of the spark application itself and the next step step would be to normalize the data and alice didn't really know what normalization meant although the mad hatter confidently suggested that they should do that right before training a model and getting predictions and he decided to explain it that basically normalization is a way to make the model more accurate and by that he meant that different data ranges of different features wouldn't influence the results if you if you try to smooth down the data and for example here we can do it through standard deviations and mean values but also there are there is a lot of other ways to do that and there is no like one specific just right way to do that but basically you normalize the data not just for anomaly detection problems but in general to avoid the model from to prevent the model from focusing on the magnitudes of the features instead of their actual contribution and uh for clustering specifically um normalization is also important because uh famines tends to predict rounded clusters and predicting the values which have like different magnitudes might might be confusing and we might get not accurate results so basically that's how the first part of the solution was done and they started to wonder what what can be done next and alice suggested to work more on the pipeline itself so basically uh in the pipeline for anomaly deduction spark context is initialized as an entry point and then the data is loaded which is already normalized because alice worked on data ingestion and preprocessing before model training and model then trained uh is trained on normalized data and we also need to set the values of of the centers of our clusters and since we are doing anomaly deduction here we can just predict one cluster and any point which deviates from this one cluster is considered to be an anomaly and it's a standard way of working with commits by defining that the center will be in zero but it can be moved anywhere you want to and you can like change that over time also distances uh are calculated uh for our data points as the math hatter already described a bit earlier uh that basically every point is evaluated for its distance to the centroid and since we have just one cluster we compare our data points to just one centroids and also we have a threshold and a threshold is used basically because we can identify how far the point can can be can be located away from the clusters center and it also helps us to see what's the anomaly in there so now they needed to actually train a model after working on a pipeline and model training is uh basically about initialization of communes and we already have our data pre-processed as vectors and we set number of clusters which is uh necessary for coming's algorithm and then we rerun our model several times why would why would you do this that because um basically commence is very sensitive to the order of objects that's why it's better to randomize that and to rerun the model several times and then we just get our predictions and we can work on the next part of the pipeline so let's dive a bit deeper also into the function which is about calculation of distances to the centroid it's it's basically the key function which helps us to implement the essence of comings and it's about actually finding this distance which will help us to see whether the point is an anomaly or it actually belongs to our cluster of the of all other data points um so the matt heather asked how they are going to run this job since it's a spark application and it can be executed in multiple ways and then alice uh confidently support suppose that they should use uh kubernetes and uh actually the mad hatter was not really sure about this idea since um the scale was kind of small to use kubernetes so it seemed like a bit over complication of the task but alice considered that kubernetes is always a good idea so why not explore it in this environment as well so just before we are going to see the actual implementation a bit of the background behind the spark application working on kubernetes specifically this native support of kubernetes so the client kind of submits the spark submit script that is written by the developer and it works in such a way that this script kind of is a sent as a request to the kubernetes api server and same as the simple kubernetes objects um api server considered that we need to create this kind of object described in this script and it will be like spark application and therefore api server authorizes this request finds proper credentials and then working tightly with the scheduler actually schedules the spark driver and the spark driver is usually is kind of in the form of the pot so as soon as the driver is scheduled we can also see in the logs that the driver asks for more resources for the executors and they get scheduled in their turn so basically that's how spark application would run with native support by kubernetes and spark submit script for this application would look like this so first of all the main parameters that we always use when we are running spark submit it's master so in this case it's going to be the link to the master of the kubernetes cluster with the host and port kind of immediate here for security reasons um another configuration is deploy mode in this case it's going to be a small cluster therefore it's cluster mode and most of the times it is cluster mode quite rarely we use some standalone local modes um another configuration is the naming in this case doesn't really matter just it's easier to distinguish by the name when we have multiple spark applications running in the system so let it be anomaly detection um another configuration is a class which is the name of the main class that we used for the whole pipeline where we have the main method containing because it allows spark to figure out how exactly it's going to run the application kind of pointing out that here's the spark context where we initialized it and here's the main message that's going to kick in and two more configurations are specific for running spark on kubernetes so we have service account name um which is the name of the service account we set up in the cluster it allows to authenticate the spark application within kubernetes without it it's not going to schedule any spark applications and one more configuration is the container image in this case it's going to be spark image to understand what kind of containers we are going to run spark containers and the version the latest so in the end we need to point out to the actual application we are going to run in the jar file format so it's packaged applications that were built with some name anomaly detection the version of spark and pointing to the local file system where it's held on the cluster so the result they got look like this it's just a bunch of layers and ones ellis said that uh they are just random numbers but uh the caterpillar disagreed with her and he uh said that they are not random at all and he tried to give her and they might have her little hints that his numbers are not the ordinary numbers but they are binary and it made alice wondering what she can take out of that information and as she started to think more about that she realized she got an idea and she said i think i know how to and then ellis woke up on the chair in front of her tv um she realized that she fell asleep on the halloween night watching some horror movies and also she figured that she had some really vivid dream about traveling with the mad hatter and some caterpillar riddles machine learning which was a very weird mix of things but also she remembered that actually there was some kind of riddle in the end of this uh dream that she was trying to solve and that she figured somehow how to do that so alice tried to kind of step back and uh figure what her thoughts were at the time and she remembered that they were talking about different presentations of the same information and the caterpillar said something about a binary presentation and that's when she realized that uh actually her idea was to present the same information in some different format which is basically more readable but she also didn't really remember the actual result like the bunch of heroes and ones she couldn't just memorize that and at that point alice realized that she was holding some kind of piece of paper when she woke up so alice looked at it and saw that there was this message written on it that the past is the key to the future and that's when alice realized that actually she needs to look for some kind of solution uh to the danger that happened to the parts and higher order functions in the past of their world and somehow it's going to point out to the solution of this problem to the problem itself and will help her to save them so the story is uh to be continued and uh thank you for attention yes thanks for your attention also happy halloween to you and uh we you can follow both of us on twitter you can find our previous talks speaker deck and we will share these slides there as well and also you can find the slides from the previous alice journeys that rook solana presented before at her speaker deck as well yeah thank you and we'll be waiting for your questions around that thank you