Scale By The Bay 2021 : Jean-Yves Stephan, Apache Spark Performance Tuning Session with Delight
it's my pleasure to be with you virtually with you for the second year um i hope next year we could do it in person and uh yeah you already introduced me uh but uh indeed i started my young career as a software engineer at databricks uh leading their spark infrastructure team then i co-founded data mechanics and in june of this year we were acquired by spot.io and so i'm now a product manager basically we're we're launching a product called ocean for spark which is basically the spark platform we had built at data mechanics uh plugged on top of their managed communities offering called ocean uh so really it's a pleasure to be with you today this talk is not a commercial talk we're going to talk about something else called delight which is a free and cross-platform monitoring tool that we developed so first i'll just say a few words about spark and data mechanics in general to to set the stage but then we're going to talk about delight okay what is the problem that we want to solve what is the vision how does it work in practice and uh to really show you in practice we're going to have a bit of a of a live demo where i'll be showing you some of the delight screens and showing you some real customer pipelines and some insights that we can get from the light to to increase the performance of these spark pipelines if we have enough time at the end i prepared a few additional slides that's really a bonus like maybe the dessert um because spark 3.2 was released last week and i'm kind of excited about that release but we'll see if we don't have enough time uh i'll i'll redirect you to a blog post that we wrote um very high level intro about spark i presume most of you will be familiar uh it's the number one you know distributed analytics and gene what makes park popular is first that there are high level apis in python scala and sql and that are pretty mixed part pretty easy to write okay and the most popular api by the way is definitely python and the second thing that makes park really popular is that it's very versatile so you can use spark to do business intelligence and interactive data exploration machine learning and data science to obviously data engineering whether it's more like batch etl workloads or whether it's real-time streaming workloads um so you know spark is easy to write however i would say that managing a spark infrastructure and debugging a spark application when there's an issue is hard and this is why i decided to create a company to make spark a lot more developer friendly and more cost effective so these are the problems that we're trying to solve really at data mechanics make sure it's it's not as tedious to get started um make the platform more open make the platform more cost effective leading to lower costs and making it simpler to debug spark applications and delight is a specific tool okay that that we developed and tries to help with with this pain point before talking about delight just just want to talk about spot and data mechanics at a high level you know spark can now run on top of kubernetes um this ability started with spark 2.4 a couple of years ago it was officially made production ready in ga this year with spark 3.1 and i really think it's a game changer for the spark community and that you can you can finally have the same kind of developer experience and also a cost effectiveness that you have with spark as you would have with the rest of the software engineering world and at a very high level you know our platform lets you interact with spark either by a ui by a rest api by connectors to specific schedulers like airflow is very popular but we support other and obviously by running jupyter notebooks and then you know we manage the backend and the back end is a kubernetes cluster in your cloud account amazon gcp azure that is automatically scaled that also has some automated configuration tuning and yeah some optimization to make spark stable and cost effective but again this isn't about a commercial product this is about delight and delight is a completely free tool that you can use actually on top of any spark platform so you can use the light on top of data breaks emr sparkle kubernetes open source whatever and obviously if you use our platform you get delight natively integrated so what was the pain point why did we develop something today when you want to monitor what happens in spark mostly you will be the only tool really that you have is the spark ui and this is a screenshot here of the spark ui uh i would say that um you know there are a lot of numbers that jump at you when you open up the spark ui but there's a lot of noise and you need a lot of expertise some people would say tribal knowledge to figure out where to focus your attention and to know what's wrong or what isn't the second issue is that there are a few system metrics so if you want statistics about your node or your executor memory usage cpu usage io usage usually you will use a separate tool like ganglia or datadog or grafana and prometheus and the problem is that these tools are general purpose they were not built for spark and so going from the start driver to different executor is not you know native at all if you want to correlate these the metrics that you see on this tool with your spark code it's very complex basically you'll have to jump back and forth between the spark ui and your tool and you'll have to match on on a certain time stamp um and the last pain point is the spark history server itself which is you know the server that can render the spark ui after an application has run it's kind of slow and unstable so to solve these problems we developed delight so delight first gives you these metrics that i was talking about memory cpu io and they captured within spark and they're displayed on the same timeline as your spark phases so that you can correlate oh you know the memory has increased right while i was starting that job that does a certain join or a certain shuffle second we hopefully with our ui we make problems obvious if something is wrong and we can tell it from the metrics it should show up very clearly oh you have an issue with parallelism you have a big issue with memory and the next step is to actually give automated actionable insights and recommendations that's not fully implemented yet but that's the vision and finally it's easy to set up so delight basically consists of an agent running in the spark driver and then the back end is fully hosted for free on our side for free and um and so it's very you know you can get started in a few minutes quick history about delight it started about a year ago with basically a blog post detailing our vision uh in november last year we released the mpp and in april this year we made it uh public ga it works again on top of any spark platform and the development doesn't stop here i mean actually since april we we added a few features and there are ambitious um you know plans for delight including these automated recommendations that i mentioned also making the light accessible in real time while the app is running because today it only becomes available a couple of minutes after the app is finished and adding new pages to the to the delight ui including a page for a stage and a page for the driver um okay so this is the vision now how does delight work in practice there are two pieces there is the spark agent which is an open source agent running within your spark application within the driver it implements an interface called spark listener so it's basically a jar that you attach to the driver and this agent will be streaming will be sending some metrics to our hosted backend i should say two important things these metrics are encrypted okay you provide a certain secret that will help encrypt them so that they're they only become available to you and the metrics that we send is not it's not data that we try to exploit they're only visible by you and in fact they'll be automatically cleaned up after 30 days um and so the back end is on our side and all you need to do is you know collect to connect to delight.datamechanix.com and you'll see uh the web app dashboard appear so if you want to get started you just create an account on dlight.datamechanix.com once you create an account you can create what we call a token which is the secret that will um help us identify that it's you running the app and and make sure that your logs are only visible to you and then you will uh install the open source agent and here we have slightly different uh installation instructions based on your on your spark platform and you can you can find them on our github page and that's it and then your spark applications will start showing up on the dashboard as soon as they're completed um i wanted to give a couple of example so here's an example um of how to install the light on top of data bricks one option is to is to configure an init script which is basically a bash script that runs on your database clusters and early on and this script and what it does is it installs uh it downloads a jar and puts it on the driver and then it inserts a couple of spark config including the the secret token that i mentioned earlier uh and if you want to install the light on emr that's pretty similar uh you just submit a few uh sparks mid options including this package option which will uh you know where you provide a maven coordinate and and this will download the delight jar and a couple of configs um like the secret to to you know encrypt your application logs and obviously there are similar instructions if you're running on a data proc if you're running on spock communities open source really any any commercial or open source platform is supported so once you've done that and you've run an application once this application finishes it will show up on the light it doesn't show up in real time okay we need a couple of minutes to uh to process the logs and make them available though we're working on making this available in real time while the app is running so on this dashboard there are a few things that should kind of look familiar like the name of your spark application whether it completed successfully or whether it failed when it started the duration the the volume of data that was read and written by the spark app and there should be a few um columns that you might not be familiar with like spar task cpu and efficiency these are high level stats that will help you track your costs you know understand am i using a lot of compute resources and am i being efficient or am i being wasteful so maybe let me explain these three statistics on an example so the cpu column is actually the the total cpu uptime used by your spark executors let's say that you have an app that runs for an hour and that had three executors with two cores each then that would represent six core hours of cpu uptime so these are really the your compute resources and this metric is proportion kind of proportional to your cloud costs the second metric is the sum of the duration of all your spark tasks so if you remember a spark task is executed by one cpu core at a time so here on this example you have one cpu that maybe ran a spark task you know for a few minutes and then that core was idle it didn't have any work to do for a longer time well maybe here the the core number three had a spark task to run for the entire time and so here you can see that even though you provision maybe uh six hours of cpu resources these cores were only used for a fraction of that time 72 minutes in my example and now the ratio between the two 72 minutes divided by six hours this is the efficiency ratio 20 here and so this is the fraction of the time that on average your spark executors were running spark tasks and so obviously um a higher efficiency is better and actually 20 is a bit wasteful and if we and usually when the spark ui looks like this i mean the reason is that you had one task that took a very long time to run a straggler maybe was processing more data because of the data skew and so this could be a cause of a low efficiency so maybe going back to the previous slide here we had one app we actually it was called notebook that had pretty poor efficiency and we had one app called daily etl pipeline that had a good efficiency yeah one note maybe about notebooks on average notebooks have a lower efficiency than programmatic apps because you know you interact with spark um in an interactive way right you you run a cell you look at the results you edit a new code before you run the next cell and so there are a lot of breaks where your sparklers are not doing anything so that that could be obviously one of the reason okay so now that i've explained uh really the high level about how the light works i want to show you concretely give you show you live screens of the delight ui um and oh yeah before i do because we started talking a bit about efficiency i just want to say what are the most common causes for a low efficiency so one common cause is the lack of dynamic allocation meaning you have a fixed number of executors while it will be more beneficial to have to be able to add and remove executors dynamically this is particularly important for notebooks because you know you might start running a notebook and then go into a meeting for an hour and so you'd like that the resources associated with your notebook scale down while you're away you could be provisioning too many executors you could be defining too little number of partition we'll see an example like this or it could be you know task duration skew et cetera et cetera so now let's let's start looking at this on real world example and so we're going to uh to look um at really customers using delight the united nations ungp program is one of them and i want to because you haven't seen that screen yet so i want to use that screen too to explain what's going on in the light so first here you can see uh some high-level statistics and that i mentioned earlier how much data was read and written duration spark test cpu up time and efficiency you can see that indeed spark task is about half of cpu of time so the efficiency is about 50 and here something pretty interesting is you can see that some spark executors were constantly added and removed so this is an application that was configured with dynamic allocation probably was a notebook and so um that's why you can see that the total number of cpus of course allocated to your to the app uh goes between zero sometimes to uh one and two actually a maximum of 10 cores here and then the app scaled down again actually to a size of zero so only the spark driver was running and then you know maybe that person took a quick break and then 15 minutes later it ran another query and again we acquired executors so again this is really the the only cost-effective way to use spark in an interactive way it's to configure your notebook so that it automatically scales up and down um another thing that we can see here is the peak memory usage for each executor here the peak memory usage for each executor is about 80 percent of the total capacity which is pretty common which is fine you can also from here have this view over time let me find an interesting executor okay executor 5 for example that was used during this this portion of activity if i find it here i'll see that the memory usage was initially somewhat low around actually a gigabyte and then it increased to about 2.3 gigabytes here you know nothing bad happened but this is an interesting screen where you can debug at which point in your application um you run into an out of memory error and so you could correlate oh this occurred during job five and stage five and so you could you could dive deeper okay when i click here this opens up the spark ui so under the hood we run the spark history server okay so it's still possible to open up the spark ui from here and you can see that oh okay this application was was converting a spark data frame to pandas and so oops and so this is why maybe the memory usage has has increased because we were collecting some data and this is why also it was it's a it's a stage if i zoom in it's a it's a stage that was entirely cpu okay there wasn't any i o going on because i was just converting my spark data frame to append as did frame okay so i hope this was useful i'm just going to close this because the next demo are going to be on a slightly different environment so we're now going to move to another uh customer who who uses data mechanics for kind of data engineering pipelines they're they're a weather analytics platform and we're going to look at a different type of of issue that we can troubleshoot with with delight here we go so if we if we look here at the main statistics we can see that the efficiency is 40 and this is actually an etl pipeline so 40 is is low i mean it means that you're wasting more than half of your compute resources now when you look at the breakdown you see that the reason why it's low is because yeah for half of the time a portion of the cores were idle and when we look here at the at the timeline we can see that from this time so around two minutes until this time around nine minutes uh all this gray area were sparkles that were um you know running they were provisioned but they were not actually a busy running spark task they were idle and so if we want to understand we would click at that specific stage and see what's going on and see oh this stage had only two tasks even though based on the the delight ui i had 16 cores provision so this is bad so here the next step would be to look at the code and see why why are there only two two tasks to partition did i repartition did i make a mistake in my spark configuration and so this is how you could you know improve the efficiency of this app maybe forcibly increasing the number of partitions before running that stage so i hope you know this shows some of the value because again this is the kind of issue that becomes really very visible very obvious using using delights and if you just look at the spark ui you really would spend a lot of time to actually realize oh this is what what you should focus your attention on all right i want to move to two last example one of them is an example of a customer that was suffering from slow shuffle and we did again we took an action based on the feedback from delight so um this app efficiency was good 86 percent so this the spark codes were rare running tasks most of the time but when we look at the breakdown we can see that most of the time was spent doing shuffle and you know shuffle are the pretty um pretty expensive uh phases of your application where you you exchange data across the cluster uh you want to minimize shuffle um and here it was very clear the truffle happened during stage 46 and 47. um what we recommended is so when when you when you write a lot of data when you shuffle a lot of data the bottleneck can be actually the disk throughput okay because you're going to be writing the data to the disk and then you're going to be pulling the data to another executor so a common technique when you have a lot of shuffle is to have a faster disks and so in this case the customer used to be running say ebs volumes uh elastic block store and instead we we have them use an instance that had a local ssd disk and so spark would start using the ssd to write the shuffle data rather than the ebs and um and the change was huge okay we went from an app that used to to run in in a couple of hours to an app that now runs in 26 minutes and the shuffle section is almost gone okay so um we don't spend the neck became something else we see here by the way that some stage actually uh failed and had to be retried until they eventually succeeded so um there is still you know something else to to improve on but already the app went from a duration of over two hours and a half to a duration of 26 minutes so it's it's a huge savings in terms of costs and you know sometimes it's not just about cost it can be just about an application running successfully and not and not failing i want to maybe show just that example should be very quick just to show you some example of either running r or running uh python and how it shows up in our application i think this one no all right this one was running r and so when you look at the bottom here you can see that obviously the in the spark executor the jvm is using some memory but you can also see an other um here other i mean i wish we could say r but you have to know that you're running an r application and so your r processes on the spark executor are consuming some memory and this is how you see it and here's another application where you're also going to see the python memory used by your executor and here you know really the benefit is to is to be able to know oh i i had an out of memory error my executor was killed and to able to troubleshoot is that because python was taking too much memory okay because that happens uh and so here we can actually see that this executor at some point reached its maximum capacity which is fishy probably uh probably you know something bad happened for that executor okay we're almost at the end of my time so i just want to say a few words to to recap the light um and and you know to thank you also and to tell you that if you want to try it you can sign up at delight.diameconics.com install the agent and then get started and feel free to connect with me i put my contact details uh you know email linkedin twitter anything works to give me your feedback and so that we can keep making the light better um with the two or three minutes you know i said that we had a little bonus content it's the apache spark 3.2 release it was released last week i just wanted to say really a few words at a high level so here are some of the main features in my mind one of them is the fact that the koalas library is now bundled with spark by default so you can use it without installing anything and koalas makes turns your pandas code into spark and makes it faster the other thing is that spark 3.2 comes bundled with hadoop 331 and even if you're not running hadoop you know even if you run sparkling communities you might be using the hadoop libraries in particular to talk to s3 and it makes s3 much faster so this is another great news and um it also comes equipped with roxdb which is used for streaming queries as a as a state store and it will make streaming queries faster and also more stable a last feature that i am you know perfectly excited about because our platform only runs spark on top of kubernetes and this is a community specific feature it's called pvc reuse i actually have a couple of slides to to go deeper into it so pvc reuse um since spark 3.1 it's possible to provision persistence sorry persistent volume claim basically kubernetes volume and mount them into the spark executors these are volumes that could be back for example by ebs on amazon and these volumes are automatically cleaned up when an app finishes so for example you would you would mount search that volume into an executor and so when an executor produces shuffle files the shuffle files will be written in that disk now with spark 3.2 the the the pvc will survive the death of an executor so if executor dies for example because of the spot kill or because of an out of memory error the pvc associated with it lives on and if a new executor comes in like exec 2 spark will reuse the existing pvc and mount it there so what is the benefit it will speed up the creation of executor 2 and more importantly the shuffle files which were written can be recovered by spark and so this is a huge game changer okay before when you had let's say a spot kill you lose all your work you have to recompute the shuffle files now you just start a new executor and you recover the shuffle files so really we're getting closer uh to the goal of separating compute and storage resources in a cloud native way on kubernetes and so that's a feature that we're excited about and that we're now starting to to try with some of our customers i think um we're over time so again i want to thank you all uh for you know your interest and i'll be moving to the um to the conference room to do some q a you