data.bythebay.io: Manish Singh, Characterizing the performance of Big Data processing platforms
hi everybody trust everybody is doing well and so yeah I just wanted to what a main team is going to be today is sort of clearing some of the clear the air kind of on performance characterization and how you measure performance of big data processing platforms there's so many frameworks out there more come everyday and everybody's got a you know there's something in the code that they want to change that you want to do in the code to get better performance but what I am as an in you know an infrastructure guy and I would like to see what kind of resources are being used by these processing frameworks to be able to do a lot of different things actually so the motivation is you know why do you want to do this when in the process do you want to do this in do you want to do this in the beginning of your life cycle of your project do you want to do it on an ongoing basis so there's multiple applications and there is no like there's no boundary like oh I have to do this at the beginning of my project and then I I can let go so that's what I want to talk about set the outline for the talk why when how and in what what places do you want to put these characterization or frame characterization let's say insights into right so so again like I said there's a plethora of frameworks previous talks have covered so many different frameworks hive you know and underneath there's MapReduce paradigms there is tears right there's there's so many of them so it's it becomes at least for me it becomes kind of difficult to navigate this whole landscape what am I going to get when I this thing into my dev cluster what does what does it mean what do I get when I get into a stage cluster and what does it mean when I get into a production cluster right so to be able to characterize to be able to protect to be able to say yes this is what the bounds on my performance are going to be this is how the system is going to scale in terms of what what dimensions of resources compute network storage is it going to consume so it's frequently you know what happens is people will tell you that this particular benchmark is there and it will run you know the fastest Stara sort has been invented by this particular vendor or so and so right but what you need is really what your problem is and you need to solve that problem right to characterize your own problem you need tools for that in your in-house deployments right it has no bearing you know whether somebody's terasort runs faster or somebody's terasort runs slower right so it's really about you know an in-house you want to plan a deployment you want to choose the right hardware infrastructure be it in the cloud would be it in a colocation Center or it could be your own data center right so we need to do capacity planning and our day-to-day operations troubleshooting performance tuning and scaling so typically people will say ok let's run this benchmark right but what do you get out of running on a bit running a benchmark right that's important so what we've done is we've chosen a couple of you know well-known just to make sure that we set the ground you know there's some common ground use some well-known algorithms terasort for MapReduce and at TPC das benchmark for hive and tears on for hive on tears and MapReduce and then the spark barf from data breaks - just so it's not how you actually measure it's not you know how fast these things run but what do you see when you run these benchmarks right and you don't need to run huge data sets to be able to be able to see how these systems are consuming resources and how they're performing right and what the trend lines are okay so any questions so far if they're yes it's a well known database benchmark that vendors typically run they're you know it's like queries being run on huge data sets and people want to see you know typically somebody from Dell will say you know T PCBs this benchmark our machines this particular hardware platform runs really fast right using you know so we've set the records or something it's it's an industry consortium that you can you can actually download this all of these and run them so where do we run it we've ran it and in AWS so the methodologies identify key characteristics and continuously once you've identified those key metrics key characteristics you continuously measure and analyze right that's the key portion you know you're continuously looking at data from different sources not just from the infrastructure but also from your looking at what application history or application timeline servers are telling you from Hadoop so you're getting information from multiple sources and trying to you know fit you know solve the puzzle basically where the problems are and if you're running into some issues and how to troubleshoot those issues faster right so what we've done is used terasort from Amazon as an illustration for this approach we used a D to extra large nodes with two core nodes and one master node and we ran like I said we you don't need huge datasets or Hugh you know our big you know clusters to actually do this characterization that is the key the thrust or the cut and thrust of this talk so here's here's something that I pull up like one of the the two dimensions for the CPU how does CPU you know what's the cost of CPU as the problem set size increases right pretty linear right for memory and for CPU its linear and the number of maps in reduces this is also linear the reduces are constant because obviously you're partitioning data set but the maps are linear this here's something interesting the average reduce time average shuffle time and time total time taken together everybody can see that No oops all right the trend line is sorry about that sorry the trend the key time so this captures whatever we saw earlier was that there's linearity in everything else right the graphs are pretty linear now but what happens is as you increase the data set size and due to certain variations in AWS in the network performance due to the multi-tenant nature of AWS you will see that the shuffle time which is the time that nodes exchange data tends to dominate the performance and there if the shuffle time results in non-linearity then you will see that the non-linearity become is the shuffle time dominates and so the translates to not non-linearity in the running time of the algorithm so less that's you know so what do we do we we actually as part of our study what we do is we look at why this is happening right why is this the case so we have a continuous intelligence and that is looking at all the data that's coming at at us from these deployments right so here's something that we plotted the shuffle characterisation shuffle the number of bytes for the shuffles are also linear right so remember so everything is linear except the time taken right so why is that why is that happening here's what we plotted when we looked at the network performance right so the network performance for one of the nodes which is pretty representative of all the nodes in this case as you can see we're up to you know 130,000 packets per second now these are really small so I'll skip over these here and on another interface on the other device so we are almost getting to a point where we are reaching the theoretical maximum not the theoretical maximum the practical maximum in AWS for this kind of node right that's why you see the non-linearity even though the number of bytes are mu bytes that are being moved between two nodes is this is linear in nature but because the network is reaching the max performance close to the max performance so you're seeing the non-linearity in the time taken to transfer these bytes so any questions right let's I've thrown a lot of so if if there are any questions so this is a d2x extra large instance in Amazon that has Amazon will tell you it has moderate network throughput they don't give you a number on it right so they will say this is moderate what does moderate mean so theoretical maximum on these nodes is 1 Gbps one gigabit per second right but the moderate ones go from 200 to 500 megabits per second so that could fall in that wide window and your application would start seeing this kind of non-linearity right so it's very important that you know you could run your tests at one point in time time T right when you are planning your deployment see it's set of results then you move to time T plus X right and then you see a different performance that's why you need continuous intelligence or continuous information being analyzed from these data sources right and I can show you a actually a demo quick one here this is our dashboard it's implemented using Summa logic so this is this is for actually for SPARC it's a little slow here so when we ran as you can see when when we ran some gosh again it's a little too tiny right but the key is you see these little jagged patterns that's when the there was you know the problem the algorithm was running and I'll get back to this part as this is just an illustration I'll get back to it but what is key is you see the amount of memory that they that these frameworks consume it's like goes up to maximum regardless of problem size these frameworks will just go and grab every bit of memory they can while they're running so it's like a greedy approach right they're trying to grab as much as they can to run the fastest that they can you can obviously set these things you can set them statically but there's no way to set them dynamically right where you are able to say can't get data in analyze and set the memory patterns appropriately right the memory usage patterns let me go back and resume here so that has enough said about you know MapReduce and this is some other trends that we saw while running terrorsaur tests in terms of memory consumption you can see how the memory went up like way up there when these tests were running and the longer the memory stays up there the more chances that your system will start you know choking literally like it's thrashing it's like somebody drowning right there trying to flap you know their hands trying to stay afloat it's seriously what that's what happens and that will result in more CPU usage as we go along we'll see that and these are some of the disk changes that your disk performance so the key elements are storage network and compute right and how those resources are being used by the by your system and this is a sample of how HDFS the underlying file system is being used so what we're saying is essentially you know it's it's easy to reproduce this kind of characterization without a lot of expensive hardware right and get get to know your application get to know how it impacts your infrastructure a priori rather than finding it out in production right so I like I mentioned we ran all this in a 1 Gbps instance where the network we established that the network was the you know was the choking point can I get 10x the performance if I upgrade from 1 Gbps nodes to 10 Gbps nodes can I get 10x the performance what would be your best guess know I cost you 10x but it's not going to get you 10x performance so what else do you get for that 10x investment in these resources you get tons of RAM you got so instead of you tongue give get tons of CPUs 244 V CPUs and you get 48 terabytes of disk sorry and this is a type of 244 gigabits not 24 d2 Excel comes with 30 point 1 gigabytes so what do you do with all of these resources right you've got you went ahead and invested 10x what else do you want to do how do you want to you know justify this how would you justify this to your CIO right or to your CFO who's cutting the check that I need this well one way is if you have in your business different applications that have different needs then you mix the jobs so that you utilize all of your resources to say at least 50 percent I think 50 percent would be a good number to you know start off with because there are obviously spikes you have but you know coming from the background I come from like networking and infrastructure typically people will aim for that kind of utilization although google claims that they can do like 90% utilization on their network when they run their application so questions any yeah linear scale sedimentation especially my application girls yeah so what correct yeah that's a good question so one thing to look for in the shuffles is obviously you don't want to be moving that much data around if you can in your application then partition better right and then obviously like you said have more nodes that will the key part is that if you've partitioned well you wouldn't need to have that many move that many shuffles in your algorithm so that that was the key takeaway from this the the data has not been very well partitioned so the faster Tara sorts that are out there have actually partitioned the data much better right so that's does that get your correct right yes yes yeah right so one of the reasons that I chose that we chose d2 extra larges this is what Amazon you know sort of recommends for your Hadoop install so we were trying to go for what they tell you and what does that get you right for that at that price point there's always a cost performance trade-off right you can always go typically what we have seen in deployments as people will have heavily over-provisioned clusters heavily over provision CPU never goes above 2% right I don't know I find that a little jarring you know to to you know see that and and I've had people complain that you know an AWS costs a lot it's not it's not inexpensive so I don't know what your guys experience has been whether that echoes with some of what you you have experienced or right exactly so you're paying for a lot of things that you don't actually need unless you can find you know that the rights so yeah so one thing is that your business might have a heterogeneous job mix that you want to schedule intelligently right so be to be able to utilize the other thing is looking at converged infrastructure where all the units the network the storage and the compute are all in one box sitting in say a four rack unit a footprint right they show better performance in the in these cases the reason we chose this is because we see a lot of our customers using AWS and you know it's it's but they complain about the cost and the cost past performance trade-off so we looked at the you know MapReduce paradigm and then query based systems or query based frameworks are also you know so popular there's and everybody even though no sequel guys have a query based front-end right because you have so many legacy applications out there that you want to move to a big data processing but but you still want the look and feel of the query based system so we use TP CDs and we're oh you wrote our own queries we use both MapReduce and ter as our underlying processing engines or frameworks and the table sizes were not huge just 210 50 GB will 1 2 3 joints these just the sample queries so this this is all very you know I'll skip over these because the interesting portions are a little bit later here these are all the number of maps and reduces with different joints oops yeah so oops buffered keystrokes okay so these are the operations yeah so these are the running times and we'll see we'll see that how the three join running time actually it never even you know there was so much thrashing going on that the system just couldn't do it this is a 50 GB table with three joints with a limit set on the queries to just 100 rows right it didn't run with this hardware which is this has 30 gigs of ram on each node right and there is one point two terabytes of hard disk on each one of them now we will see how and why and so these are the maps and reduces this is the memory consumed the trend is important you can see that the memory is with I have a trend line there which shows that it's linear but the time taken is super linear again so here's a 10 GB problem and you have a super linear almost linear in memory consumed but super linear in time taken right why does this happen so we can go into the back to our demo and see why that don't die on me it's one second it's a little slow Network come on all right so as you can see while the job was running or as you cannot see can you see the CPU over there so the CPU basically when the jobs are running the CPU is going up and the idle time goes down that's when the the most of the number crunching is happening and the key thing is why is the CPU getting busier right we're not like really exercising the CPU right but if you see here the memory is going way up high this is 50 GB right and that results in CPU doing swapping trying to swap pages in and out right and so your CPU starts going up so what's really happening is this framework they're going and trying to grab as much memory as they can right that's something that needs to be controlled and you cannot Ramin they're you know they're it's not a cooperative system anymore when somebody tries to grab everything they can right it's not it's not nice so that is a recurring theme you see that where do you see the non-linearity you know you've seen every every operation every the number of data bytes or the problem set is linear but then there are certain points when non-linearity happens right and so we are trying to find the cause of that non-linearity and each application might be different and you might have a mix of applications so it can get you know it's a pretty complex but at the same time it's it's a fun thing to solve right so if we go back to my presentation any questions well while this loads up or we have ten more minutes I don't want to keep people from this match I might get stuff thrown at me so while this is loading we can you know have some time for some questions yeah please right right yes yeah yes okay okay yeah good good point I haven't so our idea is basically we have agents sitting in all of these nodes right and we are getting data from the REST API is that all of these data processing engines expose so Hadoop timelines application timeline server yes or yarn timeline server Hadoop job history server right all of these frameworks are just examples you can get it from ganglia you can get it from any of these sources and you can throw it in to this analysis engine and that analysis engine has the intelligence to be able to figure out what's happening in your system so it's disparate data sources right you can anytime you go into Amazon right AWS you'll have cloud watch turned on ganglia turned on but what really happens is that these are being looked at by different people right a lot of times I that's been my experience the net ops guys the SIS off skies the DevOps guys they're all looking at different things at different times they're not looking at the same thing there's not a single pane of visibility into your infrastructure as well as the application and to connect the dots between the two I now job level two you can get job level information and we were using job level information from Hadoop yeah yeah how the mappers are progressing how the reducers everything and we have integrated that so shameless plug we have done this I'm not a I guess I'm not the marketing guy from high company right what we do is continuously look at all the data and we have built up with our customers a job bank pretty much and we are trying to build a system that will actually be able to predict performance right based on your applications and the right training data being supplied to that application I can show you later later I can show you the architecture diagrams that we have that is our presentation you know sort of from our company perspective I didn't want to encumber this room with that you know basically not to trying to pitch that but I'll show it to you so I'll just quickly then go through the spark tests and it seems to the spark tests that we ran come on might have to revert to the one that I have stored here we go so against Park in summary the spark tests are also dominated by similar theme except the network again plays a huge part and so again this the spark perf tests come from data breaks from their github repository their sequel tests are not running right now not compiling so we couldn't run those but we ran the spark and spark streaming tests so again this if CPU is not utilized memory keeps going up but it doesn't sustain at such a high level such that it's a problem disk also it's fine the network is what really you know is if you see the network that's where the crux of the problem is again because you're reaching a maximum of what your peak throughput will be on the network for the spark cluster so if only I could get this back up but no so I will then I can show you the so the sorry it's not showing up but I can show you the architecture of our you know what what the system that we use and if there are any more questions let me know yeah No so it's an automated you know process where once okay so there's there's one aspect which is the modeling aspect right the modeling of your application and there if you have tools for those right yeah so that you can say even at the data modeling phase a lot of people are spending a lot of time a lot of cycles of the data modeling phase and they don't know what the impact of that data modeling phase is going to be when you run that application right so we're trying to bridge that gap to so that you can actually predict without actually running your application how to get to that point right and dropping resources yes when you so you can change the type of storage on your devices for that so if I can use HDD suppose my budget is such that I can only afford HDD or I have a legacy system that used to have HDD I want to repurpose it for my you know lower end jobs right you use HDD but if you want a higher performing application you could use SSD so you can plug in an SSD and see the test the impact of that and this is without you actually running your application so is that yeah right right yes so that's a good question so one of the things that that would that is interesting there is how today's systems do all of this statically that's why you have to do an iterative process right memory consumption spark spark executors for example how much memory will it take so our approach is slightly different from that approach right we want to do it dynamically and we want to be able to so that's where we are going towards where you dynamically change things so you don't have to run one test and then scale down in one resource and see the next so you're going to see the impact of the dynamically resizing of your cluster that's yeah that's yeah we should talk no no no thank you guys thanks thanks for a very great audience thank you guys