Scale By The Bay 2021 : Eduardo Gonzalez, Introducing Spark Cyclone Accelerating Spark with...
okay so thanks for that introduction uh so yeah so let's get started um so i wanted to talk about and introduce to the world uh what's um we're calling uh spark cyclo uh and uh the title of the talk is accelerating spark with the hidden super computer device plug-in in hadoop and i'll get to why it's hit in uh in a second so first um let's go over uh the agenda of the things i want to talk about today uh so first i'll go about myself a little bit about myself not too much uh then i'll go do uh introduction of the project explain what the sx aura tsubasa vector engine is then i'll go go into details about what's inside the vector engine what makes it special or unique and how it fits into the spark accelerator landscape there are many different tools available these days for accelerating spark workloads and then i'm going to go into the implementation details of spark cyclone and then finally compare how it runs versus a v100 using rapids from nvidia and then finally i'm going to go into some future directions okay so about myself i am i've been mainly working in japan most of my life i went to japan in 26 in 2006 to to work and then i've kind of been here uh since then in 2015 i worked with the ssdc which was a research organization and that's kind of where i got really into machine learning and deep learning and doing that at the enterprise level and that's when i kind of joined skymind and yeah in 2020 i started express ai which is a new company and i recently released the book as we mentioned before so let's get into the project so the project is called spark cyclone and its main job is to bridge apache spark and put it to the vector engine now there is a project available from nec uh that does this it's called phrobetis but for vetis works in a way where it's uh it's a compatible api that you need to change your code to use but spark cyclone's goal is to make it so that it's just a plug-in so you don't need to recompile your your scala code or change your code in any way to take advantage of the vector engine and this is going to be open sourced on november 1st uh and it's super easy to set up so uh i mentioned earlier that it's the hidden supercomputer uh architecture in hadoop and the interesting thing about this is that it's already built in so if you're using hadoop 3.3.0 or later it already has uh the plug-in the yarn device plug-in for the nec vector engine and it kind of works alongside gpus and fpgas so you just set it up the same way you would set up uh those uh different devices and it works and it kind of integrates with the yarn and uh yeah and then it'll schedule the resources appropriately so uh what is the sx aurora sub vector engine and what are all these uh names and numbers or names and letters mean um so very quickly about like where this came from uh so the uh most famous i think iteration of the sx series of uh computers like computer architectures from nec uh is the earth simulator and that was a super computer made 20 years ago in 20 in 2001 2002 and it was the fastest supercomputer in the world at the time for quite some time and uh yeah and it was used to do climate change modeling and simulation and also weather prediction tsunamis and things of that nature hence why we think the name spark cyclone uh really kind of fits with this uh project and so that one i think was like the sx4 and it went with xx9 and then there's the ace and the newest or the newest one is called the sx aurora tsubasa tsubasa means wing in japanese so that's what the cards kind of look like wings coming out of the motherboard i guess and so that's the new uh architecture now the interesting thing about this is before the sebasta iteration the latest one uh the vector engine was the cpu right so it was using it was the thing that was running uh the os so it's quite a capable cpu it's not a special dsp-like thing it's you know a very capable superscalar type of processor and it's being used in the latest earth simulator four you can look at this article here if you want to go into the details it is very very big maybe not the biggest supercomputer in the world uh these days but you know with 19 petaflops and uh you know 5 000 vector engines it's it's plenty fast so um yeah so let's go into the vector engine uh and see what uh what it's made of so the interesting thing about the vector engine uh is that it has the you know really high bandwidth essentially right it has this it doesn't skip leg day it has very powerful cpu and a lot of bandwidth uh to memory and it has a lot of memory too so it's a very balanced uh cpu in that sense and so the interesting thing about the the cpu is that it has the spu which is uh the part of the cpu that works with uh regular instructions same kind of instructions you'd have on an intel machine and so on and so forth and then it has the vpu which is the place that does the vector processing and is capable of doing uh this once the single core is capable of doing 614 gigaflops uh per core uh which is uh interesting when you think about it in other things i'll get into that in a bit uh and it has uh three terabytes uh per second across all the cores right the aggregate bang bandwidth between the cores is that high and the aggregate bandwidth to the hbm2 modules is 1.5 terabytes per second and that's pretty good the vector engine came out a couple of years ago and it's only you know recently that nvidia has came up with a card that is capable of doing that kind of uh bandwidth which would be the a100 and uh yeah and so if you put all the cores together there are 10 cores in the maximum configuration you can get six teraflops per card and 48 gigs about gigabytes of ram and if you put eight of these into a system you can get up to 49 teraflops so it's pretty powerful and very significant now one thing to kind of compare put this into perspective is tesla recently announced their dojo uh computer architecture for deep learning and i want to highlight it and compare and contrast it to the vector engine because there are some similarities but some very important differences so for example this is what the dojo supercomputer core looks like yanked straight from tesla's presentation about it and you can see it has you know these 512 gigabyte links between the cores uh it has an in-order cpu so it's more dsp like and it has these matrix multiply uh systems and 1.25 megabytes of sram so one of these cores when working in uh 32-bit floats i not sure if it does 64-bit floats at all but at least in 32-bit floats you can get 64 gigaflops for this single core so it's definitely less capable uh but it's also you know it's made to be scaled out so it's it will reach up to the the same kind of performance if you scale it up enough um so while the uh vector engine has 10 cores in a single card uh you know the chip in the dojo has 300 350 cores right so uh it's kind of that classic argument of you know do you want a lot of you know less powerful cores or a smaller number of more powerful cores and you know i think in general when doing analytics the smaller number of more powerful cores is going to be probably the way to go uh another thing about this uh you know dojo core uh and also you know many gpus as well is that they're not made for doing 64-bit right they're made they're they're actually very very fast if you go down to 16-bit or 8-bit calculations and another thing you'll notice about this is there's no dram right like every core has 1.25 megabytes of ram and yeah you're gonna have to really work out how to schedule things on this so these chips are very powerful for doing deep learning and i think they'll be absolutely excellent at doing it but when it comes to doing analytics i don't think that you know these chips or any of the other deep learning focused chips are going to cut it but the vector engine does and of course the i would say the the main kind of competitor in the space would be the gpus in spark right and so uh up until the a100 all your gpus that you're going to get are going to have less memory bandwidth so that's one difference and they have you know their performance the performance numbers that they have which are you know like kind of cheating because they only apply to you know matrix multiplying uh when doing low precision uh with a real kind of performance in teraflops that they have is sometimes significantly less and hasn't really risen that much uh compared to uh previously so only the top of the line gpus from nvidia i believe there there's there's many different configurations but like the p100 and the b100 and the a100 are the only ones that um really support 64-bit um but i think more importantly about this is how fundamentally different they are to code right as they have hundreds of threads and the way they they work you kind of have to have a different programming model it's very difficult to take you know code that runs on a cpu like an intel cpu and run it on these gpus with any sort of efficiency and that's actually incredibly important for for us in which i'll go into in a second about how we implement things so if we were to put you know the vector engine in the in the whole landscape i say that there's uh you know different options now you have cpus you have fpgas you have gpus and the vector engine and you know fpgas uh they're not my specialty so i can't really comment too much on them but you know they're mainly made for doing a lot of kind of bits bit calculations kind of dsp-like things uh very very parallelized so when dealing with things like 64-bit uh numbers and and these kind of large data sets i'm i'm not sure if they're going to be very cost effective i'm sure they're going to be quite performant if you're able to program them but they're also very different in in how you program them so there might be a limited compatibility with spark things and gpus are kind of you know you you sometimes to get really good performance you have to sacrifice accuracy and sometimes in big data that's not something you want to do so the ve is really i find it as a really good alternative to the cpu right cpu is definitely very uh very good you know in general um but the ve you don't have to sacrifice a lot of things to get good performance and later on i'll show you exactly uh what that means so now let's go into spark cycle and how it works so enabling the plugin is very simple uh so once you have the plugin the old name is aurora for spark but we've changed it to spark cyclone you just set it in the class path of the executors because we do need it on the executors you add it into the jar files of the driver and then you enable the plug-in and then finally what you do is you can tell spark or yarn depending on how you're doing things how many vector engines you want to use right so you can say one or eight or 64. however many you have and then finally you can specify a cache directory for kernels so one of the ways that we work in this uh plugin is that we generate c plus code and we compile it and the first time you do that will take a little bit of time but as long as you're kind of executing the same kind of queries over and over we can leverage that cache instead of uh compiling everything from scratch and that really raises the performance quite a lot so uh one of the benefits of you know this the spark cyclone plug-in compared to you know others the alternative which is probatis even though we build on top of robetus what we do aim to do is make the the plug-in fit directly into the ecosystem of spark right so you can still in the spark ui c your plan uh this native aggregation evaluation plan this is from us right so we fit right into the same framework you can uh debug the executors you can see all the same metrics and everything uh you know it works well so it's it's great for doing things and you don't have to change any of your code again you just you just integrate the plugin and you're good to go so uh how we do this you know compilation uh for those who aren't really familiar with the the how spark does plug-ins um you can kind of simplify it to to this this is like the main flow um but essentially the plugins uh the catalyst plugins in spark can fit right in between that logical plan and the optimized logical plan uh in this workflow that everybody uh uses uh and what we can do is we can look at the logical plan analyze it match it against things that we support currently and then uh change that plan with our own uh you know physical plan uh like physical evaluation nodes right uh one of them that we have is the native aggregation evaluation plan that i was talking to uh before and then once we pass that back to spark spark will eventually get to the execution part of that plan and when we execute that plan we then evaluate you know the the table we know if there's we can use the hints of whether the columns are known to be null or not null and then we can generate some c code that specifically targets you know that that type of uh that type of data and then we compile that into an so file using the nec compiler and then once we have that we the executors uh you know get to work and the first thing they do is they fetch that x that so file from the driver uh and then they load it into the vector engine using uh something called veo which is vector engine offloading um and that's how they execute the code they we copy the data into the vector engine execute it and read it back so i can go one more level deeper with the aggregations here so this is an example of some of the code that we have and this is the declarative aggregation converter and i think it's very interesting so in general the way that it works is we take the declarative aggregate that spark has right so you can specify for example correlation or sum or these kind of things which are all kind of defined in uh catalyst and what we do is we convert that to code by um you know specifying the different phases that we have in the c plus plus right so one of the things we need to do in the c plus plus is allocate the memory for um for the you know partial results and the uh and the final results right uh and then we transform that we do the iteration so that's when we're doing the main uh loops of the thing uh and inside of those loops we'll have our compute set uh compute section and then finally we need to take the partial results and copy them into the outputs that's the the fetch things and then finally we have three right because this is c plus we're talking about and since we're generating c plus plus code we need to make sure that it also frees all the memory that it uses and then finally there's that last part which merges uh two different things and these prefix uh strings that you see uh are just our way of having everything share the same scope but have different prefixes uh for for variable names and things like that which is kind of important for the vector engine because as a vector computer um you know it it takes four loops and essentially turns it into a single instruction so you kind of want to avoid like function calls and those and those kind of things so what's uh what's i think is really crazy about this which makes it really uh interesting from my from my own perspective as a project is that if you look into the source code of uh spark cyclone you will not find for example an implementation for the correlation function and that's because we compile the expressions uh from scala directly into c plus and i can i can even show some of that uh very quickly let me see if i can bring up the ide real quick so for example this uh this code that you see here on the left uh eventually becomes uh you know this kind of code here so this is um so you can see there's this x average uh and um and these things so all we kind of take this express these expressions here and then we generate the c plus plus code from that expression uh and continue on so it allows us to implement a lot of functions without having to kind of go through the slog of re-implementing every single type of method that scala implements now it does mean that sometimes this code might not be ideal for the vector engine but since the vector engine is quite cpu-like it turns out that this isn't so bad now there's still room for improvement and specifically in a certain number of queries that we're looking at so one of our challenges in this project is how do we take these scala expressions and kind of make it into a compiler so that we're generating efficient code for the target whether it be the vector engine or possibly others the other thing that we're doing that i think is actually incredibly interesting is that we're using lee howie's source code module which is actually really great so some of these features are actually available in c sharp where you can say like what is the name of this parameter and then use that in debugging streams um but we're also able to use this in here and one of the things that we do is we have this kind of code lines debug here and what that does if i bring back the source code that we generate as you can see it knows which method it is coming from and what line it's at right so this is the line in the scala code and then this would be the line in the c code and we can use that to um you know do some print line debugging so we can like see you know how long each one of these phases is taking uh when it's executing on the vector engine so that's very cool stuff and uh and yeah we also implemented things like tracing and things like that just so that we can be very thorough in measuring uh the performance so speaking of the performance i think many of you are wondering how does it compare to a gpu so uh i would like to compare it to the a100 but they are very very difficult to come by right now um we do have some v100 machines and so i can show you the performance of the vector engine versus the v100 when using our plug-in uh so before i go into the details just a quick note about the methodology that we're doing here so uh what i chose to measure here is uh the tpch queries and the way they're implemented is we downloaded the pdf from the tpch website i took those sql queries i copied and pasted them into spark sql blocks and that is what we are uh running uh now all tables were cached before the querying so we called uh you know cache table uh whatever whatever for all 20 uh yeah for all the tables that are available and then we execute the query and collect all of the rows and we print the rows for verification but the printing part does not is not measured right so for us it's you know we measure the time it takes to execute the sql and collect all the results but not print the results so that so you know it's very realistic in terms of what you would expect uh uh from your own um from actually using uh and we use the vector engine type 20b which is not the fastest but it is uh one of uh it's almost there and then the nvidia v100 16 gigabyte and we ran uh these at three different scales so tpch has different scales one is a one gigabyte data set in total uh 10 is a 10 gigabyte version and 20 is a 20 gigabyte version uh and uh all the uh outputs are exactly the same between cpu and gpu and the vector engine uh so there's no cheating uh and uh yeah so they're so it's pretty thorough uh with a little bit more work this could possibly be a proper tpch benchmark so at uh at one which is the one gigabyte um benchmark you can see uh this is kind of like the raw data it's the ratio of the performance right so at one that means that the vector engine and the gpus take the same amount of time uh and as you can see in general uh we are winning out there are some queries uh that we're looking into like 10 14 those are going to actually be an issue throughout the the other ones um and 16 through 19 uh in general like they're they seem to be very difficult right so difficult for the gpu difficult for us um but you know like for example query9 that's a that's a humongous boost uh like it's it's like one second versus 20 something seconds it's a very big difference uh for that uh query so i can this is kind of what i was mentioning about how the vector engine has a lot more balance when it comes to you know performance to the uh the bandwidth and the bandwidth difference as the v100 has 900 gigabytes uh per second of bandwidth um should be about like 1.6 right so seeing the number two kind of come up a lot uh is important and the more complex the query the more the bandwidth uh kind of comes into effect and comes into play at this so one is very small so you know let's skip this let's take it up a notch to 10. so when we go to 10 we can see that there are definitely some problems in our plug-in it's still very early days and we kind of know where the problems are particularly in query 10 these are all string based ones and we're kind of using an inefficient string representation at the moment but you can see it's a very similar thing right uh you know 16 17 18 are very similar but a lot of these queries have significant performance advantages uh on the vector engine versus the gpu and then finally 20 and 20 has it's is interesting because 20 is now bigger and the data set is bigger than the the gpu memory because the gpu has 16 gigabytes of memory and the vector engine has 48 so now you can see even though in 10 it was you know not so great in 20 now everything is is just a little bit bigger um in terms of the query performance so yeah so i think we're we're really on to something here and we've just barely started right uh rapids has a humongous team working on it um and i'm sure maybe there's some uh you know tuning magic that they've done i did spend quite a lot of time trying to tune the gpu to work correctly maybe i'm doing something wrong there's a ton of parameters but like the out-of-box experience with the vector engine is showing really good results and when you take it down a notch and use phobetis the results are even better um because just the way the pro vettis is kind of mostly c plus plus so yeah so that is the performance now let's talk a little bit about the future directions before i wrap up here um so like i mentioned we're barely there's so many low hanging fruit uh available to us right now that we're not even close to uh you know the limitations of uh how much performance we're going to get i still think there's still room for improvement um so one of the things that we don't do is uh good column caching and i think that's that caching is something that we could do to really accelerate some of these queries if we can limit the amount of copying that goes over the pci express bus because that doesn't have you know a high bandwidth but the main memory of the vector engine does and the other one which i think is super exciting is uh x86 mode so frovedis doesn't only support the vector engine it also supports cpus and so i'm very interested to see for example how spark cyclone compares in terms of performance versus uh the things that intel is doing with their oak toolkit right um so this is something that we're going to try to look into sooner rather than later but it could be very very interesting um the other thing which is part of how the gpus kind of get their performance is doing things like uh shuffling over rdma and things like that and we can do the same thing in the vector engine if we can shuffle over infiniband using rdma i think that and do it vector engine only we could have you know significant improvements and performance even more than we have right now uh utilizing scala native is an interesting one so i do actually have scala native compiling binaries for the vector engine so nec has created a fork of llvm that does create binaries for the vector engine and i have done hello world things on skull native on here and i'm very interested to see where we can take this like how far can this go um the other kind of major challenge that we have is optimizing the c plus code from the scala code so we kind of need to make our own compiler an optimizing compiler to optimize that c plus plus and of course i think one thing that can always be better is to kind of raise our abstractions a little bit we are still still kind of dealing with strings more than i would like i really like to have you know things like a c expression well we do have a c expression class but like a c variable you know those kind of higher level abstractions from scala and then kind of use that in a magnetic way or like a you know like a free monad way to generate code and then also pass those through an optimizer to do the the one before so these are all our challenges i and uh if those things look interesting to you please visit our jobs website and sign up or visit our booth um we're just going to be there you can talk to us you can geek out we don't really care um if you are looking for a job or if you're thinking about remote we are a completely remote company this is you know we do our work on gather town we're doing a lot of stuff in the ai side of things not just in the analytics side of things so i think we have something for everyone so uh yeah if you saw something interesting here feel free to visit the booth or visit the website and send us your resume and we're happy to talk to you okay so that is everything i wanted to talk about you