Scale By The Bay 2019: François Farquet, The Renaissance for Big Data and Parallelism with GraalVM
[Music] I'm for sifaka I walk a tree collapse in in Zurich Switzerland and I'm a senior researcher there part of the growl VM team and I took today about 'grill VM itself but also about Renaissance benchmark suite which is a suite we came up with with University collaborators to squeeze more performance out of the compiler and I'll show how we did that so first of all I stopped with an introduction of what is called VM and there are different aspects you may have heard of native images of faster JIT compiler and stuff like this so let's clarify things earlier earlier early on and so first of all dwell is a compiler project and so it's not meant to rewrite the all the old JDK but to reuse the bits of open JDK but we place the compiler itself by a faster compiler and with a high with the focus on high performance and that's what has been done so you can run any jvm language java scala groovy cuddling whatever in the context of a pen jdk but if you use grant vm you'll benefit from from the growl compiler now draw vm can also run other languages like Ruby or Python GS each each of these languages sub team which has implemented the language on top of of the growl vm api so so this is this enables polyglot programming running the same process with high efficiency you can optimize through the language Boris it can also run native languages like C C++ rusts in inside the JVM so you can G compile those languages too and also benefit from the from the película programming aspects this is a necessary bit for all for running all native executions of dynamic languages like Python or okay so that's what girl VM can run but it can run in different contexts so I mentioned the open JDK context as a just-in-time compiler but it can also run as part of the node.js platform so we can run all those languages within no GS and we have an ode utility shipped with equal VM you can also run those languages inside the Oracle database so instead of pulling your data out of the database and pushing it back after doing some manipulation you can do directly the operations inside the database and also you can produce native images so you can prove standalone binaries from your your applications so this is a game changer in in for the four languages like Java and Scala because you can produce highly optimised binary which are very small so it's so useful for for the container world for instance and also it also you can also start real quickly and it's very smaller a very small memory footprint because it's compiled ahead of time okay so growl VM comes in to flavor the Community Edition because it's open source you can download the source on github it it's free for development news production news and there is also an Enterprise Edition which gives additional performance and security features and there is also the Oracle support for it note that the Enterprise Edition is free an Oracle cloud and now the Oracle cloud also has a free tier so you can run basically Enterprise for free on the free tier if you want to give it a try and compare that okay but main aspect of the talk is this Renaissance benchmark suite and I which relates to two Grell VM and so it's a suite of benchmark that has been open sourced earlier this year there is a website for Nestle's that dev it's it's it's open source and it's an extra benchmarking suite so first of all what is a benchmark and why we do we need them so benchmarks are basically a sample software that that is meant to be representative of programs you guys ran and that we run on our machines and on our clusters and the goal is to have a very representative set of benchmarks there's no competition between them we just want benchmark that span all kind of use cases and all kind of software will run nowadays so they're already existing benchmarking suite like the speck organization to consult some of many big companies who came up with benchmarks for the JVM they're also universities who came up a spot of research with benchmarks like the decapod benchmark for instance and so the question that arises is what why do we come up with a new one and the the main reason is that we noticed that the way we write software nowadays I mean the word software in general is evolving what we are writing nowadays and the software we use nowadays is different than one we were using 10 years ago underwear interesting told yesterday about this so the every everything is changing and we need to keep up with this so we have new programming paradigms we have prolyl collections for drawing a synchronous futures actors and all of that and and long ago this is something that that simply didn't exist now we also have frameworks to deal with all of that complexity popular ones like spark for big data for stream processing with GDK streams introduced in Java 8 we have the scatter streams of extensions a collector based programming nettie which is the popular for network staff and we want to be sure that this is covered by the benchmark suite because the benchmark is our target for compiler developers I'm going to be sure that this is something we are we are good on and we can give good performance to the users so one part is the the software evolves the waetford software evolves but on the other hand the platform itself evolves the GDK Valls we have now atomic operations everywhere like three data structures non-blocking i/o etc invoke dynamic new byte code introducing a seven but really starting being used in in java 8 with with jdk streams and the the benchmarking suites that have been introduced before sharra it existed they simply do not cover these kinds of code patterns so what we can notice if we step back a little bit and look at these two aspects the way software is written and the platform evolving is that there is a common theme here is that we do more and more we use more and more concurrency in probably lism and it's important then to be very efficient at dealing with parallel workloads and yeah if I if I one CPU doing most of the work and the other ones I know there is something that is wrong that could be optimized so the Renaissance benchmark suite there is a research paper you can find the reference down there which is also linked on the on the Renaissance that dev website the vision is to come up with a new set of benchmark with that how modern and diverse or modern in the sense that they use the new technologies I mentioned earlier and that stress a lot the DVM and the compiler with concurrency workloads and and poly workloads and that that also include functional programming versus object-oriented programming in order to cover all of those aspects so if you want we want this to be diverse do we want to keep that open source and open process because we wanna be stuck in the past when to keep that thing evolving and have a collaborative effort and and have people contribute to it and it's not a competition with other benchmark suite because everything is worth looking at even workloads that have been written 20 years ago it's good to be good at those even even though we focused more nowadays on other kind of workloads we don't know rests on that what has been done in the past and and this this is this means that it's not a competition it's just a great addition so to come up with a new set of benchmarks that covers those different aspects it's a bit challenging because you have to find those workloads to find what is what is important to people what is the software people use and what is popular nowadays so tools have been written to collect all kind of of workloads sample application demo demo workloads from different different projects and we found roughly 100 candidate workloads and then they have been analyzed to to match a set of criterias so the criterias first of all they have to be to use modern concurrency so highly parallel workloads using streams using actor based programming you use like free data structures and memory databases it's a trap we want those benchmarks to be realistic we want to be sure that it's not completely artificial because it had to look like software people use and they have to be diverse meaning that if we introduced like 20 spark workloads and they all do the same thing it's not very interesting you want to be only one of them for instance and we want the those benchmarks to be deterministic that's very important for benchmarking if you run the thing twice you want to be sure that it runs exactly the same thing on the other hand you want to be sure to avoid some pitfalls for instance you want to be sure to have a source code because for analysis of what's going on it's better to have source code you want to avoid being stuck in the past with workloads that are not being used anymore you want to avoid timeouts resource leaks and and stuff like this but you also want an open process and open source because it enables source code level analysis but it also makes it easier for the community to get involved and to suggest new benchmarks and after some point we may also see that some big spots are not relevant anymore and may depreciate them and even retire them so doing this analysis we came up with a set of 21 benchmarks that I mean white here that have been released at the earlier this year and in green its new ones that have been introduced since then this summer and there are like six or seven but she's part benchmarks in there the final benchmarks from Twitter there are in memory databases streams benchmarks and a compiler benchmark and and then yay and a few others so to justify that list we collected a bunch of metrics to assess the quality of those benchmarks so here is a list of 11 metrics that we found interesting to to model what the benchmarks were doing and there are directly related concurrency metrics like the number of synchronized blocks you have in your that are executed the number of blocks or methods the number of weight and notify that the totally related to concurrency the number of atomic operations the number of thread porking etc then indirectly related to concurrency we have CPU utilization because if you have 16 cores and one is doing all the work and the other ones none that's something you want to capture cache miss - because if you have straight contention that will increase the cache miss rate and also other metrics to assess the quality of the been those benchmarks is the object-oriented abstractions like number of objects and arrays allocated number of methods invoked and also a final metric about the number of executive invoke dynamic byte codes because it's a new byte code introduced in Java 7 and wanna be sure to cover that too okay then let's have a look at the numbers those those metrics have been normalized to divide it by the number of reference cycles to make sure that we can compare them and if we look at the synchronized metric we can see that the renaissance benchmark is the first bond on the chart as a good spread and covers well the sync methods some others are quite some other sweets are quite similar in that regard just a gravy m1 is quite good though now if you look at atomic operations also Renaissance as good spread spective en is the one with the least down the number of invoke dynamic instructions you can see that in the Scarab industry there is none in the capo inspector vm's of a few and in Renaissance there is a good spread but to get a better feeling of those 11 matrix we would like to see how they correlate with each other so we can use a recommend technique dimension them dimensionality reduction technique which is PCA to project that into into a smaller space and analyze the weight of each component of each metric in each principal component so if we do this we end up with the following chart with the principal component one on the x-axis and principal component two and on the y axis and let's look what what are the weights in there and four principal component one we see that object oriented programming metrics dominate with mostly number of objects and are allocated and method calls and we can see that it's similar to at the other sweets because it spread among along the x axis for Renaissance just like the other sweets but if you look at principal component two is the concurrency primitives or atomic and and thread parking and we have a very good spread a.m. so we covered those aspects very well now for the principal component three and four the number three is conferencing primitives to the cache miss rate and wait and notify and those are a bit more spread in Renaissance and also invokedynamic is much more spread this is principal component for in that in the y-axis so as a summary for the diversity of the benchmarks we have a good representation of concurrency and we have a good representation of modern language features so lambda streams etc and when it comes to object orientation we can come to the conclusion that you'd similar to the other sweets okay so all that explanation to convince you that those benchmarks are good quality and that they cover aspects that are not covered by the other sweets and especially that they focus on concurrency and parallelism but now why have we done all this and spent all this effort coming with your benchmarks so the initial motivation was to be sure to cover all the software space right to have a good a good sample of what people use but now that we have that and we had we have confirmed that we have a good a good representation of modern workloads can the compiler be better at running those workloads so it has been investigated intensively and as part of the girl vm compiler the those benchmarks have have been used to come up with new compiler optimizations mostly related to to concurrency then so you can see escape analysis with atomic operation so it's kept analysis is a compiler optimization that can reduce the the number of allocations so that will also reduce the the GC pressure and also produce more efficient machine code so escape analysis in the context of Atomics was a new optimization also loop loop wide loop wire la crosse thing meaning optimizing the synchronized blocks atomic operation conditioning also for Atomics method dental simplification for these new byte codes with invoke dynamic so great we can make the optimization and make the compiler faster thanks to those benchmarks now we can also look at existing optimization that are part of the compiler and we have noticed that the three main our compiler optimization we're helping the Renaissance benchmarks a lot so that's speculative one version Lu factorization and dominance based application simulation so now in the research paper you'll you can also find the the impact of each individual compiler optimization and compared to yes when you when you enable or disable just not optimization you can measure the performance at your benchmark and you can see sort of it's a bit small but the first four rows are the phone new optimization and the last three words out the the existing one and and the first box is the Renaissance benchmarks and then you have the other popular sweets Caliban Jacopo inspector VM and interestingly what we can notice is that the four new optimization have a very good and positive impact on the Renaissance benchmarks right because it's been designed to make those faster but we can measure on the other benchmarks and we see that if not a lot of impact also because I mean if you optimize input dynamic and you have no invoke dynamic that's that's normal that you will not get more performance out of it but but even the other concurrency optimizations we are not doing much still a little bit you can see as you can see some little improvement but not very significant so if you retake those seven optimizations we can see that they have a positive impact of more than 6% on the Renaissance benchmarks and roughly three or more or less on the other speeds I think most of them are community look vectorization no but the other ones I think the community but I would need to double-check okay then the conclusions about those compiler optimization is that they help to identify new new optimization you cut patterns that could be optimized further and we could also identify high impact optimization that were already present in the in the compiler so we can see that it's critical to have good benchmarks for for a compiler team because we can increase the compiler performance for everyone okay so related to the question this is the performance of grunt vm compared to at spot which is the default compiler in in open JDK and this is the the 21 yes the 21 benchmarks that were initially released in Renaissance so you can see that on the far right you have the Geo mean and you can see that the Enterprise Edition is two percent faster than than open JDK and that European Community Edition is six percent faster than up and JDK and also individual speed ups you can see for instance the the the big boss it's it's an apache spark benchmark naivebayes which is reacts faster running on well vm and another one that may be interesting is the dotty benchmark which is v 6 and because i but a lot of Scala people in here and and 30 is this kind of 3 compiler the performance with the Scala - compiler is is also is also very very similar on unguent vm so you get 2037 performance boost if you use enterprise and you get 20% boo if you just switch to LVN Community Edition so it's basically drop-in replacement so just give it a try even for compiling files it's it's already much better and running down the workloads it depends on your workload but definitely measure it was it and yeah so yeah Ivan yes or even clarify the the Java EE means Enterprise Edition right in the so that's the red bar and see CES Community Edition ok so as a conclusion we have introduced this new set of benchmarks I must say that it's not only our collapse that has done that it is there was a collaboration with five I think universities all around the world and now we also collaborations with other big companies so it's good to see an interest and see that those those workloads will continue to evolve so we came up with a new and open benchmarking suite for the JVM modern and diverse as it can be shown in the in the paper and in the diversity analysis and they better represent modern JVM frameworks and features that are very popular nowadays and it's beneficial for compiler development so they beneficial for GC developers for tools implementer so benchmarking suite and are not only a VM developer thing it's for anyone wanna measure performance of a given platform so it's an open source project it's an open process so country contributions are very welcome there was at the moment someone from Microsoft who is writing a contributing a benchmark we had been we had feedback on the final benchmarks from the Twitter people we had bug reports from ASAP people we could start back IBM virtual machine that has been reported in fix so it's really helping everyone and and if you are part of of one of such big framework that that is trained in nowadays maybe runs on the JVM maybe it's worth spending a little effort to contribute in to contribute a benchmark because then VM developers like us will spend a lot of time trying to optimize it further and and try to squeeze more performance out of it and then running on those those ran those workloads or similar workloads on the uncrowned VM for instance will will probably be faster over time so so please be involved or discuss means me if you if you have a potential benchmark candidate then there is a committee that will vote for four new releases which benchmark we should retire which what we should introduce etc and the goal is to keep the thing evolving in an open manner and and we hope that this project will continue and continue to grow and keep up to date and not be stuck in the past like it has been the case for other other working sweet but we are confident that we will be able to do that so I come to the end of my presentation and I thank you very much and I'm open to discuss further about well VM urban assault with any of you in the hallway or maybe if we have time we have time any questions yeah this is great I'm really excited about Karl vm's thanks for coming to talk about it is there how do up Tamizh a shion's I know Twitter has been contributing some optimizations to the community edition is there a way is there any flow from work that goes into the Enterprise Edition to community edition are we waiting for people to only submit contributions like ie I might imagine a model where after 2 or 3 years or 20 years you would take optimizations that only exist in the Enterprise Edition and move them in the community edition I just would love to hear about how how you think about what goes into each of those two editions I cannot really come comment on the business model because I'm an engineer and I'm not a manager or director or whatever but yes we have a lot of contributions to the ground compiler itself from many different people so it's good because if we if we keep making the the Community Edition better over time it will benefit everyone on the other hand having an Enterprise Edition can help funding tens and tens of people to make the community and the Enterprise Edition fastest I think it's it's an interesting business model because for for such an investment it's project that started eight years ago and Oracle starts getting money for this project only in 2019 it has been it has been announced as a product in 2019 it's also part of the cloud strategy etc so I cannot comment on the on the business itself but but for for compiler optimizations a lot of time has been spent into into an enterprise in liner which is which is very very efficient there is a paper about it and and feel free to check it out but but any contribution is welcome and and will continue to be welcoming great thank you very much for so that's all we have time for the next talk starts here in ten minutes thank you [Applause] [Music]