Scale By the Bay 2018: Chris Thalinger, Graal: How to use the new JVM JIT compiler in real life
you it's not working yet oh here we go that's not high so um I was hoping this would actually work with them you know PA system in here but it does not as you that's not very good because the talk I'm doing is like 90 percent demos so I'll be typing all the time I can't hold this thing so we'll see how that works out yeah oh my god okay I hope this is not our equipment I work for Twitter so this is embarrassing so this talk is basically showing you know let me explain this all later okay so my name is Chris I work with Twitter for those of you who don't know Twitter actually has a VM T so I used to say we have a small VM team but that's not true anymore because we grew in size like we doubled we are now eight people working on the JDK on the chip on the JVM and stuff we have a bunch of compiler engineers ah sorry we have a bunch of GC engineers we have a few compiler engineers I'm one of the compiler engineers then we have other people working on different things like auto-tune what else optimizations for Scala for example I don't know if any one of you is interested in Scala yeah okay that's that's what I thought disclaimer I was earlier this year I was giving a presentation at Scala days in New York I don't know how to write Scala I could not for the life of me do a hello world right now so don't ask what Scala questions but I do speak Java bytecode that's what I do okay so if you're going to tweet about this talk and I hope you were already tweeting about all the other talks since you're at the Twitter headquarter you kind of force between please add that hashtag that's my team so they know that I'm over here and not the other building where they are missing me so what is grown for those of you who are here to hear about growl DM you can leave the room right now I'm not talking about this so growl VM as you probably have seen the announcements and you might have you know tried it out and played around with it growl VM is is a very unfortunate marketing term that confuses people for a very long time there has been a project called growl that's the chit compiler right and then draw BM is just an umbrella project name that includes growl the cheat compiler truffle and substrate BM right I'm not talking about truffle the substrate VM I'm only talking about the cheat compiler nothing else ok so what is called it's a java virtual machine just-in-time compiler who here knows what a JIT compiler is perfect so I'll just give a nutshell then it's basically taking Java bytecode while you run your application and composite into native code so they run faster that's what it a compiler to just-in-time compiling stuff so it's being actively developed by Oracle labs there's an official open ATK project for it the source could lift some get up and it uses something called che vm CI and we introduced that in jdk 9 under chapter 43 and it's it's basically a Java interface where you can plug in an external compiler that is written in Java and Gras is exactly that and if you okay no a little more questions people know what hots for this yes do people know what c1 and c2 are ok all right so hotspot has to cheat compilers one is called c1 or client compiler and the other one is called c2 or server compiler you might have heard about client and server before there are two different compilers and c1 is a high throughput compiler so it it produces native code as quickly as a candidate of your application starts at faster and then c2 is a a peak performance compiler it produces it takes more time to do optimizations than it produces them peak performance so that you reach peak performance with your application and gurrah well or c1 and c2 are as the rest of hotspot written in c++ okay and growl it should be a replacement for c2 and it's written in Java so if you're going to take away one thing from this presentation today it is that it's written each other and this is very important because the properties of a C++ application versus Java are very different and this talk is basically mainly to explain to you the differences if you're going to try Gras that you understand what's going on it's very important when you run benchmarks or do performance comparisons that you understand why certain things are happening so that you account for them right let's put it this way okay so let's get started so where do you get Gras okay there is chap 295 of which we did in JDK 9 which is ahead of time compilation it's basically a small command-line utility that takes in class files or char files and then it calls out to growl it's written in Java by the way so it's just you know in century it's growling and asks draw to compile a bunch of methods and then it spits out a shared library at the other end that's that's how the äôt feature in nine works and since its using grah grah is already in 9 so technically you can use it it's not really advertised or anything and so what Oracle did then in JDK 10 with chap 317 was to basically say hey there's now an experimental cheat compiler that you can use and if you download city 10 or later you have pretty much everything you need okay good as I said I'm doing a lot of demos I'm doing this in the Oracle cloud don't ask me why but I could explain it to you it doesn't matter I cheated a little bit today because I only have 40 minutes and this talk the whole talk takes an hour okay so I'm kind of in a yeah in a rush here a little bit so that the way I do this talk usually is no that's acting you know what I want because I've done it already yeah yeah this is all crap it's not actually expired it's all fine so the way I do this talk is when when I watch talks where people do a lot of demos the question I always ask myself yeah well this is really nice what you're doing up there but how long did it take you to make this work on stage right so the way I'm doing my talk is usually I go into this interface and start up and a complete empty cloud instance but this takes a while and then it takes roughly four minutes to fire this up and in the meantime in this four minutes I talk about myself how cool I am and things like that but so we skip this today because I already started it up and so we go here and then we should get a public IP address here we get this one and then we log in that should work perfect a little bigger and we're having two windows here because we're going to compare performance numbers later between c2 and grah okay and as I said completely empty now you each other installed okay so the other thing we need is young we need a kid so we're doing some good stuff later and here we're going to check a child at net 11 we're going to download 11 regret this guy and then we download it whoo that's quick I didn't even have to say my my schpeel here usually what I say at this point is we are going to have some awkward pauses when we're waiting for things to download but not this time or at least not for this we're going to need something else maybe are no nuts color days darling that's colored a couple benchmark if you guys know that one you should too old but it works and it's here oops don't do this yes there's a whole prep you know I'm downloading all this stuff right now because I'm not sure how long it takes and I'd rather have it now and done later we don't have to worry about it excellent so we have kid perfect and we almost have two benchmarks we now perfect so we go in here we set Java home and we said we put it on the path because we're going to use it a bunch of times and we do the same thing over here it's really boring watching me doing this but please be impressed that I haven't made a typo yet okay here we go Java - version perfect eleven so as you know since JDK nine we have the module system right so there is a command-line option that's called list modules this there you go and it lists all the modules that the Cherokee has it's about 75 and we are looking for margins that are called JDK dot internal dot VM okay here we go so there are three the first one is GDK darling trilby mci that's Chaney MCI that's the compiler interface I was talking about earlier it's basically to the Chava part of that API there is there's obviously a C++ part in hospital as well because you have we have to talk to out spot and then the other one jdk dot until that BM doctor is basically just drop it's it's a one verbatim copy of the github and stuck in a in a module that's really all all right let me go back to the slides name oh yes I'm talking about me right now for five minutes no I'm not working on treatments for a very long time it's a lot of fun okay now I work at Twitter you I skip this so why am I doing this talk and basically all the other talks that I'm doing but it's specifically this one I want you to try wrong and usually I'm giving my talks at Java conferences but really the the big win is with Scala and they're there a bunch of other talks that I do where I show you how much CPU we're saving here at twitter by using growl for running our scholar services so you should check that out on youtube it's called Twitter's quest for the growl runtime I might have it later in my talk I can't remember I want you to try it right because you can save money and I tell you if you are running a lot of Scala code you will see a CPU reduction and for us it was for one of our services it was roughly around the 10% and you know if you have a bunch of machines that adds up it's a lot of money then I want you to try gras because I'm sure there are still some bugs somewhere and I want us to fix these bugs and then I would also like to improve trawl even and to do this we need to throw different code shapes at it right we're Twitter yeah we have a bunch of other things but pretty much all of our services are built on Eddie and then on top of fenetre and finagle and all this stuff so we like the stack is almost the same it's just a little bit of different logic at the top I want all your code their shitty production code you're running that would be cool to run with grow for the various reasons and then sometimes when I give my presentations people come up to me and say well is it safe to use because if the next pair mental check compiler right does it burn down your data center I can tell you no it does not we at Twitter we use Gras for many of our most important and biggest services 100% in production for more than a year so every to the last year everything you tweeted and saw in your app on the timeline and stuff was running through code compiled by Gras then they asked me how do I use it right and where do I get it and especially the last two questions I'm going to today and then sometimes when I actually get people to try it they usually send me later emails or DM me on Twitter and say well you know I was running this benchmark and it really sucks and the main reason for this is that they don't understand the difference between what I said earlier C++ and Java they're looking at the different metric they're looking at the wrong metrics or they're looking at it in the wrong way okay but I'm going to explain you almost oh yeah here it is that's one so if you want to know how you can save a ton of money do this and it's it's very important when you you know I don't know I like shiny new things I don't know about you but if you wanna try this at your company for example it's always good that you can go to your manage and say well you know we could save some money you like money okay back to them no because we've already done it so how do we use it you get a chatty K withdraw if you have to have one module then you're good right we verified this and then the only thing you have to do is turn it on okay so let's see how that goes we go to chap 243 here we are and then at the very bottom it tells you how to turn on raw or at GBMC I compiler really it could be any compiler so you unlock experiment to be an option because it's still experimental feature then you tell it to enable JB mci that only enables the interface so you can access it it doesn't automatically turn on the compiler and then you have to tell it please use the JVM CI compiler okay so all these three will then replace C to withdraw in hot spot you could actually tell it which compiler to use if there would be more than one but as of tomatoes and that's wrong so it's automatic and picking this one all right so now we're going to set something that's called Chava tool options and we are going to use as you all know the P fault you see since 9 is t1 and the we are going to look at a some GC logging output later and g1 logging output is a little difficult to read so we're just using parallel for our small highly scientific benchmarks that I'm going to show you it doesn't really matter what you see we're using then we're using a rather small heap because we want a CTC activity and the important part here is that we are setting the maximum and the start size at the same size and the reason for this is when we are running with growl growl is also being executed as java code and it's also allocating memory on the Java heap and so the heap expansion would be different if we were running with Robyn c2 and then she sees what happened at different times but we want to eliminate that we want to compare somehow apples to apples so we are setting this all right and then over here we're setting this one plus all this stuff here we go and then we do a version here and it says ok I've picked up all these things and prints version perfect so now there's a flat called print flex final and after you notice it's pretty helpful it prints all that all the Flex that the hotspot has and then we're looking for the TV mci flex so these are the ones you see we've enabled JVM CI that one's true we've used the JVM say guy compiler that's true so everything's being being picked up correctly and I want this one GBMC I print properties and it prints a very long list of properties and most of them are bra properties there's a handful of JVM CI properties up here and I'm specifically looking for this one JMC I in the timer because it's printing some output some logging output when it initializes JVM CI and Gras okay so we're doing this and since it's a Java property we decided to pass options to Chauncey I and Gras as Java properties instead of - xx first options so we do - capital T and then equals true and then - version alright so there's no logging of them did we do anything wrong no we did not JB NCI is lazily initialized it's only being initialized when actually a tier 4 compilation happens do people know what tier compilation is all right so when I talked earlier about hotspot having C 1 and C 2 the way the default setup you always get tier compilation and the way it works you start your application first your code gets interpreted which is really slow and then it gets compiled with C 1 at tier level 3 and that means you get native code but the the DISA native code contains code that collects profiling data and then after a while you're running this and you collected enough profiling data it recompose with tier 4 which is either C to Agra and uses that profiling data to generate performance codes since you're walking through the tiers of your system and that's chair conformation so and I said growl is tier 4 and so it's lazily initialized only when the tier 4 compilation have is happening we can look at all the compilations that are happening for a - completely for a - version and these are all the methods that are getting compiled the third column this one here is the tier level and as you can see there is no tier 4 compilation because if nothing gets hot enough to actually get to that level so what we have to do is we have to run a little bit more and for this we're going to use just call it a cup of bench work there's a there's a - L that lists all the benchmarks it has so we're going to use this and you can see oh yeah there's some output that's cool GBMC is getting initialized there's a class being initialize called Tata JVM say on runtime then you see some stuff being configuration stuff it's being read some architecture specific stuff but it doesn't seem like it's getting to finish really and the reason for this is that printing all the benchmarks that the the cup of benchmark has exits earlier before she leaves it GV MCI is actually initialized so we need to run a little bit more okay so we run a small run of a benchmark called Aurora and then you can see audio okay there's more stuff being done now we can see the hotspot GBMC I runtime being initialized to 73 milliseconds and there's something called compiler configuration factory that would select the compiler if we would have specified it but we didn't so it's automatically picking a bra and then it initializes graz magrav runtime bunch of back-end specific things this also looks like it doesn't finish but the reason for this is that the the cup of benchmark harness redirects output to files to stand around stand there so we can't see it actually finishing but it finishes trust me okay I think that was that for now let's go back to the slides and because before we I show you more benchmark numbers I actually look at my time so I have no idea where I am but we have to talk about bootstrapping that's important that's the first important thing the main difference between growl and situ because Corral is just another java application running in your cranium it's Java code right we haven't your T compile it so we don't have native code for it so while we compile while we run this benchmark for example while we compile the benchmark methods Corral itself gets hot and needs to be compiled that's the bootstrapping part so and draw while it's doing this right it loads Java classes its own classes obviously the VM loves them it it has Java methods and these methods at some point need to be compiled so let me show you this there is a way to do an explicit bootstrap don't do this ok this is really only for benchmarking or presentation purposes there's there's really a reason to do this manually as you see it will take a little bit roughly 15 seconds or something or 20 I can't remember so what I want to show you though is if we're going to run the this one and a small version and we do three iterations of the Aurora bed it would run so it took 17 seconds to compile roughly 2,800 methods of grah the way the bootstrap works is it takes all the methods of traveling object and there are only like five if I remember correctly that actually have Java bytecode and they're very simple and it schedules all of them for compilation that while it compares these simple methods all the ground methods get hot and then they compile themselves right that's how it's hot and hotter and hotter all right that's what's happening 17 seconds all right let's do this page this guy whoopsie - and then we do the sing one over here withdraw these other small houses and not too awkward actually but okay so first iteration four point four seconds and then we go down to about four seconds okay so withdraw you see the first iteration is about five seconds and then we go down to four and hopefully again four so now see much better compiler just kidding the variance is a little flaky but so it's you see there is there's roughly a a six hundred millisecond difference on the first iteration and and keep in mind these integrations are really short I mean there are only five four or five seconds right and the difference between but the time difference between the first under from the first iteration are the additional compilations that that have have to be done for Gras but as you notice it's not 17 seconds right that's one very important point because you haven't seen this but this container that I have has two two cores okay and as you know that JVM uses threads and multi-core systems to do Chi C's in parallel and there are also multiple compiler threads right and when it has the chance to grab these it uses them to compositor by default in the tiered setup that I was talking about earlier when you run grah itself ground methods are only compiled by C one and only in Tier one so it will never get recompiled and we do this for two reasons the first reason is since c 2 is a high throughput compiler we want code as quickly as we can and the second reason is we don't want the the compilations for your application to race with the with the compilations for growl itself so that's why we do it this way and it's perfectly fine as you can see the startup maybe I should know it takes too long but it compiles roughly 5000 methods additional draw methods in this case and and the additional startup that you get it's it's roughly 600 milliseconds so startup in general is not an issue and we at Twitter as I said earlier we are running a bunch of our services on draw exactly in that setup and there are thousands of instances of these services and we deploy multiple times a week and we've never had a problem with a startup taking too long or something to timeout or or anything like this alright so brief summary bootstrapping cabassa lot of methods yes absolutely about yeah if you do it explicitly like 2500 but but it's roughly actually 5000 if you start your whatever a bigger application up all of drawl that's that's being used and gets hot is roughly 5,000 and so you can do it either pronto during runtime never do it up front really there's no reason that the only reason why you want to do the explicit bootstrap as I showed it to you is for benchmarking purposes then it's it's valid and then by default Gras methods themselves only get compiled by Z one you can change them by the way with a flag but it's usually not not necessary that brings us to the second big item Java heap usage and crawl is who is paying attention and hasn't fall asleep yet exactly right so that means that since it's written in Java all the memories using to do compilations is being allocated from the Java heap C to Malick's from the C native Heath but Gras allocates from the Charlie there are pros and cons for this one one advantage of this is in a container world and we're all in a container world right now you have to yeah you have to reserve memory for achieve Liam to do compilations and they can get rather big especially when a methods are large you do a lot of in lining loop unrolling polygraphs can get big sometimes multiple 100 100 megabytes a man with the biggest compilation I mean it's really we've seen them once ok but the biggest compilation we've ever seen at Oracle was one gigabyte ridiculous but you have to at least reserve 3 or 400 megabytes so that c2 can do its job and you have to think there's not only one compelling thread right yeah usually too so and if you're outta memory in your container boom your net with grah you can technically take these additional let's say 500 megabytes of memory and give it to your child because when you start up to actually have slightly this yeah because most of the compilations for that are happening are happening in the first 30 seconds 1 minute 2 minute right then it tapers off there's still some company she's happening but the maturity is at the very beginning and at that time your applications aren't even fully up yet you know later yes when it's fully up it probably uses all of the Java he but at the very beginning it's not so draaga can just take some memory do the compilations and then when everything is compiled and done then your application can take the memory all right Java heap demo yes please let's do this we are going to use the same benchmark and we have log in GC here so I'm running this with C - Aurora is a very compute intense benchmark it doesn't do a lot of object allocations as you can see after the first iteration we only have 40 megabytes on the heap and then we collected it down and then only 32 collected it down so there's really there's really not a lot going on and if we do this over here with grah then you can already see yesterday you see activity so in the first iteration we allocate 130 Meg's collect down another 140 collect on plus an additional 16 so these are all the memory that that has been used to compile the methods for the benchmark remember growl methods are compiled by c1 so they're still using malloc memory but much much less because it's not doing so many optimizations so this is all for your for your application in this case this tiny little benchmark and then the second iteration oh yeah their before and after at benchmark iteration the harness does a system to see to clean up the heap so these are the ones in between second iteration yeah we still collect 135 and then down but in the third iteration there were no allocations anymore so after yes granted small benchmark but after 10 seconds we're done compiling everything right so it's usually not a big deal and I can tell you from experience at Twitter we've we've not seen a problem with this we've never had yes I should mention this because I I should one thing that could happen technically is that you were later in the game actually using all your chocolaty and you have one megabyte of memory free because they are all tuning like crazy and then suddenly a compilation comes in that needs 50 minutes what will happen out of memory error the problem is since there is no heap isolation as of today there will be in the future but as of today there's not the out of memory error could end up on any frame if it ends up on a compelling read you're lucky because the compiler will catch it and say oops sorry okay I can compile this but if it ends up in the application your application probably can't handle them so Krog uses Java heap memory and there's no heap isolation yet as I said most memories use during startup that's where most of the compilation happen for your application very important memories being used anyway right it's just math it's just different from where where where it comes from good how much time do I have left i okay I can't really distant I would show you how to download draw compile it and then run within super simple I've given this presentation before so you can watch a full version on YouTube it's basically cloning and then running MX and it builds a modular jar file and and that's it yeah instead I can also not show you this one I would be interesting I would run a fenetre example that's called Twitter clone it would basically show you that the the startup overhead is really magic of all it's roughly a second and that's it and it's not getting more because once draws one one once all the compilations for growl methods are done they're done right so don't worry about this this one I'm doing because you guys kind of people oops alright so we're going to increase the the heap size a little bit because otherwise we'd get too many collections and two gigabytes excellent and then we run a default size and we do it twice of a benchmark called factory this will take a while same over here two gigs okay in the meantime while this is running I go back here and then benchmarks and then this factory here and it says the factory benchmark tool kit for deployable probabilistic modeling to extract topics use it latent tear each allocation I have no idea what it is doesn't matter it's written in Scala that's that's all I know and funny thing I give this presentation once someone who can't remember so after after my presentation guy came up to me and say wow yeah cool cool presentations we are using Lda what it's that thing there they're using this in their company I don't know does anyone know what it is really my god very good I don't know have we met before no okay still running yeah these are the awkward pauses at times running you already showed me the five or do I get now x over x out - okay no that might work we're almost done on this side the first iteration took 47 seconds and then 46 and so we're running this one here withdraw and we have to wait for it unfortunately I could answer a question in the meantime [Music] so the question is what's the what's the overhead of memory because of the compilations withdrawal I'm not sure off the top of my head so when you turn on print compilation for Grall it actually tells you how much memory it allocated because it's pretty simple to do it in java to figure that out hey Pete maybe a hundred mix I don't think it's it's more than this because we don't have a lot of very complicated methods so it's I don't think it goes beyond that to be honest okay well this is done and it took 25 seconds to run this thing this ridiculous rate I mean this is all this is like 40% improvement or something like this the reason as you can see on one of the reasons is we did here 53 to 95 that's roughly like 40 GCS ballpark and over here it's 32 to 53 so it's 22 Z's so we cut the cheese in half and the reason for this is gras has a better in lining implementation and additionally a better escape analysis implementation I don't think we have the time now to talk about escape analysis but there is like the traditional approach and then there's something called partial escape analysis and it's just able to eliminate more object allocations and this my Scala friends is very important for scholar as you might know and this is this is the result I mean this is the best benchmark I can find right I mean it's not representative for what we see a Twitter but it can do that and this is why you also should try draw for whatever Scala stuff you have because it might do this for you we're seeing the sub Twitter it reduces memory allocations what it then what's a result of this is that latency p99 times go down I I have a target just a that Java one as like to call it where our p99 times go down by 20% just by turning on bro okay I think I didn't forget anything my summary I think you know what to do right all right so please please please you're the right crowd when I talk to Java people is usually known or whatever for you it really matters I cannot tell you how much money we're saving by running withdrawal but it's way more than I get paid that's a high check yeah that's it thank you very much tweet about all the sessions [Applause]