scale.bythebay.io: Flavio Brasil, Building a high-performance Future
Recording: scale.bythebay.io: Flavio Brasil, Building a high-performance Future
so yeah thank you for attaining my talk today I hope you who enjoy it I'm going to be talking about an open source project that I worked on that is a future implementation and also about the tools that I used for building is high performance future so it might be very useful if you are working with optimizations and going to have a high performance cold so just some background on how this came to be so I now work on the core data library steam here Twitter but previously I was working on the tweets backend service so as you can imagine like we have 330 million users right and for each read request we have five request charter services and 99.99 requests of the requests are reached so imagine like this scale that we are cooperating right so any savings are considerable in terms of like cap and usage of hardware so when I started profiling the system actually the business cold didn't even show up on the profiling tools so that's a super-funny like I didn't see anything like about tweets there so most of the resources were being used to do our pcs to run futures and do batching of calls to other services so the first main target that I started working on was the stage free moment it's something I don't know if you know is staged but it's a way of batching calls to remote services that is more convenient and it's a free moment stage is not free open-source yet so there is another project that is very similar that is called clone I actually works on this library at Sound Cloud because stage wasn't open source so it's actually just a version of stage and you can have like an idea of how search works looking at the clump the computation so this is one called it calls track service and then gets a bunch of tracks given a lot of track Chinese and then for each track ID sorry for each track around the track service I get the user IDs of the creators of the and then I call the user service you get the users and then I create like I have the tracks I have the users I create the map that is between users and the actual user and then I create this in rigid track that has the user and the track ad and that's really like here it's not so complex but if you see like calling many services and doing batching it's really really complex so what stage in curves does is allow you to express things as single single single single key computations so here for instance I am traversing the track IDs and then I call the track sir source get on track get one user but this is not going to do two calls for each element of the collection the free monads the interpreter interpreter is going to make calls batched to the external services to the track service and to the user service so that like makes the code much much easier to reason about and much easier to read but at the same time it's really hard hard and challenging to optimize this kind of code one problem is if you see the future version you are calling the track service and this is creating a future right and then you can call flatmap this is another future it's continuation of the future and then you call user service and map so you have total four futures and that's like not to make too much but with the version using clone port stitch we have to instantiate one stage one clone for each track source track stars call for a specific track ID and one for each user source call right so you have exponential I don't know how many tracks we have like I think the batching that size was 20 tweets sorry 200 tweets and we would have like 400 allocations here so that's really really inefficient so we did something so we switched the implementation turning limitation to use Java collections in a race that's really terrible but that's what we had to do we also use while loops beautiful internals as well to be able to optimize the X fusion we avoided many allocations I think that was like the most educated objects in the system was at Apple in this column people normally do like if I have to do a pattern matchy match I have two objects I create a tuple with two objects and then match them just creating this temple was the most allocated object in the system so yeah so we just changed them pattern matching should be a nested pattern matching and we also implement created a new implementation of stage using arrows that's I think it's really really really interesting it's probably another talk but if you want to talk more about this that you know I'm going to give you just an intuition of how ever we use arrows so there is this suite I saw actually when I was preparing this talk then the person's has three moments should be seen as a rapid as rapid sessions not ESDS that way it's easier to understand that they can't be analyzing their entirety just the first rep align or that means everything before the first flatmap call so if you have a task or you have any free moment you know everything that's happening until there is a flat map call the flat map body is going to be executed only after the initial composition in issue s ast is executed so this means that we have to allocate and etc so the way that we solved this is using arrows so when we instead of doing Traverse and calling tracks force user source we create an arrow that is a static representation of the transformation transformation and you use this static representation for building the flat map something similar to be to your flat maps so that saves a lot as a lot of allocations and a second target that is the main topic of my talk today is the trader future to optimize the trader future so the trader future is battle-tested it's being used for a long long time at a really high scale by many types of systems that were there and other companies as well it's really well optimized well as well like it was written by very good engineers and I decided to go with clean room approach so instead of trying to optimize the call itself I decided to rewrite and actually just get the api's and try to rewrite and see how I could like the type of optimizations I could implement so my goals with this implementation was first avoid the locations that Cynthia was a problem we have many many systems that are GC bound so applications are really problematic have code that is more friendly to the just-in-time compiler and where's another goal and also have future parity with the Twitter future that is having the same API have local support in interrupts support as well so I worked for a while on this project and release the name is train I old future and there's something interesting it's in Java 8 so I know it's like mostly a scholar conference so that's almost that's movie but yeah so you're going to see some Java code please don't worry like it's not going to be terrible but you're going to some certain Java code so this is how just to compare how these new implementation compares to the Twitter future at the time when I reduce the project this is a benchmark and it's very simple just create a promise and map ok and the map is a cached function so you have less noise so this is profiling session using your kit and I was wondering how many allocations do we have for this transformation using the trader future so their answer is six objects and total of 128 bytes that's a lot yeah and then the new version it has actually only 2 object object allocations so it's one promise and one continuation that is the map and only 48 bytes of allocations so that's a huge difference right and also in terms of throughput it's much faster so the trader future can do ninety nineteen million operations per second and the new implementation what's happening sorry the new information can do 72 billion operations per second so it's a little four times three times so I would like to just to show you how hard it is to avoid allocations especially in Scala let's say let's see say that we have this code here this fine person we have a collection of people we have name age and we filter the collection by the name and age okay so how many locations do you think we have here is it one application just for the collection is it four locations is it eight locations or is it like in allocations depending on the size of the collection so who thinks it's one location what four locations no one eight locations okay you know one so it must be any allocations right okay cool so you are actually right so this is a profile session of the execution of these methods passing a collection that has only one element so as you can see here we have four applications we have the collection in a few other objects being allocated so what do we have where are these allocations coming from so they end up calling the method filter implementation from this color collection implementation so the first allocation is easy to see it's the new builder it's creating the new collection right the second allocation comes from this plus equal it's going to create an immutable list for each element so it's head and tail head and tail and create like this graph of head and tail objects and then the other two locations are not really so easy to see so the other one is the closure that the filter operation is creating so this is a function that's going to be an objects in memory it's going to have 16 bytes of object header and also two pointers one to the name variable in one part to the age variable so and each pointer has eight bytes as well so you have like some usage of memory here and the last allocation can someone spot where this location is yes the far lobe so it's called a sugars far loops using far each so you would expect like something like Hawaii loop but it's actually this dot for each and then a function that is the body of the for complication that's this little green and this far compeition body function is going to have two pointers actually three pointers won't you be won't you P and one two is flipped so yeah if you'll see like it's allocations everywhere and bytes everywhere and that's something that's really hard to see you using Scala code so now I'm going to talk about some optimizations that I used and we also implement it at it for the Twitter future so the first one is avoiding closures this is a collect method if I made it by the trader future the initial version before the optimization I don't know if you can see but there's a High Line highlighted here that is a curator dot next up respond and this is creating a closure here right and this closure has four pointers so it has pointers to FF FF size results counts and B okay and this is like each one is eight bytes if you are using compressed ups if not it's sixteen bytes and that's a problem because the eye curator is going through all futures and for each future creates one closure with eight point eight pointers right so that's not efficient instead of having one object with five pointers for each future we can have only one object that has all the pointers so that's what the Train trainer your future does it has this collect promise okay the clock promise has all these structures that we saw before so it has the results it's thick it's a promise itself it has a result and also the count and the list of results so you are saving instead of allocating this object for each element of the collection future future collection you allocate only ones this object so the CSL team has been working on these changes and Vladmir works on this one and it led to 50% better throughput and 18 fewer locations right 18 percent fewer allocations and that's nothing interesting because we it's only about allocations the organization right but the gain like it's mostly on throughput because allocating things are really expensive it's not cheap actually so and that's only the work for allocating there's also the work for garbage collecting and etc right the second one that I'm going to talk about is the merging interfaces technique that is here I have the joint methods that is it get it gives you give it a collection of futures and it returns the future that will be fulfilled when all futures finish and it has these three structures here that is a counter a promise and update function so we can merge all these information into a single object and that's what the trainer your future does it has the giant promise and then the joint promise is a promise itself as the counter has the list but it also is a responder that is Marla is like the closure but that's the Java version of it and julien from CSL also works on this change and she it's more or less the same result up to 50 53 % better through foods and 18 better in terms of allocation so the CSL team has been working on many optimizations I wrote like a design doc and they have been delivering many changes and they are like working on other optimizations as well and we they implemented only the changes that are non-breaking so they don't change behavior or API and the results for the tweet service is 16% mark we'd served for the same CPU usage and four megabytes per second fewer allocations so that's if you see like the size of the cluster it's huge it's a very good saving and we are just like halfway we still have the breaking changes to work on I also works on specification for this color future it's very similar to the implement the document that I have for these Twitter futures internally so you can check it out tiny URL coms the slash it's color - future and it has like many years on how optimizing we could optimize this color future and Victor clang is also working on a new version of this color future that implement some of these ideas cool so now I want to talk about the tool set that I've been using to do this kind of optimizations so the first thing is that's like the best tool for working with performance is benchmarking and if you are using the JVM you must use J&H that's really important it's the because you can make so many mistakes if you are not using jmh it takes care of many things that you could do wrong all right and also using J image is not so easy because the just-in-time compiler is always changing your code and it's hard to understand how it's changing your codes so that's a few have that you have always to mind when you are doing benchmarking I would like to share with you some options that I see people sometimes not using with jmh and that you can use so you can enable the built-in EGC profiler that's something that's really useful to see how your because sometimes you can have something that's really high throughput and it's generating a lot of garbage so you have always to mind how much you are generating in terms of allocations there's this option also to fail this execution if the benchmark fails I've seen benchmarks doing benchmarks that only check that the cold fails super fast basically because the default behavior of the game a mage image execution is only continuing executing if there's a an exception so you want to see the exception and see that your codes feigning it's also good to set the j JM j PM arts so you can have the ice table execution across different environments one thing that I use is Farsi as you see between runs that's something that's useful if you have benchmarks that are isolated with very low noise it's very really good to have that you see between rounds and the final final option is generating a CSV from your results to make it easier to analyze the results in compare another type of tool that I use is profiling there are many profiling tools out there the one that I use is a commercial product that is your kit I've been using for I don't know 10 years so I don't know if others are better I just know this one practically so you can export other profiling tools and it has free licenses for open source projects that's a that's very useful but the profiling is at the method level you can't can only see if a method is expensive you can see what's expensive within the method that's something that's a big difference and then there is this other tool that is like for advanced profiling and it's really really amazing I love this tool but it's also commercial and there's no free licenses for open source so that's a problem that is the intel vtune amplifier so it only works for Intel processors on Windows and Linux there is no Mac support that's so problem ok I actually decided to buy a PC just to run this tool it's more low-level so you can see how your cold interacts with the CPU caches you can see how your code interacts with at the assembly of level for instance with the processor and you can see within the methods you can see that there is something the methods that is more expensive than the other things for instance and you can also see a source versus assembly code that's being executed by the chicken sorry executed natively using digits in time compiler so the outputs of the assembly quotes that's going to be generated by the machine so I have this example here that these are some code from the future implementation that I have in Java and it's well that's used a lot and it's not really easy to see what's like the most expensive operation here but first it's just assigning a variable to current and then there's like some checking this is changing the state of the future and then I was wondering what is like the most expensive operation of this method because the your kit provider was saying that this is expensive right and the intel vtune amplifier said to me that it's actually assigning the state to the current variable so why is that right that's because the state is a volatile variable so reading the volatile is expensive at the cpu level because it has to do some walking to make sure that the things are consistent at its CPU level and you can it's amazing because you can also see the assembly code right so you can see exactly what's being executed on the processor level that's really really cool and the last tool that I use is JIT refining there's a project that is called JIT watch you can enable options on your JVM to all to put JIT compilation blog right and then this fool can read this log and it's really great like there it's many features you can compare the source code versus the JVM bytecode versus the native code the assembly code that's really cool but you're the only feature that I normally use is one that is really great these suggestions so you can just like open your open your log and click on suggestions and it's going to give you like all the suggestions for like for instance this method is expensive and it's not being lined because it's just the cap collie like who is calling this method is too big so I could try to see how I can minimize the size of the methods try to England is called I actually tried really hard but this one's really hard line yeah I couldn't so that that's something that's really interesting for refining your just-in-time compiler output so just to recap recap benchmarking using j j MH profiling you can use your kids or other tools advanced profiling using fetal I don't know other tools that are similar I haven't seen any other and do it refining using jet watch so that's like my recipe for doing advanced perform performance work so now I would like to show you some benchmark results using this implementation here I have just one operation that is napping a promise so you have a promise that is something that's not fulfilled yet just a promise Kenny this is a result the Train are your implementation can do I don't know how many 50,000 no five hundred five hundred thousand operations per second more or less and the other is I like really not good compared to this one right so this color future is really not so good Java is Inc so Java has two modes that is day sink and sink mode the async mode is not so great but the sink mode has build performance and the Twitter future is not really optimized well for this scenario and the memory profile the allocation is also better as you can see here like Twitter allocates a lot its color allocates more last and Twitter and trainer you always like if there's like very few allocations and this is the other one is mapping a constant future future this is where the trainer your implementation is much much better if you have a constant future if and it can be much much faster so as you can see here it's yet orders of magnitude faster than the others and allocation is also very low so what about colors at i/o that was released yesterday right you maybe are asking so actually I didn't have much time to and is the benchmarks I think limitations you know like all implementation is done or if it's correct but I did some testing to see how it compares with it and it's actually compatible it's I would say that's a good implementation this cause at i/o but there's like some scenarios this Trina yo is faster like few times here are these gays but there are some cases like this one I don't know why like I have to analyze I couldn't analyze this one but this is like the scholars edge is faster but it's I would say that's a good implementation as well all in terms of and then we profile profile and throughput so and that's all that I have like my team here at where there is hiring so if you want to talk to me like come and after the talk or he can yeah I can say to you like what we are working on the exciting projects that we are working on so feel free to grab me after after the talk and if you have any questions and prefer questions now [Applause] yeah I know about this other tools and we use it here them at Twitter but I have much more experience with New York hit and I think your kit has more features for especially for memory profiling it has like some animals you can now analyze the heap much better with your kids I think yeah any other questions okay thank you [Applause]