scale.bythebay.io: Pankaj Gupta, Adaptive Scrooge - Adaptive Thrift Decoding
Recording: scale.bythebay.io: Pankaj Gupta, Adaptive Scrooge - Adaptive Thrift Decoding
you hello friends my name is Pankaj Gupta I manage the core data library's team at Twitter my team provides libraries like summing bird scalding for processing any kind of data a tutor batch or real-time what I'm going to be talking about is an optimization we did for decoding thrift objects better for certain use cases and this is a use case that is very common at Twitter I would imagine it's common in general so the way I am going to be doing this presentation is that instead of talking immediately about what this optimization is I want to provide some context and then I will take a case study example and then walk through what the problem is and how adaptive School solves it so bear with me I'll come to the actual implementation details but in a while so let's start from zero what is thrift right most of you probably know it but just to reiterate it's an interface definition language the idea is that you define your data schema once and then you can generate code for it very easily in any of numerous programming languages why do you want to do that because it supports schema evolution you can keep adding more fields to it all systems can be new data or new systems can lead all data all that stuff the wire protocol is pretty efficient you need to need to keep reinventing it it provides language independence your different parts of infrastructure can work in different languages and talk to each other for our purposes it also supports efficient code generation because the code generation is automatic over a period of time we can keep evolving it and providing more options and adaptive Scrooged is another option that it provides you so you just drop and replace and benefit from optimizations and this is made possible because of code generation in thrift second part is Scrooge so what is Scrooge Scrooge is a thrift code generator so thrift comes with its own code generator which generates Java code which you can read from Scala but it's not very EDM in the sense that it generates mutable objects and as you know in Scala we don't like mutability so Scrooge is built for creating idiomatic Scala code these objects are immutable and also they make this nice design choice where your thrift types are actually traits they are not implementations like in the Java implementation and what that allows is that you can change the underlying implementation of those types without changing any of the user code so your application continues to run the same way you can just plug in a different implementation and that is what also makes adaptive Scrooged possible so what is the problem that we are trying to solve the problem is that many jobs they access very few fields but they end up deserializing the entire payload by job I mean a real time job or a Hadoop job or even a service anything that is reading thrift object the way you would read is is that you have a data source and the data source is built for general usage not just for your job right so it might have a lot of stuff and usually you are interested in a subset of it but you end up deserializing the entire object and which in many cases might be more computation than you are actually doing it might be the major portion of the computation that is happening and I will go over an example of that so the typical way of this is a well-known problem that there are many jobs which want to access a subset of fields and the typical solution has been to use a columnar format like part K or o RC so what is a coroner format in a columnar format the data is stored by columns and not rows what that translates to is that you have a stream of events and each event events data it's stored together which is what we call row so you have the bytes for the first event then the bytes for the second event and so on this is the way data is regularly stored in columnar format you store data of together not for the event but for so let's say your event had five fields in it let's say name a gender blah blah blah what you would do is that you would store data for a column together for the name fields of all the events they would be together as contiguous bytes and then the other column and so on and so forth the advantage is that you can read the data for only the columns that you want so if you were interested in name and age we don't need to read all of the other columns right so this is a great very good solution so why not buys it not a good fit for all use cases the reason is that there are some cases where it doesn't work yet and in some cases it may be theoretically can't work as well the first case is real-time data as far as I know Kafka streams and not Augustine good Kafka and other queueing implementation that we've seen they give you event at a time and they don't support production push down right into the stream you still have to read the data even at a time so you don't benefit from column row formats there even for batch data we've seen in practice that if you have lot of columns then this breaks down for example if you have a schema where it has hundreds of columns it's actually faster to read it row by row then reading it column by column because every column adds an overhead for a few columns like four or five it doesn't matter but as you go into hundreds of columns the overhead actually kills performance so in those cases practically we still don't use parking so there are actually a lot of data which doesn't use a columnar format you could say that you could have data in both formats right and depending upon our use case you use this data or that what storage is a huge cost doubling the storage is also very very expensive so in the end you end up with one format and then you end up in a situation where you're back to square one you have a large object you only want to access a few fields and that's the use case that prep of scrooge addresses so as I promised let's start with an example this is a very simple application event in event out let's say your services are generating events and somehow it goes to adamak sir right it could be that you're writing to some low categories that prescribed or floom it goes into HDFS then there is a d-mac serve which does categorization right you categorize this event into separate horizontal categories this is a practical use case which we have for some jobs and let's look at this event here this this is a typical event where let's say there is a type field which is an integer this is all you new need for categorizing this data what category does this data go into is dependent upon only this field but there is all the other data in the event like this could be an expensive string that could be a huge map sad and it could have nested event structure which is very typical at Twitter like we have like 10 levels of nesting see so in this you just keep in mind that we only need the type field here which is integer now let's go inside the d-max what happens you get this event which in raw bytes form then you deserialize it which means that you create object in JVM memory which has the map materialized thus at materialized all the nested structures in there all depths as well as the type field so UDC realize it in this in memory you take up a lot of memory you do a lot of work for digitalization and then you check the type field and categorize it after categorizing it you write it two different categories it could be different Kafka streams or different locations on HDFS for serialization you take this materialized event in memory and then converted to one into bytes now you can see the actual logic that you are applying is very tiny it's like almost nothing we checking a field called type and deciding category but all the other work that you're doing is so much more expensive materializing this stuff in memory and then your GC can go up because of that and see utilizing it back again typically for such a system you would actually write custom code right because this is like a very this is a pretty this is a very specific example where this problem blows out of proportion I chose it because it shows this use case very very well but typically in a when you're working on this project you will say like let's put the bytes with the object let's not DC lies it again you light a lot of custom code that you have to write test maintain all of that if you don't do that then you pay the price so those are the options now let's see how adaptive school solves it and then we look at how it works internally so at the top level what adaptive screws does is that it learns from the exes pattern it learns from your executing code and sees what fields you are accessing and it figures out that type is the only field that you're accessing and then it DC realizes only the type field it skips all the rest when you read the bytes it would skip everything else as fast as it can the code is optimized for that and it materializes only the type field and it also changes the representation of the event it completely gets rid of the fields that you don't need so in your memory you do not creating the map thus add the nesting structure all that is gone and then after you categorize it and you want to send it to different categories serialization is just copy you take the original bytes and just copy that over and you already had the bytes right you got the bytes already so it's not any additional memory location it's just a simple copy which is a very very fast operation so overall you back to or close to your actual logic and avoid this overhead that is the use case for adaptive Scrooged let me go over the details of it up to Scrooge one more time so it learns from the access pattern it's similar to just-in-time compilation in Java like in JIT you look at what portions of code are accessed and you optimize those code to run with native this is a similar idea we learn from which fields you are accessing and it modifies your code to be optimized for that use case in this case all the type field is materialised we don't materialize rest of the field it does carry around that the original bytes but those were already allocated so it's not a new memory allocation there is an implication to carrying it around and we look at so the downsides it saves memory usage because you're but realizing very tiny objects the part that you're accessing you avoid all the rest the rehabilitation is free not just for the actual object but even if you were projecting and looking at a nested field inside any level of nesting it would basically push copy the portion from the original byte array that you need so still a copy even for projections it eliminates entire national structures and it is a drop-in replacement it's very simple to replace it your code does not need to change at all with that in mind it would be not fair to omit lazy screws here lazy screws was the step that we had before we went to adaptive Scrooge this is not a completely new idea this already existed for a long time and we've benefited a lot from it in practice it gets us halfway there what it does is that it makes the strings lazy and as I turned out for a lot of our loop jobs just decoding the string was very expensive and lazy Scrooge made that lazy so only if you access that string it would decode that string four bytes so it was a step in the right direction it also avoided realizations so it also carries around the original bytes so if you want to ricci realize it then it's just a copy so it exists it solves those two problems but it does not solve the maps and sides case those are still materialized and sometimes those can be expensive and it has it still goes through all the necessary structures so only the strings are lazy still has to even if you're not accessing the entire big nested structure it still has to go but it was a good first step that led to wrapped up Scrooge let's go deeper into how our depth of Scrooge works as I said earlier it learns from field accesses at runtime and learns and adapts based upon this information so first part of rapid scrooge's which i call learning is that we wrap the original script object so whatever the default thrift representation is we generate code using that and we wrap that in a wrapper where whenever you access a field it updates some counter somewhere and you can specify this learning period for how many events you want to do that so in this first process let's say you say 10,000 events for those events it would through this wrapper learn what fields are getting accessed and at the end of it it does the adaptation so what it does is that it modifies the decoding code where you are converting bytes into an object representation and it generates optimized code which skips the bytes as quickly as possible there are like zero locations there it just like reads to the bytes to get to the next thing that you need and very quickly cheaply skips data and second thing is that it does is that it also it also creates a different implementation for your thrift object the one that does not at all have the fields that you don't access so in this case this adapted event only has the field that you access and this decoder generates that representation of the object right so pretty simple at the top level you're rewriting the decoding code as well as the representation of the object that you materialized in memory for the user codes the same interface it continues to work the same way the way adaptation is done is via a bytecode manipulation we take the actual generated code at at runtime we use ASM to manipulate the bytecode we put hooks in the original it'll bytecode so that at runtime we can modify the bytecode and load the modified bytecode for that class at runtime using a class loader so why did we choose ASM we did not actually start with ASM we started by generating you know the code as a string at run then using Scala compiler libraries at runtime to load this code you can do that right it was very very slow it would you take like three to four seconds for the big thrift structures that we have at Twitter and some of our MapReduce jobs they run for only like 30 seconds so this is a huge overhead like 10% of the time you just like it's not viable at all and Scala compiler is a huge dependency like some of these services there in Java they don't even end up on Scala now there is another thing that they need to depend upon ASM so we tried ASM then and like focus on optimizing this and ASM works pretty well we were able to bring this time down to 100 milliseconds and this is for a very very large script but typically it would be like order of few milliseconds so that was a huge improvement and now it's viable like even for a 30 second map job this is like a third of a percentage it's like basically noise so that becomes viable then even this hundred millisecond pause though could be problematic from strums for some services they can't afford to have performance go down even for 100 milliseconds so this transformation happens in the background and then it's applied atomically so there is no pause in the application when you use this another advantage of SM is that it's a tiny library it doesn't depend upon many of the libraries it's like pretty simple library and it's a very lean dependency so when you bring in a SM you're not bringing like a whole bunch of libraries with it editor we depend upon finagle a lot most of the applications do and finagle already depends upon the SM so for us it's like zero there is no additional cost to depending upon a SM but even for outside like if you are using screws this is a very tiny dependency that you need to add so one aspect of adaptive screw just fallback so you did this learning what if you learnt wrong right somehow for the first 1,000 events this field did not get accessed but later on it got accessed so in that case because it has the original bytes it's able to fall back it will go back and deserialize the bytes again and then you work around that so your application will work you're not tied into that the learning has to be correct it was silver correctly and the fact that you did not see these excesses in the first 1000 or whatever you configure events means that this is like a rare event and in that case if you pay like you know 20 30 % extra performance that's fine and you can tune also like how long you you learn second fallback is that if we realize that all fields are exists then we fall back to the original drift because the whole point of rep to Scrooge is that it's optimized for subset of Alexis's and we don't want to add any penalty if you're accessing all the fields in the object and there are actually cases where it's very common to access all the fields for example if you are partitioning events grouping it and you need to do a hash code of the event the hash code actually goes through all the fields right or if you're converting the data from one format to other let's say you're converting from roof to parquet you actually have to go through the entire event so there are actually practical cases where you end up accessing all the fields and there you don't have to pay any price it would learn and then fall back to regular thrift there are a couple of tuning parameters you can train for how long so for example bad jobs they hardly ever change in the X pattern so you can have a small portion of reach there for a service that is running for days you have more flexibility you can train for longer you can also decide at what percentage access does it feel get materialized it may actually be reasonable to not materialize a field that is accessed rarely so like I said that if that field is indeed exists will fall back to the original thrift and we can decode it there is some performance penalty to it you can compare that to the cost of deserializing they that every single time for example if a field is getting accessed you know two times out of ten thousand you might want to like ignore it don't digitalize it in the regular case and as a man it is accessed deserialize it so those are the parameters that you could tune typically you don't need to do that much so as we know in software there are trade-offs there are very rarely you know complete bins there are some trade-offs with electric schools as well so it it does not account for changes in access pattern so if your application is running for days and days and now there's pattern changes it doesn't know it did the training and now it doesn't know and we could fix that but we haven't seen a good use case for that typically services get deployed pretty frequently and the code is pretty presentable what fields you access and such it could be built but we haven't built it yet second also is that the nested structures they all access the entire thrift blog right so if you have a natural structure and there is a tiny nested struct inside that you're cashing in memory it will actually with itself X store the entire byte array which could be problematic for caching write notes carry around the whole byte array in that case you could clone the object so there is little bit of work required in this edge case this out talked about already that if you do a hash code right now it would materialize all the fields and adaptive Scrooge is defeated but this need not be the case we will probably optimize it in future for calculating the hash code you don't need to materialize the object we could infer that from bytes itself another downside is that the jars are a bit bigger because now we generate more code so your bundle that you deploy could be a bit bigger depending upon what portion of your application is generated code this may or may not be an issue for you there are some wild cases we've seen where this might matter using it is pretty simple basically ultimately this boils down to converting by its two types you provide your settings you create a DC réaliser and then you can carry out type from bytes pretty simple these are the results so we built a benchmark for it which is part of the open source repo you can run it if you want and this is off this is this benchmark is built to show the utility of a wrap to Scrooge so take it with a pinch of salt but in this case there is a big schema very few fields are exist it's actually 20 times faster than regular Scrooge and 10 times faster than lazy Scrooge but this is built for the OP case fit for it we also saw 10% real job production job that runs and which was actually candidate we thought was fit for this optimization there we saw 10% improvement we ran another benchmark with not just a desolation part but actual real-time topology so this is a real-time topology running on Heron with a simple map and some by key which is counting events basically and in this case the spout is where most of the deserialization happens right and we saw 25% reduction in CP usage on the spout so this is not a real job again because this is a benchmark it has to be repeatable so there are some differences it's also built to show the utility of relative Scrooge so coming back like is it better than Parque oversea no if you can use common formats it is they are better because they do less file i/o they do less Network reads and less decompression the data is typically decompressed on HDFS from where we read it but for the cases where it doesn't work which are very real at Twitter for that use case and I prove Scrooge is optimized and built in conclusion you can DC relies only what you need it's possible thank you [Applause] you