BDSBTB 2015: Ryan LeCompte: Quantifind's story: Building Custom Interactive Data Analytics
thank you appreciate it I'll try to keep it somewhat brief as I know you guys want to go for those tasty beers soon great so yeah a little bit about myself I'm a software engineer over at quantifying leading our back-end infrastructure team I've been there a little over two and a half years now so brief outline of the talk will we'll talk briefly about what quantifying does as a business we'll talk about some of our technical challenges various use cases that we have we'll talk about some existing technologies that we've tried to to solve that for those use cases but the bulk of the talk will be centered around some custom infrastructure that we've built at quantifying to effectively address those use cases and it's become sort of the the the main set of infrastructure upon which all of our data analytics products are built on and we'll talk about some of the lessons but lessons learned if we have time so what does quantifying do so at a high level we're trying to find in 10 full conversations on social media sources like on Twitter and Facebook that correlate to a customer's KPI or key performance indicator and essentially find anomalies between those two when you tie them together and try to serve us up actionable insights so if you look at the input from an input output perspective what's the input going in would be consumer comments tweets on Twitter comments on Facebook and and other data sources that we that we ingest and third-party financial data in the form of a KPI that we actually get from from the client and what which will be output are the top level actionable insights and interactive exploration functionality which the custom infrastructure is directly supporting this ability to sort of interactively explore and ask questions about the data so some of our technical challenges it really boils down to this very large multi terabyte annotated data set that we have that contains billions of comments tweets and other items that we've collected over millions of users right and we basically want to ask questions over thousands of different dimensions basically flip the data around in various ways we want to slice and dice very quickly over any any dimension and we want to perform some fairly non-trivial operations on this data set we really want to ask questions at the user level so one of the key themes of the talk is you know a lot of times I think our gut reaction is to sort of pre compute things or aggregate things offline and large Hadoop or spark jobs and then present sort of that pre aggregated data set and sort of like a dashboard and the user is only able to slice and dice as you know basically at that level of that you've aggregated the data and then if you want to add slice and dice in a different way you usually go and update your Hadoop job and spit out some different type of aggregations so so just just keeping the aggregation bit in mind as we go through the talk so so quantifying does a lot of different things but sort of that one of the three things that we do that I'll talk about is use cases to have in mind as we go through the architecture the infrastructure are these three things here so flexible time series generation co-occurring in grams which sort of comes up a lot in machine learning NLP type problems and something that we call cohort analysis and I'll go through each one of these so you know time series is essentially a time series of of consumer conversations of people commenting on a source like Twitter or Facebook or some other source and so if you look at this plot this is really just the frequency of of conversations that are matching a particular date range and a particular set of keywords so what we want to be able to provide is to be able to generate this time series much like you would from it's from a traditional time series type database but we want to sort of take it a little bit further and be able to do sort of some custom logic such as arbitrary date benning we also want to do sort of user level dee doop and different arbitrary periods so in graham co-occurrences I'm sure most of you given that it's a big data talk you've probably come across in grams in some form but basically in grams are just sliding windows of tokens in text so one of the things that we do a lot of quantified as we're as we're analyzing these conversations as to sort of break down the text into in grams and perform all sorts of interesting algorithms on top of that so being able to do this very quickly is important to us to generate these by you know grams by grams and trigrams so another motivating use cases cohort analysis so the main example here is you know let's say that there's a release coming out for a movie let's say gone girl and what we'd like to do is essentially China China's spotlight over a group of users that are talking about gone girl that are showing some sort of interest in the movie as they're reading reviews or whatever and essentially be able to shine that same spot light over that same set of users after the movie has actually is actually released so that we can do some interesting algorithm and and further analysis on that same group of users so we call that cohort analysis just following the cohort over time so I'll talk about some of the technologies that we've tried to you know over my time at quantifying to sort of satisfy our to build our products on top of so sweet ride spark postgres and elastic search so one of the issues with spark is we sort of ran into some issues when we have when we're trying to do our in graham co-occurrences for very large data sets so basically just just trying to generate these in a timely fashion was was sort of troublesome for our twelve node cluster that we have it quantifying and what we found was it was pretty tough to reduce the search space of the entire data set so what I mean by that is you sort of have your your multi terabyte data set sitting on disk and you unless you sort of manually put your you know a certain slice of your data set in directory in another slice in another directory so that you avoid having to look at the entire the entire corpus it's it's somewhat hard to do one spark usually when you have an rgd over your full data set and you do a filter on that it's still doing a full table scan essentially over every record when you do a dot filter on your r DD unless you sort of tease apart your data set like I said in separate rd DS which can be a little a little cumbersome if you have a lot of different dimensions as we do and you know at least for us internally we have our infrastructure team in our data science team and you know sometimes you collide with them with trying to run your sport everyone wants to run their own flavor of their own spark job and there's some back-and-forth fighting of resources and and frustration around repackaging jars and typical development which also typical development challenges which also exist with Hadoop when you're trying to change your Hadoop job how you have to repackage your jar and ship it across the cluster so then we thought well you know what if we jam all of this data in Postgres right and so in Postgres but have sort of table partitioning for the different for the different dimensions but you know what we found was it was nice if you were just querying a sort of a small slice of your data set but when you try to do things that would sometimes you do want to do a full table scan and that was very painful for her for postgres the query performance would just which is Doug rotate which is degrade and you know trying to represent all of your business logic in the form of sequel queries is is pretty limiting right if you it's similar if you're going to use some other sort of like sequel sequel on Hadoop technology you know it's it's tough to represent all of your logic and sequel queries and then if you want to add custom logic and custom functions that's also a little cumbersome right it's kind of hard to bring the data close to the to the computation layer we've also tried elasticsearch but you know we we were essentially annoyed with trying to rebuild our indices as often as we need it to as the business needs dictated and you know it's still pretty tough to to build that custom logic elasticsearch offers a lot of nice things they you know different probabilistic style querying and whatnot but but it's still pretty tough to sort of introduce some custom logic that you want which we would always run into as as we would build our products so kind of like stepping back you know we sort of asked ourselves like what what do we really want right we want to be able to access our raw data right we don't want it to be pre aggregated or pre computed in some form you want access our raw data in memory and and be able to address these use cases like time time series cohort analysis and a bunch of other things that we do directly on the raw data in memory because that is the fastest way to access your data right if it's all in memory i mean i think what what a lot of you know what a lot of users one or developers one is just all of the data sitting in memory so that you can just just deal with it as quickly and as lots of me as possible so we and we want these operations to be on the fly sort of interactive and and and fast right we don't want to get into the habit of writing these bad jobs so we thought we we thought well you know we wanted in memory but we you know the one way to go about possibly doing that is just to pack it in a very compact binary form and just jam it into memory right however that doesn't really scale right it doesn't really fit in a single machine no matter how tightly you-you-you bit pack the data so what we ended up doing was taking our compacted bit pack data and spreading it out across multiple machines and communicating across those nodes with acha cluster and what you can essentially do is take an HTTP request that comes into one node and instead of that one note trying to trying to make a decision it just forms out that question to a set of servers that are communicating over akka cluster and hitting bit pack data sitting in memory right and then you get back your answers on the fly so I think if you've done a little bit of akka programming or you know or even just digging into how you know Hadoop works this is essentially sort of a you know an on-the-fly scatter gathered style type operation so here's an initial sort of architecture diagram so we have an HTTP handler that handles all of the incoming web requests for things like time series cohort analysis and in graham co-occurrences first for certain dimensions of this very large data set and so it goes to the RTC master it forms that query out to the workers which then send their partial answer back to the master to then be aggregated and sent back to the users so so each of the workers has a slice of this packed bit pack data in memory so digging it to some more of the components we have the pack filewriter a master node and the worker node so this pack vol writer is really the only offline job that that actually produces the bit packed representation of this very large raw annotated data set and it packs it in a very cussin and a custom binary protocol that we've come up with and i'll give you some examples of that in a minute and it packs it and organizes the data in such a way that it can be easily sent out and farmed out across the worker nodes in the system so the master like I said handles the incoming HTTP RT C requests and routes them to the to the workers to sort of orchestrate that scattered gather chain of events and it basically tells all of the workers hey you're responsible for this set of PAC files yours this guy's responsible for that set of PAC files and that's sort of all the workers see and the workers job is is to read that though set of PAC files from disk and jam it into memory in an efficient way excuse me the other thing that the master does is it it sort of has an adaptive caching layer that for very frequent frequently access permutations or requests that don't change over time it can cache those permutations I'm so that it doesn't have to recompute a recompute a particular part of a request the RTC worker through akka cluster discovers and registers itself with the master so this is pretty easily um horizontally scalable we can just fire up a new worker and it sort of discovers the master and attaches to the master and and asks for basically some work so to manage a portion of those pack files and then it loads its set of PAC files into memory and handles the request from the master so I'm not sure how many of you are familiar with with acha cluster but I mean it's if you've done acha programming before it's it's it's a very it's a very nice continuation or extension of just normal acha programming of just sending messages to actors except in this in this instance you're sending messages from two different jvms actors living in their own actor system in two different jvms and you know it handles it basically can notify you when a when a node falls out of the network falls out of cluster or joins it really sort of takes the pain away from that normal network style programming that either we implement ourselves or we end up going relying on zookeeper for that kind of stuff so let's talk about the compact data so there's a custom binary protocol for for the consumer comments and also their their user records each of the records if you're writing it like a custom network protocol you know the standard practice is to have a header which gives some information and then you've got a body and that's sort of the the pattern that we that we follow where we have a fixed header with a set of known feels and values so this is sort of what it looks like so you've got a 2-byte header this is an example of one record that we would store a 2-byte header eight bytes for a user ID eight bytes for a timestamp and then you'll see that we sort of break up the text so like a normal string that you would that you would allocate or that you would have liked in a JSON record or even in memory it gets a little tedious because you have to sort of go back to your C C++ style days where you're you know where you're keeping track of the number of characters and the number of bytes and then and then expanding so on and so forth because we sort of are trying to be efficient with the space we we try to find ways to sort of store things in an interval in a more compact representation so in the case of text we can we can use a 64-bit sip hash for that and then have a corresponding dictionary for that so that in memory you're just dealing with long use and then if you need to go back to the string you can then look it up in a corresponding dictionary for that and I don't have an example of that here but we can we also have some fields that instead of using a traditional buy it or short or I'm sorry interests as using like an integer or a long we can use a smaller dimension field for that to store different values so we can sort of bit pack a bunch of boolean values just in a short for example and be able to read that on the fly so this is all fine and dandy however what we what we ran into were and I'm sure like as many people run into this is the garbage collector getting in our way right so I'm sure all of us have experienced has experienced and out of memory error in the JVM so you know JVM style programming is nice in the sense that you you don't have to manage the memory yourself you just allocate all these objects and you go crazy doing filter map flatmap collect and all these things and you don't really have to worry about objects coming you know references to those objects but when you're really trying to squeeze the most get the most performance out of your system the garbage collector a lot of times can be your worst enemy and can get in your way and and can cause things like garbage collector pauses as the as it's pausing your system to try to try to look at all of them markings of all of your objects and saying which ones can be released and which ones can't be right so this is where we where we ended up moving our system to off heat memory so I think this is something that you're starting to hear more and more about at conferences and whatnot but you know the JVM really gives us this nice backdoor to like a C or C++ style means of allocating memory using unsafe so this is currently like a hidden reference in the current JD and I think in Java 7 and 8 you have to use reflection I think to get at it but I think in Java 9 they're actually going to put a proper public interface to get access to these underlying utilities but it allows you to do C or C++ style malik operations and be able to sort of you know request from the operating system a chunk of memory to to use for things to store things in the only downside is while it comes at a cost in the sense that you have to manage that memory yourself you're literally just getting a chunk of memory that the garbage collector is not keeping tabs of and but you're responsible for how you lay out your data in that off heat memory and making sure that you release portions of it if it makes sense and all of all of that that comes with you know our old days of Siesta + + style programming so this was the same diagram from before but the pack data the green boxes at the bottom have been replaced with off heap data so each of the workers will load their slice of the pack files but store them in an off heap memory and so they're they're managing that so working with off you data is a little different from like normal JVM style style programming you know you have a reference to this unsafe object which gives you sort of access to this black box of memory that you've allocated excuse me um so so for example here for a timestamp method you know normally we would just have a long that's part of a a case class for example right but in this case what we're doing is we have we don't have that because it's not materialized in the JVM like it normally would be so here you have a reference to your unsafe object and you're saying get long and when you have a reference to this you actually have a reference to the to the starting memory address of the block of memory that you've allocated so when you say here get long to plus 8 i'm just being explicit and I'm saying skip past the two byte header skip past the eight bytes of the user ID and then put me at the where the timestamp falls and that protocol that I showed earlier and extract the long value and if you if you see here it's a def right it's not a vowel because moving to the primary goal we we never want to materialize data that we don't have to materialize right we we really don't want to throw objects on heap for the JVM to have to manage if we don't have to so this time stamp we'll just sort of be sitting in off heap memory and will only be extracted when you would call this method so we really just going back to that we really don't want to materialize things that we don't have to and this becomes really important for things like strings because strings living on or you know it will take up more memory when they're represented on heap so just you know don't worry this is a bunch of code on the on the screen but but don't don't worry about getting too much into the into the details here but even things like off EEP like binary searching right most of us use java.util.arrays binary search and you have a materialized array that you're passing into it but that's really sort of off limits in this style of programming right so in this case the binary search method you have your passing you're on your unsafe reference to your off heat memory and essentially the bounds in off heat memory where you want to do your binary searching so the key thing really is just like the vowel term equals unsafe get long that's where you are actually pulling the value to compare so that's kind of like the main thing to sort of look at where you would normally just do an array reference look up here you're doing and off heap look up so okay so I mentioned the search space earlier the the search space is actually really important for us because for a lot of our queries we may only be interested in a particular facet like a particular client or a particular date range of data and making sure that we don't do full table scans where we don't have to is actually really important for us and so what we what we basically ended up doing was um creating these these facets these facet indices so these are custom facets that are represented as tree maps or you know they could just be arrays themselves but they're essentially just locations to offsets an off heap memory that pertain to a particular facet that we care about so it could be a particular time of day could be like a month or day something like that so if you're only querying for things that happened within one day you you can just come you can very quickly ignore all of the other data that's in the system and this is important because you know where as this system is not getting thousands of requests per second it's getting few requests but in a single request we're trying to really do millions of operations and in the sense of visiting millions of Records so if we can avoid visiting a billion and instead just just visit you know a thousand that would potentially match then that that's a huge win for for these types of queries so extending the architecture diagram again so each of the workers has their off you data that they manage but then they also have one or more facet indices that are essentially pointers into the off heap data so right so here's an example I kind of touched on this earlier but right most of our queries care about a particular time range so then how can we avoid searching over every single record when we only care about an hour or or a set of minutes for example something like that so what we can do is we can build them as these workers are loading their data and shoving it in offbeat memory they can build up these these indices so in this case it could be a tree map of a long to an array of long where you know the keys are so I should sit back a little bit so when you start doing things like this when you start using these normal data structures they do live on heap and so the the indices live on heap the data lives off heap so in this case the the keys are the time stamps the rounded down time stamps and the values are offsets to the off you records for that for that particular time so when you want to do operations like a you know a set of days you can do you can do log in style range operations on this tree map data structure to get back very quickly which offsets you need to look at so in this case you're you're preventing from just how you you don't have to search the entire all the arrays of Long's that are in this tree map you only pull back the ones that could potentially satisfy that facet of the query so once you have this array of long offsets that you want to visit right how do you how do you sort of traverse this array as quickly as possible and do the computation so this is where within each of the workers each of the workers are multi-threaded so the worker will find which array of long offsets could satisfy potentially satisfy the query and then it just fires off a bunch of additional sort of mini style worker threads to tear apart or to traverse this array so one thread might do a certain number of offsets another threat might do another another region and so they are so the workers themselves are answering questions in parallel but then with each of the workers there like many workers that are slicing and dicing the arrays and doing a mini style aggregation to be sent back to the to the master and we found that this helped out quite a bit as opposed just doing just plowing through the offsets one at a time so i'm not sure if me if you guys have follow kelly summers but she's she's she tweets a lot of great things about infrastructure and and you know she's really big and a like testing a whole bunch of different like open source technologies and giving her her comments on that so she tweeted something which which really caught my attention one day i hear about how immutable structures are awesome but here's the thing you want millions of operations a second you need to stop allocating memory so this was one of the key things that we learned that you know for our requests and to return back in a reasonable amount of time and to not piss off the garbage collector we had to really keep an eye on our on our allocation so that's really what RTC is about is is limiting the allocations to do to do satisfy a query so some of the lessons learned you know I've keep talking about this reduce allocations as much as possible avoid allocating you know I'm like 90 hundred gig JVMs that have many objects that are stuck in this tenured generation that's just really not good for the for the performance of your application and you so the ABM expensive data that you've calculated or that you've that you've answered can be kept around in lr you caches and that can that can improve subsequent query so that you don't keep repeating the same amount of work now if your data changes frequently that's kind of hard to do but in our case the way that the cadence of refreshing our data this this LRU cache mechanism helped out quite a bit so right oh so another thing that we that was so kind of adding on to the seller so most LRU cache is right the way they work is they're they're essentially bounded and then when a new element needs to come in you affect you you evict the the the element that wasn't used the most however there are cases where your LRU cache may be holding very expensive objects or very large objects and you may not be putting something into the cache you may not be putting something into the cache but the JVM is experiencing pressure and it and you would like for it to evict to that element out of your cash for you instead of waiting for the next request to push something out so if you one of the things that we learned is if you wrap your values in soft references and that sort of tells the JVM you know hey I would like for this thing to be around if it can be but if not feel free to just evict it for me and allocate the memory so this was something that was that helped us improve the stability of the system some other a couple of tricks just chunking very large requests into smaller requests that sort of goes into the workers having yet more mini workers to file through things and you know scholar collections are great but for certain for certain operations that we were doing and certain you know when we when we analyze the memory we found that just using 12 collections or even just dropping down to primitive arrays really really got us to that brought us to the next level so almost done so was it worth it you know building a custom solution um you know we we definitely think it was worth it you know we really tried to use these other you know open source technologies even other ones that I didn't mention here to sort of satisfy the use cases that we have for the types of products that we're building and you know what we found some of the core benefits is you know that the the the business guys might might come around with some or lead in the date our data scientists might have some idea for some really cool new algorithm that's that has some various you know performance challenges and the first thought is you know okay how do we make how do we make spark do this or how do we how can we think about the problem the way elasticsearch wit would would want us to think about it we because we own the system we can just go right in and and and come up with a and implementation for that query that is like squeezing out the most performance as possible so sort so we can adapt to the new to changing business requirements and we've addressed some of these things already in terms of further improvements but we we want to add more redundancy a lot of this just comes for free when you upgrade to we're using a bit older version of aqua cluster just because we have we have we have like spark and Anaka cluster in the same the same SBT built files so they sometimes they clash we might want a newer version then what then what spark would build in so we're working on teasing those things apart so we can add more stability by getting some of the improvements and in future versions of aqua cluster so in summary you know the whole system really boils down to you know acha cluster being there for the for the network communication using none aggregated off heap data so you know the key thing there is to remember that these records are in their Rost farm they can't be in any further broken down form it really is an individual record that you then do arbitrary ad-hoc style interactive querying over and just keeping an eye on the way that you write Scala code to try to be as efficient as possible was a win for us and for the for the for the product so thank you and of course at this any of this stuff is interesting chat with me afterwards we're we're also hiring Thanks thank you ran like we've got plenty of time for questions did you at all consider utilizing Avro in place of a custom binary format we did we tried um we looked at a few we looked at something from google called flatbuffers we looked at Avro he looked at things like message pack and things like that and I think the issue with at least with flatbuffers it was a couple a couple of us trying these different things but i thought when i looked at flatbuffers i think what was happening there was the the fields they allow you to specify these optional fields and so I think they end up end up allocating like a short I forget what it was a short or something else before each of the fields and we couldn't really find a way to just to stop that from happening so so what we found was we could be a little bit more efficient with our own custom format than by using one of the off-the-shelf libraries but I mean that is that is a piece that could definitely be replaced by some other open source library but we ended up you know we only have like two or three different schemas so it's not a huge maintenance nightmare but if we have a bunch of them will want to revisit using something that allows you to just do some sort of high-level scheme and then generate that efficient code hey Ryan is a great talk I got a question on the actually I got two questions for one is on the other side is that you send the message you send is it a large or small so then that's my first question yeah sure right so we have run into some issues where that where the results can be kind of large but we ended up turning on the protobuf serialization and compressing it a little bit and and basically enlarging that frame size that tuning parameter that octave lets you tune and we haven't had any issues of going over it but yeah it was a slight annoyance in the beginning where especially in some of those in Graham co-occurrences those can be large data sets coming back so yeah we're in the same problems because then to need a buffer size and it goes we increase the memories and yeah my second question is you're using unsafe for the Alpha heap memories notice that sparks tungsten projects and also into the exactly the same thing so laying out of their own memories are using on saves so right right with that approach in your thinking spark can solve your problem partly uh you know yeah I mean so so I mean I should mention like I mean we're still like fans of spark right like we're not we still use spark our data scientists still love using it for for truly ad-hoc style whitebark jobs and ml live and things like that yeah it was kind of funny like we implemented the the off heap change to RTC and then yeah then laterz one of my friends sent me the article about you know but what is it tungsten tungsten Roger tungsten right and so yeah I mean you know I support like all of these open source projects evolving and I really would love for some generic solution to be there to satisfy all business needs but you know like as you saw in the previous talk like the flank guys you know they think that they can differentiate in some way and everyone wants to do a different spin on these things so what we've built this custom tailored to us it would be really hard to open source it I think some of the ideas are open source of all if you can open source ideas i don't know like approaches like these are things that you guys can use in your own products and things like that but yeah i mean it would be great if there was some silver bullet that would that would implement everything you know with all the different use cases that we have knows yep yeah so i have two questions like everyone else right so the first one is have you guys created because what you're describing there is a great approach but for me it looks like a nightmare for scalability I mean if your company grows like a million X or a hundred text right like maintaining that that are you what are the resources no resources but what are the things that you guys are doing in your code base to actually think about that is scalability right because of hip memory of hip memory sounds really good really cool right but we all know why we are not writing in C++ code anymore right yeah so that's my first question yeah sure yes I can answer that so actually the the bit of unsafe code that's going on in our system is really probably about maybe a total of like 100-150 line something like that so what we've done is we've put these views on top of this thing right so so if someone knew where to come store to come and develop on top of the system add new functionality they they don't have to know about unsafe or off you memory or any of that stuff right because they're just using to them they're just using a plane Scala object that has you know methods on them right so like time step method or things like that so if your model if you're adding new if you're modifying the off heat the way you represent data off heap than sure you have to you have to manage that and you know we have a pretty good you know is we're only like four or five infrastructure guys we all know that code based pretty well and you know and keep in mind it's it's custom to us so we're not we're not you know so in terms of like a scalability there's there's the scalability of the project itself from a code perspective which we haven't had any trouble adding new features or anything like that because off people's holding us back or anything like that and also just from a performance perspective it's a horizontally scale scalable system so you just fire up new nodes and then you know the data gets spread out more evenly and from from a performance standpoint that that works I think I think that's what you were asking right well i was asking more like you're saying you're four guys and I've been there I know what that means but I am also a team right now we are higher than 50 engineers and I can tell you it's a pain to maintain a code page 550 in years when you have one guy doing memory location all over the place and I'll guy trying to delegate but that's not the point yeah that's a scary as you make it sound I think you know pull requests and github and everything is of course of course I mean it's have to be mature developer of course of course yeah it's all about the game station right um I'm a second question is did yd did you guys consider something different from a scala I know this is a scala conference sorry guys but what they mean is like well you're saying like we have all these problems with the AVM and whatever and and these strings are pain like why why use the scala why not use like know C++ or Java or something that allows you to have more control because for example Java Java 8 allowed allows you to do memory location and the location manually right you can actually do that you can do a fib memory location using Java facilities right right right in Java 8 yeah so why keep using a scholar yes so that that's a good question i think we are we definitely definitely across our minds just to go straight into you know c c++ implementation of it um but you know RTC is is one bit of infrastructure of our of a much larger stack right and so what we found was if we could like tuck away this off heap low-level memory management stuff into like a small module but then the rest of the code base actually benefits a lot by just being written in scala because of the high level things they're not so not all of our GC has to be done this way there are there are many parts of it where we actually really enjoy having scala collections especially once the hard work is back going to the master so in the worker side you want to be as efficient as possible right because those are the guys that are that are doing all the all the hard work but then once the responses come back to the master the way that you aggregate and the way that you do further computation on top of that having Scala and it's in its collections actually is really great because you don't have to be you don't have to be as performant there because the hard work was traversing those millions or billions of records so if you can minimize that then it gives you more room to have fun with the scholar collection so if we had to write all of that and see your C++ it would be it would be annoying and it would be sort of this piece that didn't integrate very nicely with the rest of our system but we were a java-based company in the beginning and it was sort of through spark that we transitioned and sort of fell in love with with Scala so um your use of the off memory of heap stuff is it's only read only right it's read-only correct that's right yes so RTC itself is this it's read-only and it's sort of this immutable thing right you can't there's no notion of like a session where you can update create your own table kind of thing like that it really is just solving our specific needs it's read-only and so you're actually making a copy of bits of it as you need them correct your materializing them as you said yes right so in this so in the cases where where we have to do things with strings like use methods on the string object we have to materialize that we have to bring it from off from that bike representation and instantiate a scholar string but we try to you should be able to put it you should be able to put a mutable string on directly on top of the data right oh yeah yeah yeah yeah you definitely can't just like we did the binary search completely off heap you can start building even more data structures or more views directly on off heap tube to to minimize that but you know it kind of goes back to the question that the guy asked earlier where you know it's it's nice to have some of the some of the niceties of the JVM and Scala and if you can if you can you know if you can handle a little bit of performance degradation and those little areas then it's kind of nice to just write some simple code to do it but yes I mean we we we do have other things where we have like we have bit sets that are just off he bit sets they're not like the java.util bit set or things like that so where we have to do where we have to write off heap data structure we do use that we have time for one more question yep hi petition that you guys do are there any literature surrounding that whole half it seems very interesting so I was wondering if you guys have any numbers are you know performance gain with oh you mean yes like doing certain operations in this new system versus the verse of the old one no what will the specifically with you have 15 so specifically with off you yeah um no we so so the one thing to keep in mind with off heap is that it's not that things off Eve doesn't necessarily give you faster performance it just lets you store sort of a lot of data in a JVM kind of thing right so that so there is so when you go when you go from like accessing an array that's on heap doing a rate in to see lookups is actually faster than actually going off heap and doing that so I so that the like if I wrote the binary search method just using the comparing it to the java.util one that one should actually be faster than the off heap one it's just that the off heap lets you have a ton of data loaded in memory that is at your disposal whereas if you try to like shove all of this data and like a like a scholar hash map on the JVM you would quickly run into all sorts of you know hashmat resizing and the garbage collector getting in your way that's that's really what drove us to off heap was because we we really wanted to use the memory as efficiently as possible but I I'd have to do more research but I think going off heap is just a slight performance hit but but on heap i believe is generally known to be faster but if anyone knows better than me correct me but i believe that on heap is still faster than off you because you have to cross that boundary out of the JVM or pseudo out of the JVM i mean it's still if you look at the process it's still the JVM process that has all of that memory allocated it's just that the garbage collector doesn't see ya cool thank you ran great thanks guys you