data.bythebay.io: Gene Pang, Open Source Memory Speed Virtual Distributed Storage System

data.bythebay.io: Gene Pang, Open Source Memory Speed Virtual Distributed Storage System

Recording: data.bythebay.io: Gene Pang, Open Source Memory Speed Virtual Distributed Storage System

want to thank you guys for being here today and and listening about alexio which uh used to be called ton and essentially it is an open source memory speed virtual distributed Source system um and I'm you know I'm Jean Pang I'm a software engineer at alexio so before I get started about talking about alexio the the the the software first want to briefly describe the the company it was founded by some of the creators and and top committers of the open source Luxio project and it was uh funded last year by andreason hitz and you know this is our website you can check it out later and here's a little bit more background about myself I'm Jean Pang I am a software engineer at alio I'm one of the core maintainers and contributors to the open source project of alio and before I joined last year um I was finishing up my PhD at at UC Berkeley from the amp lab where Luxio came out of and um before going to Berkeley I was I was at Google for several years so here's sort of a brief outline of what I'll be discussing today I'll first talk sort of like a brief overview of alexio of the system then I'll talk about some of the motivations behind alexio and then I'll go into um the use cases for alexio so the first question you might be asking yourself is what is alexio what is this thing that I'm I keep talking about well alexio how we describe it is open source memory speed virtual distributed Source system and I know that's a lot of words up here and um it's a mouthful but I'll sort of break it down and sort of describe different components of of the system so yeah so what does this mean so it's open source um it's one of the um one of the more F it's it's a f it's one of the fastest growing projects out there in the in the sort of the Big Data IND uh ecosystem it's also a memory speed what this means is that alexio really emphasizes keeping data in memory um and it's the architecture really tries to emphasize and aggressively uses memory it's virtual essentially uh Luxio I'll talk about a little bit later um has a unified name space so you can it's essentially it Shacks away different storage systems for applications and developers and it makes it a lot easier to deal with different storage systems and also it's a distributed system so it's been designed to sort of scale out um like many of the other distributed systems out there so first you know alio is open source uh I'll talk a little bit about how it sort of started uh it started uh in 2012 in the summer of 2012 it started at at UC Berkeley in the amp lab and the the creator of of of alexio at the time was a grad student there and he and this was part of his project so he created at the time was called tachon he created in 2012 this is the same amp lab that created uh both mesos and Spark that's uh heavily used today uh yeah and uh it was created in 2012 it was open sourced in uh April 2013 the following year uh and it's it's had the Pachi 2.0 license recently it was renamed from tachon to alexio and the latest release that we have so far is uh was released in February uh which is 1.0.1 so this is a it's it's an open source community of alexio and it's actually been really exciting to see the growth of the of the contributors to alexio and so here's um so I think currently it's close to 250 uh contributors uh to the project and also just in the past year it's grown I think over three times from from like a year ago and so here's sort of like a like a bubble of a lot of the top contributors I think it's the top 100 or top top 100 or so of the top contributors you know relative to the how much they've contributed to the to to the code base so it's really exciting to see how how fast it's growing and here's sort of the graph view time series of that um this is sort of the number of contributors that have contributed to the project over the over the of the existence of of the project and so Luxio is really been around for about three years in the open source community and so over those three years we've it's grown to basically over 200 contributors and so on this graph the luio is like the top blue the blue line and so uh it's just really cool to see how fast it's growing just in the first three years of its of of existence and yeah so I I joined about a year ago and uh yeah it's it was around like 60 or 70 people at that time and now it's like over 200 so it's really cool to see that and there also a lot of different uh contributors from different companies as well as different users from different companies and here's sort of a a smaller sample of that um and it's just really cool to see how so alexio is very flexible as you'll see later and there's a lot of different use cases that you can apply alexio with and so it's really cool to see how different companies are using alexio and contributing to alexio um in in their in their work so next I'll talk about how alexio is memory speed and what this essentially means is that the architecture heavily focuses on keeping data in memory um and aggressively uses memory for storage of data uh so you might be asking yourself you know why should we use memory for storage like why do we need to do that um is that even like feasible today well I would say it is and sort of two reasons for that here's the first reason first there is a performance Trend uh in memory uh memory is has it's it's known that memory is always faster than you know spinning discs and even ssds and actually uh the yeah the you know memory has lower latency has higher throughput than than than traditional discs and uh the performance so there's a large performance Gap but there's also the the the performance Gap is actually increasing as time goes by and so um in order to have interactive applications to have um just quick response times memory locality for your data is very important so here's a graph of sort of um of ram performance and dis performance and this is this is looking at throughput and on the left is the RAM and as you see the the bandwidth is is is increasing uh exponentially whereas the performance for dis is not increasing as quickly so oh yes is that all spinning discs or does that also so this is just spinning discs okay yes yeah but but even if you consider ssds um The Gap is still pretty pretty large yeah and so yeah so this is one of the main reasons this is one of the one of the mo motivating factors for trying to use memory for for data storage the other Trend that's been that's been uh happening is that the price of of of memory is actually dropping and Dam is becoming more and more inexpensive and here's a graph that sort of shows the cost per gigabyte of dam and as you can see it's steadily dropping and it's sort of like the Moors law of dam maybe the opposite of mors law but essentially every 18 months or so the cost of dram is um is you know is is is is uh becoming half and so uh if you take a look it it's becoming far more affordable to purchase a lot more RAM for machines and a lot of companies are actually doing this and so maybe the early 2000s only select few companies will actually do it and then slowly some of the bigger companies started doing it and now a lot of you know even mediumsized companies are buying a lot more memory for their machines for their clusters and for um for their centers and so there is a lot more memory available for compute and for storage and so alexio is trying to take advantage of that and try to um really help manage data in memory so you might be asking yourself like why yeah what if the memory is just not big enough like what if even with all these Trends you still can't fit all your data in memory like is alexio sort of useless then well no it's not alexio actually manages other forms of of storage Medium as well and so today alexio can manage three different types of tiers we call them storage tiers uh one is the memory tier next is the SSD tier and then lastly it's the the spinning drives or hard drive tier and so alio can manage all this all these local storage medium um on on the on the on the nodes and this is actually really useful because the the characteristics of the tiers are are are different and they're trade-offs right the the highest tiers like the memory tier is usually the fastest and the highest bandwidth lowest latency however it's probably the the least uh the the the least has the least capacity in all the tiers as you go further down the tiers you get into sort of tiers that have a lot more capacity usually but has a lot um it's not as performant as the highest tier and so alexio helps you manage all the this uh data in in these different tiers although it does try to emphasize the memory tier the most and these storage tiers are actually configurable you can actually configure Luxio to use just memory maybe or maybe memory and hard drives or maybe not even use memory and just go straight to ssds and so there's different ways you can sort of mix and match the different storage tiers for Luxio and uh Luxio sort of helps you do that so with these storage TI um Luxio also have sort of pluggable tier management policies uh to move data around between the different tiers and this essentially um we have a plug plugable policies to move data sort of down down different tiers or move them up the tiers and this is actually really useful because you know at a high level you really want the data that you want uh the data that you want you want it to be in the in the top tiers you want it to be in the memory tiers it's the fastest and it's the uh quickest to get to whereas the data that you probably don't need soon or don't need uh you want it in the lower tiers and so with this intuition we have these sort of plugable policies that you can use in alexio to help you manage this and so this this is actually configurable and also you can extend it you can create your own policies and plug it into the system and use your own policies if if if the ones that come with the system are not appropriate and next alexio is a virtual distributed storage system and so this is also pretty loaded um but you know first all yeah so I mean it is a it's an actual system it's a distri it's a distributed storage system but it's also it's a virtual system a virtual storage system as well and so I'll talk a little bit about that now if you look at the ecosystem today there are many different Frameworks out there like you know spark map produce Flink and things like that uh there are also many different storage systems out there uh like S3 Swift hdfs um even you know TR traditional filers and NFS and things like that so um there's a lot of different places to store data and there's a lot of different ways you can access that a lot of different ways you can uh do computation on that data and so this can be problematic because trying to interface with all of them uh well is can can run into problems and so what are some of these problems well it can be very costly can be very costly to sort of integrate everything and even costly to um ETL data in and out of different storage systems um it can also be expensive in terms of um getting data where you want it to the computation where you want it uh where you want to do the computation on and so that there's maybe a lot of duplication or movement that is required to to manage all of this also you might run into Data silos where you know you can't really access the data you know in other places or in different ways and sort of locks you in into in a specific um area and it can just be hard to manage all the all this different data and all in different ways you're accessing data and so all these different situations um can really lead to longer time to extract value from your data managing all these things are is not what you want to do you want to actually extract value from it and so um there's a lot of different problems with that so alexio tries to solve that tries to unify access to your data and so it essentially is a layer it's a system that sits between computation Frameworks and applications and different storage systems that you may have and so you know you know back uh in a few slides ago we saw there were a bunch of computation Frameworks and applications and there are a bunch of storage systems and so alexio actually sits in between there and helps you manage that and also provides a lot of performance performance characteristics as well so by being this uh by being the system in between computation Frameworks and and the storage systems it really can enable new workloads across different storage systems instead of having to try to interface your data um try to interface on on specific different storage systems for your data alexio can abstract that away for you and you just have to Applications just have to access um through alexio and and and um using the unified Nam space can access the data in different storage systems very quickly this also helps you to be very flexible so you can easily add a new a new computation frameworker application to your system and since it just has has to talk to alexio it doesn't have to try to integrate with all the different store systems out there that you may have and finally this is something that we've sort of been seeing recently is that um companies have been trying to sometimes have been decoupling storage and computation and one of the reasons for that is is because the resources for storage and the resources for computation are actually very different storage usually needs a lot of storage media a lot of dis or ssds and not necessarily a lot of CPU or Ram whereas computation resources usually require a lot of CPU a lot of RAM and so um it's really it can be really good to be able to scale them differently and allocate them differently uh independently and so uh instead of having to scale out all the resources at once whenever you're live uh hitting some bottleneck you can scale out just the resources that you need and so with alio this really enables any application to access any data from any storage um at memory speed so next I'll talk about how you use your Luxio and how it fits into your system and so going back to the picture it is essentially A system that sits between um computation and applications and storage systems and so here are the basic use cases that I'll talk about today first alexio can accelerate access to remote storage it can also um help you share data between different applications at memory speed and finally it can help you transparently manage data across different storage systems so the first first example I'll talk about is how alio can accelerate access to different storage uh to remote storage and so this um this actually happens a lot more than you think there's a lot of different uh new storage systems out there that are essentially inherently remote meaning uh like something like S3 or like a Google Cloud store or Aur blob store these are all going to be remote from typically remote from your data center or from where your computation is being done so already the storage is remote from uh computation also like I mentioned earlier um companies like the idea of being able to scale different resources independently and so that's another reason why you would have uh independent uh storage and comp compute clusters and so there are a lot of different scenarios where you would actually have remote storage and alexio can really help with that so I'll go through a little example um say you have you know a spark job that wants to access uh data in S3 so in order to do that it would have to read from S3 would has to go has to go over the wide wide area network and then has to you know maybe write it out to S3 and so it can actually be very slow and um and and it has to go through the network uh has to go through the network and it would it would have a you know High latency and and network throughput for all that data access so with alexio it looks a little bit different so alexio doesn't actually get rid of the network access it um it still if it needs to access the data from S3 it still needs to talk to the wide area network so it still would have to be slow however once it's an alexio as long as it's it's close to the computation the spark jobs only need to access through Al to alexio and so that's much faster and it can even be in memory like I mentioned before so spark can access all the data from its local essentially its local cluster maybe local machine from its local memory and which is much much faster than trying to access data from S3 each time and so keeping the data in alio can really help accelerate the access to data next I'll talk about how alexu can help applications share data at memory speed so this um is common when essentially have maybe a pipeline or iterative um computation on data and so essentially what that means is sort of the output of one task or computation is the input of the next computation or or job and so when this happens typically uh applications share that data by writing it out to some storage system and having the next job or or computation read it from that storage system and then that's how they would share data and so this is what it might look like say we have you know this example of you know say a three stage pipeline so we have a spark job write some data that the map reduce job would then need and then and then finally the spark job would need that that that uh last uh input and so um in order to do this you know these these applications would have to you know read and write the data um from a storage system and so in this case it's S3 but it could be any storage system so it it would um it could be slow it could go over the network it it would definitely hit dis iio and so there's a lot of different um sources of of slowness and latency uh involved in this and so all the network and disio can slow down this sharing of the data now uh with alexio we can actually help uh share this data at memory speed so you know after so so I say the first job has to read the data once it's in alexio they can all share the data at memory speed uh within alexio and then sort of write out that last um result uh back to the storage system and so by using alexio these jobs these different jobs can share that data through memory instead of having to hit the network or hitting dis IO um all the time the last situation I'll talk about is sort of how alexu can help you manage different uh data and different storage systems that you may have so in the simple world uh it's very easy if you have one application in one storage system it's very easy to sort of integrate them to and sort of work together um but uh that's not always possible it's you might always want to be trying out new storage systems for maybe maybe they're more cost- effective and so maybe in this case you heard that like you know people are using S3 to store a lot of data because it happens to be you know inexpensive so you want to try that out and you realize that it is a lot more it's it is inexpensive to store S3 data and so you have to then start integrating with S3 and then finally you want to try out the Google version of it so that you try Google GCS you have to integrate with that and then say you have a new application that needs to access this data you then would have to integrate with all your existing storage systems and finally the next application that you need have to integrate as well and so this actually gets very complex it's very inflexible you have to do sort of manual work in order to incorporate new storage systems or computation and so this can actually uh be a headache just to manage so with alexio alexio can also help uh in the simple World it doesn't actually look very different from uh the previous World applications would interact with Luxio and then then alxia would help you interact with the storage system however as you add new storage systems and applications they only have to integrate with alexio and they don't have to sort of do this multiplexing of all the different combinations um of of integration and so by by Luxio being in between computation and storage it can really help you manage all this data in different storage systems along with the different computation Frameworks you may have it's very flexible you can easily add new storage systems you can easily add new uh Frameworks um and applications don't really have to change anything they just just had to point to a different um directory essentially that gets mounted uh a new store system can get mounted into the into the namespace and applications can easily just point to that without having to change any code so it's very uh very very convenient so now now I'll talk about how Luxio is being used out there uh by you know uh real companies and how um it's been helping them so first I'll talk about BYU um they uh they've actually used this for a long time and uh they um they've been a pretty early adopter of the technology yeah so for BYU um it they use spark SQL uh for their computation framework and they use their BYU file system essentially I think it's based off of hdfs so it's maybe a variant of hdfs but they have a um by file system and they use alexio configured to use both memory and hard drives for their two two storage tiers and they've deployed it on over 200 uh 200 nodes and it matches over two pedabytes of space and so um they actually recently published a white paper I think you can check it out on our website um and they essentially have with alexio they got 30 times performance uh performance gains using alexio in their current in their in their um environment and so um this is actually on top of just switching to Sparks so I think before they were using maap ruce or maybe Hive and when they switched to spark I think they got maybe like four to five x improvements and then on top of that after they uh uh added alexio to their system they got 30 more times uh performance improvements so this really has helped them out and you can check out our our website to to see their their stuff study uh next I'll talk about Barclays uh their big Bank in the UK and um they they use spark uh for the they actually use spark rdds and Spark data frames for their uh I think machine learning type of workloads uh they configured alexio to just use the memory tier so alexio only handles the memory for their system and doesn't use any of the other storage media and in their situation they actually could get um in their situation they would essentially have to do this expensive like ETL type of process uh to get it into spark and so every time they had to restart a context or start a new context they had to do that expensive operation again and that took like 30 minutes or or 30 minutes to an hour each time to do it so um essentially by using Luxio they only had to do that sort of once and after that they could keep it sort of in a Luxio space and not have to not have to continue to um redo the expensive work to get the data uh to to the computation and so they actually wrote a blog post uh maybe a few months ago uh sort of talking about how it really helped them and how they deployed it and how they what their architecture looks like and you know how it really um sped up their their workflow um and the last last one I'll talk about is this company called chunar they're um a company in China that uh I think they're sort of like a Expedia like a travel travel type of website um and so their architecture is actually very um very involved they use a lot of different Frameworks they use you know spark streaming they use uh Flink streaming they use and those are for the streaming use cases they also use spark and Flink for their you know analytical workloads and olap type of queries and so they have all these different types of Frameworks they actually have different um storage systems as as well they have have many different hdfs clusters some are local some are remote and they sort of need to talk to all of them and so they put alexio in that system to really help them help them with that they um they alexio helps them manage the different uh Al uh the different storage systems different hdfs clusters it helps them share that data between different Frameworks like spark and and Flink both streaming and analytical type of workloads and so they to put it in over 200 200 nodes and what they told us is that um alexio enabled so some jobs just would never finish in their situation um I don't know the details why it would never finish but it would just never finish and alexio actually Ena them enabled those jobs to be able to finish and so it enabled impossible jobs to finish uh with alexio and also uh in many cases they got 300X 300 times performance Improvement using alexio and so that's a huge number um uh yeah so um they're actually I think in the process of writing a uh writing a study or a blog post about this and so um I'm actually pretty excited to see uh what they talk about in in that post so yeah so finally um this so that was Al Luxio uh Luxio is essentially in know open source memory speed virtual distributed Source system and essentially it is uh a storage system between computation and storage and it really can provide uh performance performance benefits Can it can provide some cost savings as well and also a manageability uh of your ecosystem and here's you can find some more information we have the alexio project website has a lot of documentation uh we have the the company website here um all of our code is in GitHub uh uh the open source code code is in GitHub and we have a Meetup uh Meetup Group here in the Bay Area and you can always contact me at Gan alo.com thank you very [Applause] much yes um so in this use case we likey to see that various applications can connect to various St right so when we want to use alio are we going to build an En new cluster once again or can we leverage existing hard clusters of orust of say 200 no and so on that's a good question so the question was um do you have to provision a new cluster for Luxio or can you reuse a cluster um that you already have and so it really depends on the the the environment and work load so it's pretty dependent on that however in order to get the the best benefits so for the greatest benefits we recommend actually putting um alexio with the computation side of things so if you if you're using map produce or spark you would want to collocate axio what we call workers with those workers as well and so that will actually give you the most benefit it'll give you direct access to all your data and memory um and it's it's less important to collocate with storage so if you wanted to if you wanted to reuse computation I guess clusters that you may have you we actually do recommend using those the same cluster the computation cluster for alexio nodes and looking at how it works in memory is it like going to be a resurrection of M because now everything can happen in memory and nothing is going to be spit on this so so the question was will this resurrect mapu um so I don't know but uh uh I would say in my mind I think we could enable uh different Frameworks to effectively use memory so before yeah so like say in the map ruce case um it wasn't Des it wasn't designed for sort of the memory use case and so spark sort of took took care of that but uh with alexio it sort of it matters less which framework you use I guess and if you if you you like if you know uh an organization chooses to chooses a framework um the data locality can sort of be provided by alexio and not have to be built into the computation frame itself and so I think it's sort of alexio provides the flexibility to choose the framework that you that best fits your needs thanks yes question number one uh your the meaning of unified name space are you thinking that just same mounting uh way way to mount different storage this pres as a storage um so how do you mean what okay okay so I'll describe it a little bit more sure so for UniFi Nam space what we mean is that um you you may have many different storage systems out there so say like hdfs or or Nest or like S3 or something like that and so what we would do what what you would do is you would essentially Mount those different storage systems into alexio namespace and so you can do like Mount htfs or Mount S3 or Mount SL whatever and so using the same client same interface you can access the data um in the same Nam space yes even in the spot and case you are like using memory SP lot so put it in the same are competing us of how did you address that yeah so that's a good question the question was um how so spark and Alexia both want to use memory how do you um basically how do you allocate the memory um that's again that's a very specific to different workloads um but what we what we have seen is that um so typically spark uses a lot of memory because you call essentially like persist or cash on rdds um essentially that if you use alio with spark you don't need to do that anymore you can save it into Luxio instead of calling cash explicitly or or calling persist SP for example a lot of so were many memory based on what me so the data collate closer to communication especially even for machine more compute same so you're already dealing with memory and now you are introduced another later nothing probably don't need to use because might like Lo and you had compute you don't really save it right way back to the and assistant store no matter it's memory place or not so then what's the value that I can get yeah so what I I mean I was saying before um that's the reason it it it manages it for you is because it's you're calling essentially you essentially persisting the rdds in memory in spark memory in the computation memory and so um how I would address that is essentially saying um by doing that it's hard to it's hard to reuse that memory for any anything else um so if if the if the data was actually sort of outside of the computation framework say like say say it was in alexio instead then that spark job could use it but another spark context is use as well the same data and so I mean it clearly depends on different use cases but if there are many different jobs sort of accessing the same data or similar data it makes a lot of sense to store that data sort of once uh in memory and not having to sort of duplicate it in different Sparks memory need toble duplica I mean there there's some duplication obviously but um there are ways um that you don't have to always duplicate that data yes yes in the back um and this the can you use it for writing as well so one one thing that we have it's a source of some difficulty is one thing we do in our spark shops is we run thousands and thousands and thousands of large uh Ingram queries over a Lucina index uh we've been doing that against elastic search for now and what you end up with is a very large expensive elastic search cluster that's maxed out and you end up with 10 15 second latencies uh per document uh on the spark jop both of these things are really you know unacceptable uh so we've been kind of looking for way to actually have a raw Luc cluster index uh per compute Noe in the EMR cluster um obviously just putting it on the instance door uh is an obvious and performant option but the deployment of something like that is actually quite a headache not least because when it comes time to update it now you've got to figure out who owns it and how to make sure that it gets performant copied back to some place and we've been thinking about putting it in hdfs but it's well known that hdfs and line are not actually such great uh um you know companions this seems like something would be very promising if we provisioned you know far more random we actually need let's say used only memory optimized instances and figured that you know 30 gigs of RAM per per EMR node was just for alexio to put Lucine index in just you know RI that off to that is that sort of thing that would actually work and then we could actually you know write to the index in memory in alexio and then considered to be updated and write it to non stor later um that's interesting so we haven't had any use cases or we haven't heard of any use cases using alexio and Lucine um so I'm not sure if it'll work but um I did see someone ago where he was talking oh really well then maybe it does work I it's it's one little you know thing on the entire internet and uh you know of course is the project evolves you it's one of those things where there's probably a lot of investment in my time and it's not going to you know it be good to have other people like have sort yeah so I personally um have not worked with alexio or takon and Lucine or I also don't know if my colleagues have either so I don't know for sure but um I mean maybe we can talk about it a little later cool has work well with Yar or has been Yar so we do have yarn and Mesa support um I think it is easier to to provide data locality if it's outside of yarn or mesos um but I don't I I don't know for sure yeah reason why because there many propy between now SP yeah so I think how some people have done it is they've um they didn't give yarn each they didn't give all the resources of a machine to yarn they put some of it uh to like I guess alexio outside of yarn and then gave the rest of it to yarn and so that was one way they got around with uh to some of the some of those issues but um but Alexia so we do have some support for yarn I think I think it's actually um it's initial support for yarn and msos so I I wouldn't say it's like you know the most uh um either performant or hardened integration but there is some there is a there is a beginning of Integrations of your anmos okay well thanks and let's give a round of applause for thank you