data.bythebay.io: Alyssa Morrow and Eric Tu, Distributed Visualization for Genomic Analysis
troduction all right just a quick refresh my name is Alysa Morrow this is Eric 2 and we're from the amp lab right over on the bay at UC Berkeley and today we're going to be talking about distributed visualization for genomic analysis okay so let's talk a little bit about a background right now so what do we mean to analyze a genome what is genomic analysis and what is our goal in genomic analysis well we can think of genomes as the source code for Life the human genome in particular is 3.2 billion base pairs program of a t G's and C's and these are split across 46 files or chromosomes so we can actually think of these 46 files as 23 files but but each one is duplicated once so across a human species in particular genomes are 99.9% similar and what this allows us to do is it allows us to take all these similarities and kind of squash them together and find an average genome and this is what's referred to as the reference genome so if we're 99.9% similar what about that other 0.1% difference is it important well it actually is extremely important as this .1% variance gives rise to a very diverse set of traits and diseases across a population so really at the end of our day at the end of the day our goal here is to draw conclusions from these this small amount of variants so kind of how do we draw conclusions from variants well what we'd like to do is look at a population and say okay at a particular Point what's the Frequency at which we see this variant what are the individual genotypes from the individual samples that Express this variant do we want want to look around the region of the variant to see if there's any patterns in the surrounding areas of the individuals who have this variant so this kind of workflow I've just discussed is a very great case for genomic visualization is we'd like to explore the variance in the regions around the variance and the individuals that might um might have mutations at that point okay so let's take a step back and say how do we actually get this genomic data in the first place this might be a review for some of you but we'll just go through it quickly so we get our genomic data from what's called a genome sequencer and we can actually model this problem as a simple sentence it was the best of times it was the worst of times so what we do is we take our DNA or in this example our sentence and input it into a genome sequencer and what we get as output is just a bunch of smaller subsequences of our original input and we can see they're kind of overlapping in no particular order so what we're going to have to do next is align these sequences to get some idea of what our genome actually looks like so if the genome sequencer could take our DNA and as input and output a perfect string in the correct order none of this would be a problem and we could just skip these steps however because we have these subsequences we have to align them and have some probability for at every given location what base do we see so the accuracy in which we can find a base at a given ation is kind of reminiscent of the coverage so the amount of replication at each sequence we see that's kind of referred to as coverage so higher coverage provides higher accuracy and today for only about $1,000 we can sequence a 30X coverage copy of your genome okay so we have our genomic data we've aligned our genomic data what's the next step the next step is to actually find the variance so we've taken those subsequence quences and we have our resulting alignment that we've aligned to that reference we talked about earlier so now what we want to do is find where in this sentence do we find any variants and what we see here from this example is that over on the worst it looks like we have a variant from an O to a u and because it happens four times with very high probability that is actually a variant okay so we have our genomic data we've processed our genomic data so at the end of the day what does this leave us with well back in the early 2000s we had the Human Genome Project and this actually left us with only 10 gigabytes of data this is very EAS today this can be very easily fit on a single machine but if we fast forward to 2014 projects like tcga are producing three pedabytes of raw genomic data and if we extrapolate out further these projects are producing more and more raw data and this is in the amount of samples and in the coverage that are making these um making these data sets larger so if we have all this data a question we can ask is how do we actually visualize it three pedabytes I don't know about you but doesn't fit on my individual laptop so this is the case for distributed visualization you can do these visualizations in a distributed environment to lower the cost by running on commodity machines while scaling up to these larger data sets so there are current genome browsers that allow you to visualize genomic data but the main problem with these genome browsers is that they don't scale so let's first talk a little bit about what current genome browsers look like and then we'll talk a little bit about what the ideal browser might look like So currently today's landscape in a genome browser we can view a few chromosome samples at a time if we're viewing variant data this can be about 60 GB um so it will easily fit on an individual laptop if you'd like to view a single chromosome furthermore they can explore and zoom out to about 0.125% of the entire genome what this means is if you have a bunch of chromosomes really all you can view is a very small slice of your data set and this kind of inhibits our ability to explore around the genome lastly they only provide single node scalability so if we're trying to view data from the tcga we have three pedabytes of data and we can't fit this on a single laptop or a single computer we run into a problem so now let's talk about the ideal browser what does this look like well ideally we could view compare and explore thousands of not only chromosome but whole genome samples and for this if we just take the variant data not even the raw genomic data for about 2 to 3,000 samples this can be about 1 to two terabytes of data already secondly we'd like to explore in Zoom to whole chromosome resolution what we'd like to do here is compute U summary statistics of larger regions of the genome that give us an idea of what we're looking at if we'd like to explore from a larger resolution and lastly we'd like multi multi-node scalability like I mentioned earlier we would like to scale horizontally to larger data sets but also run on commodity machines to decrease the price so in light of these shortcomings and the goals that Alyssa just pointed out uh we set out to make a genome browser that scales and this is our work um it's called mango so the first thing you need to know about it is it scales so uh you can run on a cluster you can store as much data as you want if you have a 100 uh gigabyte uh data set or if you have you know like a data set of a few terabytes you can store that as well you can operate on it uh obviously if you bring data into a distributed environment you have additional computing power so we can compute over much larger regions and we can return that uh information to you much faster and once we have this additional compute and storage power a question we need to ask is well what can we do with that well uh we want to make sense of very large regions so we can use this additional compute power to really look at very large summary statistics and we provide visualizations do that so you typical interaction with our browsers you look at uh regions of a few million bases and you sort of zoom in and zoom in and zoom in until you really get to the nitty-gritty and to what you want to view and lastly the thing to know about mango is we're built on existing analytics tools so you can easily plug in mango into uh many other tools within the Hadoop ecosystem and uh we're built off existing open source technologies that have lots of documentation make it very easy to develop for so you may have heard of other projects within our research group The Big Data genomic research group at UC Berkeley and uh I'll just sort of briefly describe them essentially uh all our projects are based off core atom API and atom is a project that brings in genomic data into the world of big big uh Big Data Systems so it reads in Legacy genomic formats and then you can sort of format them as Apache parquet and you can operate on them uh with spark so for right now all the projects Adam avocado and Noki these are all parts of the genomic pipeline analysis so these are large computational processes these are very large batch operations and we want a slightly different goal but we fit in right alongside that uh mango supports fast multi- sample visualization where our workload isn't these large batch computational pipelines but rather very low latency selective queries that we want to display in format for visualization so how do we do this I'll briefly describe the architecture uh similar to many of the components within our research group uh we use a layered approach for for modular components this means that uh for the data formats you can use not only Legacy genomic for formats such as bam VCF the stuff a lot of Bio formais are familiar with but you can also read in Apache par and it works just just as well you just point it to the data that you already have staged and uh we can use it in addition you can also sort of swap out the front end component while we think we've made lots of great visualizations and we used D3 to do that um you can sort of swap that out and then mango essentially just becomes data servicing layer where it just returns you the data you can choose what to do with it we utilize commonly used Big Data Technologies so uh with spark and atom these are all very easy to use you can tune your jobs you can see what's going on um you know for visualizations we use D3 so it's really easy to make your own custom visualizations uh mango also runs both locally and distributedly so there's no need to have tools uh on your laptop if you just want to look at something with higher latency um you know you just want look at a small file you can do that it scales elegantly to the amount of storage and compute resources you have so you don't have to do any parameter tuning just boot it up and it should work of course when you're doing low latency queries spark wasn't initially built for this so we've provideed a lot of optimizations within the mango architecture to have these optimized low latency selective queries and I'll briefly go over them so the goal with all these optimizations is to achieve interactive latencies on top of a spark and atom which is batch oriented platform and interactive latencies we describe to be around 500 milliseconds so you can sort of have uh you know interactive um interactions uh with with your browser and we provide three different types of optimizations the first is persistent store so when you want to read in data sets you want to read them into spark you pull them off persistent store first so how do we reduce this initial overhead cost uh once we read it into memory how do we organize it in memory so we can have efficient access in compute on it and thirdly what is the type of computation that we perform on it and how do we make that efficient and meaningful for visualization so for persistent store to limit the initial overhead cost uh we have essentially a two-pronged approach the first is selective access so instead of just loading all your entire data set into memory and operating on it in spark we only want to access selective sub chunks of that data for Park and for atom files we can use a PR predicate pushdown and project to only fetch in the fields of data that we want and the data that spe specifies a given predicate which is usually a range in the genome and for legacy genomic formats we don't really have the luxury of having predicate push down but there are genomic indexes that we can use and in addition because what we make is an application uh a lot of times when user sort of browsing around there will be periods of idleness and during these periods we can uh sort of materialize regions of the genome so that when they actually view them uh they're already there in memory ready to return and right now we optimize for a panning interaction so it loads in the data from the left and the right for memory optimizations we also use two prog croach the first is something we call Laser materialization and the rationale behind this is that uh because we're only viewing small chunks of the data not the entire genome it doesn't make sense to uh exhaust all cluster resources so we have this working set of data uh based on user queries that we just slowly build up over time with the assump that users sort of look at regions they previously looked at and won't really Jump Around uh once we actually store all the data we we store it in a structure called an interval rdd so an rdd uh in spark is basically Spark's default storage abstraction it's it's backed by an array that makes sense if you're loading and all the data but for us uh we want to optimize for an operation called a two-dimensional range query so you can think of this as yeah I have a given range let's say a th000 base pairs on chromosome 20 and I want to fetch all alignment records or all VAR within this overlapping range um we can use an interval rdd which is backed by an interval tree that fetches this very quickly and within the structure we also store the data for multiple samples so just WR there where you need it okay so the last type of optimization we made is computational optimizations so once we've loaded all of our data into memory how do we store it in a way that can provide low latency queries for a visualization setting so our approach is a data tiling approach and a lot of visualization tools use this data tiling approach but previously they're mostly meant for numerical data tiling so we just had to edit the data tiling approach a little bit to work for sequence data because the main abstraction for genomic data is sequences so what we have here is an image of a six- layered approach which is what we use so at the very bottom layer um here we view the actual raw data from The genome sequencer so you can see down there at layer zero we can view the actual raw reads of the data and at every point in the genome we can view every insertion deletion and mismatch at every single read so on the next layer up we can't exactly view all of the raw data it's a little overwhelming especially for the browser so what we do is we calculate summary statistics at every given base pair in the genome so at a given base pair we can calculate what's the Frequency that we see a certain mismatch what's the Frequency that we see a certain insertion or deletion and that's what we display in layer one the next four layers take a little bit of a different approach they use a sliding window approach so what we do is we just take a sliding window in compute an Aggregate and run this sliding window across the genome to compute a bunch of um a bunch of Aggregates that give you an idea of what mismatches are at a given base pair Okay so we've run both local and distributed comparisons um the local comparisons were mostly to um provide parity against existing browsers so this graph you see here is uh the latency of an existing very um a very popular existing state-of-the-art browser and it's run just on a local machine the query pattern we use here is very um indicative of most scientific visualization workloads so we start from a zoomed out region of the genome and zoom in all the way to about a thousand base pairs and then we Zoom left and right around the region of Interest so we can kind of see this spike in the workload um of the latency and this is basically the first point in which the genome loads data so we get a little bit of Spike of latency and then it goes back to around 2 seconds note that the dash line at the bottom is the interactive threshold so this is what we'd like 500 milliseconds for interactive resp response times okay so here's our overlay with the results for mango we have a little bit of a startup overhead once you initially start up mango and we're working to drive those times down but we can see a pretty close to um close to parity for existing browsers on a single machine note we're still far from the 500 millisecond latency however um mango isn't necessarily just intended for a single a single note as we will see in the next slide okay so our next question was how does mango actually scale and this is run on the a th000 genomes variant data so this was about 2,5 100 samples and these queries are run just on one chromosome of data um just like the local uh the local slide we still have some overhead on the startup but we can see past eight cores we start approaching interactive latency we see that it kind of starts flattening out from 32 to 128 cores this is kind of due to a bug and Spark that we're looking to uh fix but once we F that fix that bug we um we think that the scalability will um scale out a lot nicer so so we've made a lot of progress but uh there's still a ton of work to be done the first thing we want to do is improve the flexibility of our application so right now uh when servicing data me provides a very simple interface where you basically input a start and end range and it fetches all the records within that range but we understand that a lot of genome browsers provide essentially their own query interface where you can issue uh pseudo SQL queries and have uh much more advanced queries um we can do this by sort of extending the spark SQL package and doing so will dramatically increase the usefulness of our data servicing layer we also understand that in scientific and genomic analysis reproducibility is really important so uh the creators of the spark notebook gave a talk earlier today about uh you know integrating atom and some of the work in our group uh to the spark notebook but if we sort of extend that to Mango and uh custom genomic visualizations that'll increase the usefulness of genomic analysis and finally one of our optimizations right now we have a very uh simple pre-etching algorithm that fetch materializes regions to the left and right um but if we model user Behavior we can better uh issue suggestions for interesting regions within the genome and materialize those to further decrease the latency and Achieve interactive times so Mango's under rapid development uh right now it's just us two working on it uh but if you want to help out if you want to uh check us out you know give us feedback please visit our GitHub bigd genomic mango and if you want to look at all the other projects developed in our research group uh go to BD genomics. org so thanks for listening and now we're open to [Applause] questions rainbow colors and how many colors were there oh sorry there's actually I'll rep I'll repeat the question oh uh I was looking at your visualization of colors and I was wondering how many colors were there on this graph right here or this there's actually only four really yeah uh atg andc um oh I guess six so we have a color for uh insertions and deletions um but this is at a pretty large range in the genome so some of the colors kind of merge together um uh and also I was wondering what the value of the VIS visualization was because you know there's a lot of data to take in do are you trying to take advantage of the human ability to pattern match yeah so um there are cases uh in in which people use visualizations when they already know what they want to look for so in that case I wouldn't recommend mango I would recommend a local machine where you have a specific point query so yeah uh it's the ability to pattern match and the ability to explore so so what we wanted to support is those Point queries on large data sets as well as zoomed out regions so it is it is um it is depending on the individual's ability to pattern match but if if we keep trying to abstract that uh process it often turns into a terminal you know so it's it's a very uh fine line between allowing the user to explore and then uh just you know creating a command line uh interface that people can use um but yeah I I do think we could do more in pointing users in the correct direction we've looked at U user modeling and data modeling um algorithms that can actually refer users to specific areas based on what they've already viewed we just haven't implemented them yet so cool just a quick question uh thanks a lot for a great talk um I might have misunderstood the problem that's being solved but um have you found a need to re imp mentioned a range Tree on the client in the JS or is it more um you know the query small enough to just send to an individual client every time uh right now all we have is sort of back end that that sends it to the client every time but we're working to um like you said sort of cach the results on clients so you you uh you sort of shortcircuit that roundtrip time back to the server thanks great well uh thanks a lot thank [Applause] you