sfspark.org: Frank Nothaft, Scalable Genome Analysis With ADAM
Recording: sfspark.org: Frank Nothaft, Scalable Genome Analysis With ADAM
so thanks all for coming out today i'm frank nothaft i'm a starting my third year and my PhD up at up at berkeley across the bay so i'll be talking today about adam a tool that we've been building to do high speed high throughput genomic analysis on top of the paci spark I've tweeted these slides out so I'm a F no tapped on twitter if you'd like to share them with your with your colleagues please feel free to so you know I don't I don't know what the makeup of computational people vs. biologists is is out here tonight who here is a computational person and then who here is a biologist great so so i'm going to i think that my my assessment that most people here probably don't know everything about genomics is probably a fair assessment so i'm just going to give a brief background to what what genomics is like what are some of the analyt you know the analyses that we want to run what are some of the algorithms that we're running here look like fundamentally when we talk about analyzing genomes our goal is to go ahead and come up with some understanding of the source code for life so you know the human genome itself is about slightly over three billion letters long it's split into you know one might say 46 files so where each file as a chromosome so this is it just a long piece of DNA you get one copy of each chromosome from your mother one copy of each chromosome from your father and then inside your you know inside this cook this whole genome this 3.2 billion character long file 99% of places in the genome are identical between everybody in this room more or less you know that it's it's kind of shocking you know even though we all look very different we all act very different you know we all have very different personalities we're actually ninety-nine point nine percent the same so the the changes that we're looking here the changes that give rise to diseases like cancer or Alzheimer's they're really needles in a haystack but one of the things that is also kind of interesting to note is even if we're talking about one change in one you know in every 1,000 places when you're starting with a 3.2 billion long billion character long file that is still a lot of changes so we've got a lot to a lot to crawl through here to find out what it is in these in these minor these small edits most of which are just a single character substitution that leads to you know to us being as a human species so wide and diverse and then what leads to the diseases that people get and the way that we've been studying this it you know really this is a trend that's picked up in the past about ten years is through this abstraction called genome sequencing so you know you know everybody here is oh you know genome sequencing is so hot you know there's so much more genomic sequencing data that's being created now then five or even ten years ago and a lot of this is is due to this relative this revolution called next-generation sequencing so what we're doing you know we take that 3.2 billion character long file we replicated a bunch of times and then we cut it into small sequences so you know for those of us who are more on the like the machine learning the statistical side of things essentially sequencing is just a it's just a plus on substring sampling process we take your genome we replicate it we cut out these you know right now most people use strings these little string fragments are between 100 and 250 basis long and you know we we sample from that pool and under a process that looks essentially like plus on you know plus on sampling and you know just using this process for you know for a thousand dollars given a couple of days we can go ahead and get a copy of your genome that is just you know a bunch of these fragments you know about 1.4 billion of them however as all of you are probably sitting here and saying what am I going to do with these you know you're telling me the genome is 3.2 billion bases long and you're telling me that these substrings that I'm getting our hundred two hundred and fifty bases long what am I going to do with that information the the general thing that allows us to do this is this this process that we kind of in the field referred to as genome ricci qin Singh so you know probably everybody remembers back in 2001 there was a lot of fanfare about the Human Genome Project what the Human Genome Project did was it went ahead and spent a billion dollars to go ahead and generate a really high-quality draft of the human genome so you know we know you know we established earlier we're ninety-nine point nine percent the same so the big thing that we're going to do with these little substrings is we're going to use them to calculate a diff so you know we're just going to calculate you know figure out oh you know where does a string map to the genome is it different from the genome at that place is it really different is it just some error that we saw so once we've gone ahead and computed you know once we've gone ahead and captured the sequence we're left with two important problems and one of them we have a fairly good handle on today that's how do we actually go ahead and compute this difference but the second one in a bigger one is what do we actually do once we've gotten these differences you know what do we do to make meaning of these small edits that are sprinkled throughout the human genome and so the so the way that we get started with calculating this diff is through this process where we align the substrings so you know we take our you know our bag of strings we just place them to the genome and then we we use the statistical process to say okay well you know so let's say instead of seeing an e here ice on a you know it's a nonsensical word you know there's no tames you know that are you know ti mas that's not a word that anyone knows but is that a random error or you know did the sequencer just mess up when it read that base or is this actually a real change you know is someone you know one of those youth you know that are always going ahead and misusing words on misusing words everywhere giving rise so you know crazy new vocabulary you know are they trying to trying to change the word x into the word TI mas and that's essentially what we're doing with this whole genome assembly process we're doing this alignment and then we're using a simple model you know the model may even just be you know again for the statistically inclined you know model that assumes that there is some you know binomial distribution of variation in the genome and then you know we just look and see okay well you know given our given our error model do we have enough evidence that says this is an error or we'll do have enough evidence to say that this isn't an error or do we think it's just something we can throw away a fragment you know something that was lost during the sequencing process so the you know what does this kind of leave us with right now you know we we have these tools and we're going ahead we're crunching the genome but you know when you take 3.2 billion letters of text and you replicate it 30 or 60 times you actually wind up with a lot of data and you know to be to be more precise if you're looking at the human genome this you know about a 60 x replicated human genome is about 200 gigabytes of data so the way that we're processing this right now is in this long pipeline that runs all these stages cascaded one after another we're essentially each stage is some you know it's it's some sort of an algorithm that's cleaning up your cleaning up what we see you know it's in some of them actually even map to really common you know really common analysis steps that one might do you know this mark duplicates stage as its name implies would map to a data deduplication yo stage that you might run in a common etl pipeline you know these these recalibration realignment stages they're just doing some data cleanup they're just going through finding systematic error that's in these fragments that we've sequence and cleaning them up then you know kind of as we run through the rest of our etl like pipeline we're going ahead running a statistical process we're cleaning up the data we're down sampling and what we actually wind up with at the end these variants that we have here that we can go ahead and analyze for you for whatever disease process we're looking for you know that's about a hundred times smaller than the data that we started with so we might go from a 200 gigabyte input down to you know a 123 gigabyte output but there's there's a few problems that we have with this pipeline right now and you know actually this this one that i'm showing you right here is a a pipeline out of MIT and harvard that's it's called the genome analysis toolkit the name is very straight forward to the point it's probably one of the most widely used tools out there but it's you know it's it has a lot people who you know who detract on it because it's kind of slow you know it's it's hard to tell what all these stages are doing this whole pipeline to run ends and takes you know three days to a week to analyze so you can imagine you know I'm someone who comes in I have cancer I want to find out you know is there a genetic cause for my cancer could I use this to pick a better treatment you're going to have to get your genome sequence and then spend three to seven days waiting on the results of this pipeline additionally if we look at it from the from the lens of you know what what are the best practice tools out there in data analytics right now a lot of these systems are relying on technologies that were commonplace when you know when the sequencing trend picked up you know the the files that we're starting with when we get these this raw data in some of the time it's even stored in just a pure text file you know we have you know we might start with six hundred gigabytes of data stored in a text file we might have sorry we might have some of these some of these binary files like BAM or cram that are used for storing genomic data they actually have standards that are just hand implemented painstakingly by people you know I was actually I was going ahead we were playing around with some some legacy data yet yo just yesterday we're trying to make the the reference java implementation of the VCF you know the variant encoding file format talked to the see if the see reference standard implementation and we couldn't get them to talk because there's some slight error in how they process header files and this is coming because there's some you know there's some humans sitting in a research lab somewhere who wants to analyze their genomes but you know they've been told by their you know they've been told by their advisor that they need to work on the on the standards file getting that to work and hey you know some human error slipped in somewhere additionally these architectures that people are using are you know they're very they're very bare-bones they're very flat you know one of the one of the big abstractions that we rely on in genomics is that all data you know most of the data that we're working on is a line to a position in the reference genome so you know if we're trying to look oh you know how does my genome differ from yours you know we might say oh well you know I'm interested in the brick of one gene you know I want to see if Frank's brick a one-gene is different than someone else's brica one gene and we say oh well you know we know from studying the human genome exactly where the human genome is or sorry exactly where the brick of one gene is and we'll zoom in on that the you know the way that the way that these legacy tools will let you process data is you get you know you normally get one abstraction and it's just we give you a sorted iterator you know just you seek on the iterator until you hit the position that you're looking for and you know I'm sure that programming against a sorted iterator is the level of abstraction that most of us would like to rely on in our day-to-day life you know we just wanted iterator that's all that we wet all that we get you know none of these fancy map things none of these fancy MapReduce schemes that we can program against so you know people actually really have to kind of contort their algorithms to fit this sort of really bare bones programming model additionally you know one of the one of the funny things that we you know whenever I go and talk to biologists you know one of the tropes that we have is you know some of the time the most efficient way to processes data is I just go ahead I'm processing five gigabytes of data that's in a text file format by you know by processing it with said all can grep on a command line you know they're they're better abstractions to be using to do these ad hoc queries you know we have nice tools like spark SQL we have Impala things like hive that we can use to do these queries so these low level abstractions aren't productive we tend to use flat architectures that lock you to a specific you know a specific computing architecture like a lot of these tools are designed for hpc farms with really expensive share data storage and then you know one of the really subtle things is that when you're programming at this level bugs creep in you know you have to keep track of all of these you know weird invariance about your data in your mind you can't rely on a higher level programming environment to do that and we've actually run across several you know several places where these tools have introduced bugs just because oh well you know I was program against sorted iterator you know I didn't think to check this weird invariant that I had my data that would have been trivial to check if I was writing you know say MapReduce and then checking for something weird happening when i'm doing a reduce so you know about two years ago we started thinking hey you know at the same point in time that this next-generation sequencing got hot in 2006 a lot of things were heating up in the analytic space you know I think the first first production release of Hadoop was 2007 spark came along 2010 2012 and you know there's been a very similar growth in analytics as there has been in this genomic sequencing area so we thought hey you know we're going to go ahead and we're just going to use the best of breed tools that are out there right now try to define a new green field approach try to build some of these common genomics analysis tools on top of that and you know really the first thing that we did was we did what any classical database person would do and say hey can I can i come with the schema for my data in the the big reason that we did this is that you know if you if you start to plan ahead a schema provides a very nice narrow waist you know so we actually use Avro this is a slightly hacked version of our schema to make it a little bit more reading readable but we use so we use Avro for the schema for defining the skeena schema for most of our data what's nice about this is that we can easily then process our data inside of spark we can process it inside of Hadoop mr we can pull it into spark SQL we can change any of the way that we're actually doing our processing up here without having to refactor our schema so you know once we once we came up with this schema one of the big things that we realize is well you know we're talking about how there's a lot of a lot of processing and genomics relies on this abstraction where the genome is a coordinate space and we align to it so we said okay you know we have spark spark is pretty performant for most things there's actually a lot of genomic processing that you can do with just map filter group by but you know if you start looking at more you know more advanced tasks we can really accelerate some of them so you know one of the big things that we did was we came up with specialized joins oh jeez so you know to deal with this this genomic aligned data where we have data that is put at a certain point at a coordinate space frequently you know I want to take one set of data and overlap that with someone else's set of data so you know maybe i would take my genomic variants and John's genomic variants overlap them and see oh you know is there a lot of commonality between these with where these Gina you know where these variants hit in the coordinate space where where we have areas where there's not an olive commonality we might then take a description of you know where genes are in the reference genome overlap those and see if you know see if there's any interesting results and it turns out that you know if you're smart about how you use partitioning and spark you know if you're smart about the algorithms that you actually use for doing these comparisons you can make this you know we call it an overlap join you can make it into a really efficient like sort merge join so we had a lot of fun hacking around on the algorithmic side to accelerate a lot of these common genomics patterns additionally you know one of the big things that we did at an early pay at an early date was we thought about what was actually the correct storage format for this genomic data and you know there's a few common you know a few interesting things so genomics relies a lot on slicing and dicing of data you know for certain for certain algorithms I only care about a subset of my fields and that's very common you know like if you went back a couple slides our schema just for describing these short genomic reads you know these these fragments that were sampling it has 33 different fields in it but most analyses will touch a subset of maybe eight of the fields so we don't want to materialize the whole record whenever we do one of these analyses where we touch you know we touch less than one-fifth of the fields in the record additionally you know we commonly don't touch the whole data set you know we might say oh you know even though I have this whole genome I'm just interested in a gene that's on chromosome 20 or a family you know an interesting set of genes that are on chromosome 20 so we very commonly want to go ahead and only touch a subset of the data so this actually matches really well with kilometer storage so we use the the Apache Park hey format to go ahead and take our you know take our schemas and materialize them to disk in in the park a format and we've actually been doing a lot of work recently to try to improve you know to try to improve some of the performance of both parque on disk and spark and memory by you know one using parkade actually store data on shuffles and spark you know so this can give us some compaction we have some fields that have a lot of repetitive metadata you know by storing it in a kilometer form when we when we do the shuffle we take up a lot less space when we're doing the shuffle you know we pay a slight cost for going going row oriented two kilometer but you know that's that the the space savings are worth it and additionally we've been working on some optimizations to park a recently to try to improve the to try to improve the performance of some of the filtering code which we rely heavily on sorry about that and so one of the actually very interesting aspects of going ahead and saying hey you know we think this columnar file format you know this columnar approach is better for genomics is that we've actually gotten a bit of criticism from the genomics community that says oh you think you think our file formats aren't good and here you come along just like everybody else who has come before you and you're going to introduce another a fifteenth file format whose problems does it solve so we actually had someone tweak this at us a while ago which was which was kind of fun but you know one of the really subtle points and one of the great things about having chosen this schema driven approach where we're using a scheme as a narrow waist is it doesn't matter what we're storing our data in as disk you know it's it's actually very trivial to go ahead and say okay well you know if I want to process if I want to process legacy data you know then I just define some way of going from this legacy file format to my schema you know I just I just define some conversion that I can do and this is actually really easy to do so you know one of the one of the things that's kind of great is that in the process of developing a flexible fast Shin processing processing stack for genomic data we've actually developed a Swiss Army knife for converting between genomic formats and everyone hates us for it not I jest i jest but you know kind of the simpler way of looking at it is that yeah you know if you put your database hat on this is essentially a view you know if you squint at it you know this is essentially a view so this was this is kind of one of the nice things that came out of the stack based approach so you know just to shift gears for a little bit you know I want to talk about some of the some of the work that we've been focusing on recently which is how you actually use this you know how you use this software stack to go ahead and analyze these genetic variants so there's actually a very big very big challenge in this where you know okay so we know we know that fundamentally you know the differences our genetic code drives the differences in our you know physical appearance mental state what diseases we get however you know commonly when we when we go ahead and we analyze your genome and we wind up with this you know I guess what we have a bag of three million variants at the end a lot of these variants don't really have straightforward explanations so really you know what we what we wind up with is this challenge of how do we find what variance is important you know how do we how do we go ahead and say okay you know this deletion that occurs at base 300,000 in chromosome 10 is why you are ten percent more likely to get Alzheimer's over the course of your life and so the big question is you know how can we actually go ahead and make sense of this variation and ultimately there's really two drivers so you know one is one is very simple you know a variant can go ahead and break the product of a protein so you know if i have if i have a variant in you know in the body of one of my jeans it can actually change the outcome of you know translating that gene into a protein right so you know it's it's essentially i change the recipe i get a different outcome but the other thing that's that's very subtle in that we've started looking at is that a variant that you have even if it's not inside the body of a protein can change how much of you know some protein in your genome is expressed so I'll talk a little bit about this one a little bit about this one later but the you know really the question is how do we how do we turn these genomic variants that we have into a profile for you know what diseases you might have what disease as a population might be susceptible to and the big way that we do this is you know it's kind of obvious in retrospect it's just you know through statistical modeling and some of these some of these are actually really simple simple approaches that we use you know like one that one approach that's very common is just to use a standard chai squared test you know where you you essentially you run an aggregation query over every single spot in the genome where you have a variant and you say okay well you know I'm going to look at who have you know people who have this variant people who don't have this variant how often do they have this this trait or this disease that I'm interested in and I'm just going to run a try square test on that alternatively if we have you know if we have maybe a more complex trait you know we want to look let's say at something that's a continuous thing you know like human height for example we might go ahead and run a more complex model so you know we might run this aggregation query and run maybe a linear regression model on top of you know as part of this aggregation so this this body of work is kind of generally known as genotype to phenotype testing and you know one of the great things again with using spark is that these sort of aggregation queries are a very natural map to the computational framework that we have so you know there's a reason that sparks became popular in machine learning but you know these aggregation queries or you know a lot of these aggregation queries are actually like fundamental etl tasks you know even you know before you get to the to the machine learning to like to the machine learning side so it's just a very good computational fit but one of the problems that we run into really quickly you know okay so i did my statistical modeling but something that's kind of a disappointing result is well you know very commonly i'm going to have these two changes you know these are single single base modifications we call them snips single nucleotide polymorphism there you know maybe maybe these two are 500 bases away from each other and it turns out that they have the same probability value you know both of them are correlated at the same you at the same rate with me having a disease or having a change in how much of a copy from a protein that I make something like that and this phenomenon is you know it's called linkage disequilibrium so it's essentially just a simple way of saying you know what a lot of us know about locality that you know well if you know two things are close by to each other in the genome they're probably going to get inherited together so you know even though you know probably only one of these at least for the for the rhetorical purposes of this talk let's say only one of these variants you know causes a certain disease maybe one of these variants makes you five percent more susceptible to diabetes you know because they're close to each other they frequently get inherited together and thus you know we can't go ahead and figure out which one is actually the culprit so the you know one of the questions that I've been thinking about recently is you know can we go ahead and break up these blocks and you know in some cases this is easy so you know getting back to when we're saying oh you know if the variance is in the body of a gene then maybe we can predict how the gene is modified and this is this is a common your common thing that people have done where they'll say okay you know I have you know I have these three base pairs you know I have this repeating thing of three base pairs that goes ahead and tells me what amino acid to make so if i have a variant that is in one of these that changes what amino acid i'm going to make you know say i have it i modify an a to a see that tells me you know to make something completely different maybe that's going to change what protein i actually get maybe it's going to change how the protein gets you know spliced or folded together so at least conceptually when we have you know when we have variants that are inside the body of a gene if they don't modify the protein if they don't modify you know how the protein is spliced together we can probably go ahead and discard it so you know if if we knew oh you know this protein didn't cause any functional change this protein did and they were in the body of a gene then we could say okay well you know snip too you know you're highly correlated you're not actually important we could toss snipped you out but you know this this picture actually you know kind of just a bit of biological no imagery nomenclature this little arrow here implies that this is where a gene starts being transcribed into a protein so these protein or these variants don't actually have any they don't cause a functional change in what you know what protein not get it so the the big question now is well you know it turns out that the part of your genome that actually codes for proteins is about 1% of your genome it probably stands to reason that that 99% that isn't actually you know getting transcribed into a protein it's probably important for something it's probably not there just because you know nature really likes replication really likes leaving comments in its source code and you know just a little a little bit of credence for this kind of comes from a project that we've been looking at that's focused on acute myeloid leukemia and what we have here is just kind of a decision tree a based approach of looking at how yo how mutations are acquired in this cancer so you know if you branch off to the left that means that you did you know so actually each one of these bubbles is a gene if you branch off to the left at one of the bubbles that means you did not have a mutated copy of that gene if you branch off to the right that means that you did have a mutation the first thing that you'll notice in this chart is we have this huge long tail of people who have no mutations and this actually this accounts for a fairly substantial portion of people who have acute myeloid leukemia so this tells us that you know these are the people who have known no mutations in their genes you know no mutations in the coding bodies of their genes but yet they're still getting this cancer and we know that cancer is a genetic disease so what's going on here so one of the big things that we've started doing is you know there have been a group of people who are kind of like in the theoretical biology systems biology community who have proposed essentially a grammar for what these modifications mean so you know you might say okay well if I have a modification that occurs yo let's say 500 basis before I start changing the sequence into protein then you know if that goes ahead and changes how two proteins bind to the DNA to actually do the translation process then maybe I'm you know maybe I'm not going to get any of that gene or maybe it's going to modify my my the rate of expression of that gene and some harder to predict way so we've been going ahead and building on top of atom using a lot of these you know these techniques like these aggregation queries that were running these coordinate space joins so you know we'll go ahead run a coordinate space join to pick out these these areas of interest you know will then go ahead and run these aggregation queries to try to associate some some level of significance with you know with what we're seeing to try to understand this and the you know the big goal you know what we're trying to get out at the end of the day is we want to take this statistical the statistical model that gives us these you know these probabilities scores you know and go ahead and and annotate it with more data so you know we want to annotate it with more data add more data in just keep on joining data in until we can fundamentally understand better you know why is it that snip to is actually the causative you know why is that the causative variant why is that the causative mutation and not snip three even though their inherited together so you know that's that's what we're working on I you know there are lots of great ways to get involved in this in this sort of work if you're interested you know for our group at Berkeley we we have a project called you know the Big Data genomics group were organization on github we have a variety of tools that we're working on kept there all of our work is apache to licensed some of our projects have actually very large numbers of open-source come contributors some of the time it's people who are in labs sometimes people who are working at biomed corporations and sometimes it's actually just people who are you know developers who you know in their spare time said hey you know this sounds like a cool idea i'd like to learn more about this and just said hey you know this is a small patch that I can make to help so you know we we try to keep track of issues for all of our projects and wherever wherever possible we try to tag them with pick me up if they're an issue that either doesn't require too much spark knowledge or doesn't require too much genomics knowledge to get started you know additionally if you're interested in in the subject but want to get more of a tutorial one of my colleagues or a loss of sin from Cloudera put together a chapter in the advanced analytics with spark book that that just came out from O'Reilly a couple months back it's chapter 10 and it just runs through an exact example of how you would use spark to implement an example genomic analysis so he's really interested in immunology before he before he went to Clara he was he was an immune ology PhD student so his analysis focuses a bit on how you can use spark to understand and you know the immune system a bit better additionally we had a paper come out on Adam just just back in June or just back in May so we have a lot of the more technical details on Adam in that paper and additionally i'm involved in a standards organization called the Global Alliance for genomics and health I didn't talk about this much in today's talk but you know biology faces a big problem where data is frequently stored in silos and data sharing is very difficult so there you know this this standards body is created with the sole goal of coming up with computational solutions that allow it you know that make it easier to share data one of the projects that they have that they've been really looking for software engineering help on is this project that is their their server implementation so it's just a way that you know people who you know maybe I have a small lab I don't know a ton about computers I just want to go ahead and set up a simple server that I can use to share the data that my lab is collecting with people at other labs it's actually I believe it's just a python-based web server so they're always looking for people to contribute code the code base is fairly clean fairly simple fairly straightforward to get started on so you know these are a few good ways that you can you can kind of jump in and you know if you're interested in kind of the computational open source environment ology so you know thank you all for listening today I hope you found it interesting I would just like to thank yo although I am standing in front of you this work is the product of many people's laborers you know we've actually had over 40 people who have contributed to the atom project over at more than 12 institutions so I just want to thank my collaborators thank you definitely sure sure so the so the question was can I can i talked about how Adam relates to the Google genomics platform so you know i know google google has been working on their genomics api for about the same amount of time that we have been and so they have several different api's some of which i'm more familiar with than others so you know the caveat is that my information is a little out of date but you know summit so one of the things that Google has tried to make fairly easy to do is they've tried to make a set of you know kind of pre-canned common analyses available and easy to run so there's a there's a large you know a large bevy of common analyses like variant annotation you know trying to annotate these causal effects on to the variance or trying to go through and turn reads in two variants that they've made easy for people to submit their codes are easy to submit their data to through kind of a combination of a web-based interface with a yeah I guess with a tie into data storage on the google google cloud engine for atom itself we've you know our so one our tools are less platform dependent you know obviously we rely on apache spark but other than that you know we're we don't like we're not tied to the amazon cloud infrastructure you know we've actually run it on standalone clusters at you know at our side and at other institutions amazon cloud the azure cloud so you know we're very we're very portable from that sense our AP eyes are a little lower level you know if i if i wanted to kind of capitulate what Adam is as a tool Adam is actually the the first level of an API for going ahead and defining a lot of these genomic queries so it's very general and then we've built several tools on top of it to implement these algorithms you know it's like for me I spend a lot of time working on ova kado which is our variant calling pipeline you know we have several other projects you know that are focused on doing gene expression analyses and and whatnot so you know there I think they're similar technology or you know similar stacks you know I think the Google stack under the hood lies on many technologies that are similar to what yet to what we're doing I've heard a lot of really wonderful things about people who are using it especially to analyze variants I just haven't played around with it a ton myself oh oh sure sure so the so the question was on this slide when I was talking about about cancer data I implied that the data was very homogenous when most of the time cancer data is said to be very heterogeneous I think I had misspoke when I was talking about this so part so we actually so my point is that we have this long tail here that we can't really explain through conventional variation like variant analysis like variant annotation methods but actually you know there are several clusters in this tree that drop out into very clear heterogeneous clusters so if you know if you go in there's like off the top of my head there's about eight different subtypes that that you'll go ahead and correlate with well understood mutational groups you know like some of the common ones are like there's a flit three internal tandem duplication that has two different subtypes that associate very clearly with it my point was just that you know there's this long tail here that our conventional methods don't really explain because they're capturing you know non non exonic variants yeah so this so this so the question was is this patient data or from the cancer itself so this data is from the TCGA AML cohort so it's about 200 patients that were sequenced at think mostly wash you in st. Louis the the data is published so we actually like this data I can't remember if we got it actually from the mutation calls or from their spreadsheet that annotated the mutation calls with the genes but yeah it is from real patient data and it's it's from them you it's from the mutation calling pipeline so these are the mutation calls got it got it yes so so okay so to be a bit more specific so the way that this data was obtained there was by so this is a diff actually from the germline against the the cancer cells so what they did was a sequence the you will they sequence the you know like I think skin cells for the germline and then they sequence the blood cells for the quote-unquote tumor you know it's a ml so it's not really solid tumor and then the variants that they've identified here are the variance between the the non mutated cells and the mutated cells sure um and you know this this was actually something I was talking about with Alexei earlier so the question was if I'm if I'm a newbie coming in I don't have much biology knowledge but I have a lot of computational knowledge I have a lot of experience with spark a lot of experience with Scala something like that how can I go ahead and start contributing and so there's a few answers you know one of them actually you know we've had a lot of people who come in with computational knowledge but not necessarily the biology knowledge who say oh you know I'm going to start out by maybe I'm going to go through and I'm gonna find one of the algorithms that you're running I'm going to profile it i'm going to put in some performance improvements and then go ahead and you know that's how i'll learn my way through you know what this data looks like what are some of the algorithms that people are running and you know while that sounds like like a bit of a joke we've actually had several people come in i think we've got in one algorithm we got about a thirty percent speed improvement out of a person who just came in from the open source world was like oh you know i don't know much about genomics but i know a lot about profiling code and you know they they got us about a thirty percent speed improvement one you know one big algorithm we run and ten percent in another one so that's that's one place and you like especially like for us we realize that there are a lot of things where you know it's easier to get into some parts of the code based if you yeah well some car some parts of the code base are easier to get into without too much biology knowledge so you know like when we have our issues when we you know when we post them on our github we try to tag those with hey you know you can probably get into this one without knowing too much biology on the other side you know some of the you know while some of the algorithms are really you know some of the algorithms are really difficult to play with there's a few things that people have just been like oh you know I'm just going to look at this data with algorithms that I'm already familiar with you know there's there's a few people who've come in from the open source world people coming from the software side without much you know much biology expertise we're like okay you know I want to you know I know in biology I want to understand how these variants correlate to populations and I know how if you know if this was if this will is normal you know quote unquote normal data that came out of an analytics pipeline maybe i would go ahead and try to figure out how this data is segregated by running a k-means model or maybe i would figure out how it how its segregated by running some other type of statistical model so there have been you know two people who've come in from the open source world who have been like oh you know i just kind of want to get my hands on the data what happens if i apply some of these statistical pro like pro statistical methods that I already know to the data just played around with it a little bit and you know that's that's been kind of a way that people are like okay you know i'm going to like bite off a problem that sounds like kind of reasonable and just do some exploratory data mining and see what happens and it's actually led to some pretty cool work so we have a few blog posts if you go to our website BD genomics org that are from people who have done that and who explain what they've done sure sure so so as the question was ok so I'm interested in biology I'm going to take an approach you I'm going to take some courses on EDX or Coursera or something to try to get some knowledge of the space that way how easy will it be to take the knowledge I've gotten from these courses and translate it to the genomics area and this is this is a really great question i'm glad that you brought this up and i'd actually really like to highlight one of the MOOCs that you mentioned you know there's a there's a professor at Johns Hopkins been Langmead he's you know he's one of the most you know most highly cited people who are about his age in genomics does awesome incredible work actually he doesn't he actually does some of his own work on top of Hadoop actually and he's teaching a he's teaching a MOOC I guess you know right now we're starting very soon so yeah I know everybody a lot of the people in the genomics area are really excited for that and you know more you know getting back to getting back to your question getting back to the point more you know that's actually how a lot of people who are you know in our group well you know our our group broader you know a lot of the kind of people who are at Berkeley who come over from the computer science side have gotten into this so you know one of my colleagues Matt Massey he's you know he was the person who actually started Adam he's the first person who committed any code for it that's how he picked up a lot of knowledge about about genomics you know the two of the people who are the drivers on this AML project actually you know one of them is a computer networks researcher and the other is a computer file systems researcher you know he actually cut his teeth doing like mock colonel design like back and you know back when mock was like the cool thing in kernels they got you know they learned most of the stuff that's led to the work that we've been doing in the like in the cancer space from from watching mute from watching MOOCs actually so I think you know it it takes a little bit of creativity to translate you know MOOC into you know into algorithms but you know especially like some of them like the bent the bend Langmead course I think will will have a very straightforward transformation or very straightforward translation but ya know it's it's definitely a great way to get started to understand some of the background more sure so the so the question was this the platform that we're talking about is mostly designed for handling eukaryotic genomes so you know genomes of you know humans so on and so forth versus prokaryotic genomes which are you know like viral genomes and so on so I haven't done any analysis of viral genomes with it conceptually there's no reason why not and you know i mean like one of the big things that i kind of get the sense in the in the eukaryotic genome space is that there's a lot of people who want to you know instead of going with alignment based approaches they want to go with assembly based approaches so I've actually been doing a bit of thinking along with some other people in our group about how we can efficiently support assembly you know one of so one of the processes you know we talked about the human genome project many slides back what the human genome project did was you know since they didn't have a human reference genome they went ahead and they assembled one from scratch and a lot of these algorithms that you run to go ahead and assemble a genome are there graph-based so we've actually been doing a lot of work using graph x which is a spark library that provides efficient graph processing sport on top of spark to do these sort of genome assembly tasks so you know the short answer is I don't know if anyone's pushed you know that sort of data through it yet but I believe that you know long term it shouldn't be something that we can't support yeah cool