sfspark.org: Alexy Khrabrov interviews Frank Nothaft
Recording: sfspark.org: Alexy Khrabrov interviews Frank Nothaft
hello everybody I'm Alexa crabber the organizer of sf's park in France and here we are allocation chartboost and we have a very special meet up tonight it's called open genomics skallon spark for good how open source developers can help in genomics research and cancer research and it's a very important topic coming out in a personalized medicine we have two great talks today from employed in Berkeley and from driver genomics and here we'll also have frank not huffed who is a PhD candidate at my pad berkeley hello Frank it's great to have you great to be here so I think Adam and spark set up as the project from a blob it's really well known by this point in time and the community and so you're one of the folks directly working this so maybe you can tell us a little bit about that project sure sure so you know we've been working on the atom project for about two and a half years right now and the atom project is an attempt to use the spark and the Hadoop stack to try to process genomic data in a faster easier to work with way so you know when we when we started looking at the problem most of the tools that are out there in genomics right now are focused on a single node and they are you know they're based heavily around some of the legacy file formats and legacy technologies that were very common when the next generation sequencing error really started getting kicked off you know two thousand six two thousand eight uh you know since then you know one of the big realizations that we had is actually this this huge growth in genomic sequencing has really paralleled how you know the rise of all these high-performance analytic technologies came up in the you know in this big data space so we saw a very good opportunity to go ahead and take some of the tools that were you know coming out on the scene whether it's a tool like spark you know I'll format like Park hey you know data serialization method like Avro to just go ahead and make it easier to process large amounts of genomic data at a high rate of speed hmm ah you know so over the past two years what we've been working on is trying to essentially replicate and improve on some of the current standard practice in genomics so we've been trying to tackle some of the big problems that people have in the data management side of genomics you know whether it's making sure that the data can be easily processed by a wide variety of tools can be easily accessed queried you know one of the big problems that we've run into is that the legacy tools aren't very easy for people to run ad hoc queries about against so you know there's there's kind of this meme where if I if I'm going to run some ad hoc genomic experiment you know I'm going to be doing this through command line tools like awk grep said so on and so forth that's all right if you just want to look at a single person's genome but you know when I'm scaling up to tens to hundreds thousands of genomes it just doesn't work well so you know we've been we've been working heavily on these pipelines we're starting to focus more on the ad hoc kind of the data science side of genomics so we're pretty excited about where you know where the technology's heading so this is a mention seven or cry because I think ten years ago I'm in like a previous year so cut these books coming out from areli another publisher is no pearl football for my expired and photographer my physics right so it's kind of interesting maybe you can explain to kind of folks who are not familiar with specific technologists why spark is a better vehicle for this kind of problems than you know 70 can Perl and Python sure sure well you know there's a few things and the the big first thing is if you look at the computational patterns that are in play you know you have these tasks that are processing very large amounts of data you know a single human genome in you know non summarize and it's just raw as form is about 200 gigabytes of data mm-hmm I you know when you start talking about these large file sizes moving to a parallel system like spark that's really optimized for making optimal use of data locality high-performance file systems is just you know it's it's a good step in the right direction um additionally you know one of the big reasons that we think spark is a good fit for this less so then you know legacy Hadoop was is that a lot of these pipelines rely heavily on iteration no they may be a lot of the algorithms that people are running are similar to machine learning algorithms you know you have lots of iterative algorithms but additionally even the non machine learning like algorithms tend to rely on you know a long pipeline that is you know job cascaded after job cascaded after jocks three matching right then things like that exactly and it's you know that's a good fit for spark additionally the you know one of the big advantages that we see you know over over a lot of these legacy tools where you know people are going ahead and they're piping tools through command lines and so on and so forth is we've taken an approach that focuses a lot on having a you know a schema as a descriptor of data instead of some binary or text-based format mm-hmm and there's there's a few advantages to this you know right now you know I can take this you know I can take this ad hoc query that I'm running on the command line and I can go ahead and I can move it to a tool like sparks equal or Impala and I can run my you know I can run my data analysis queries in that tool in a higher-level language then you know someone who's just piping to your piping things around on the command line yes yes and that really reminds me right to one of the themes of the upcoming it's called by the bay conference we call it no et al right an etl basically means you dump something in the text and then you par or par so you lose that the schema you lose the typing information so your bicycle it's interesting that you're focusing the same advantage that they are keeping the the information in between the stages yeah definitely and I know that you use an average Bar K for this purpose mm-hmm exactly and you know we've been doing a lot of optimizations recently to you know that try to improve our performance you know with those technologies you know there's a variety of yo we got very good performance right now but we found a couple of you know actually a glow hanging fruit that allow us to optimize things even further huh interesting I hope to hear more about it so one question that in a lot of developers in this community ask especially bc huge infuse I'm about you know the tonight's event will have more than 200 people signed up people comes with different companies and the first reaction when we share this will share this at night Ron the shares are the company says basically how we can help right so people even note in in biotech space 12 for instance enable their purse or developers to work on this right they want to contribute and obviously this is not as easy as kind of picking up a how-to for a linux project right you you need to learn more so since you've been dealing with this area for a long time can you summarize what is a good way to for somebody who is open-source developer who is enthusiastic about that but does not much biology right what's a good way to learn more about the space about the problem and maybe kind of then kind of understand what algorithms need improvement how they can happen with the whole area sure sure that's that's a great question and you know I know that that's something that I found kind of difficult when I came into the field myself you know so we've been trying to do some work in the you know in the atom project try to reach out do more more developer facing documentation you know one of my colleagues we lost our syndrome cut from cloudera had a chapter in a recent O'Reilly book on you know how to use how do you spark to do data science he had a chapter talking about you know basic introduction to genomics and showing how this maps into a spark based pipeline my colleague Massey has a similar similar chapter coming out in a Hadoop focused book mm-hmm so we're you know we're working hard to try to put together some examples there we're trying to put together a tutorial article as well you know otherwise I think you know part of the part of the problem can just start with understanding where the data is coming from and what some of the big challenges are um you know it's kind of daunting the literature in the field is you know its vast and wide but one of the things that is good in in biology is that we do have a very large you know a large collection of review articles these are articles that try to give more background you know on specific topic specific analysis needs that try to give the history of the problem and current approaches to solve it mm-hmm so there's a variety of good articles there that that are you know great for people are coming up in the field the other thing and I think one of the you know one of the areas that's very good and actually you know an area that I kind of eased myself in is that I you know even if you're a computational person who doesn't yet understand the algorithms you know a lot of you a lot of how you can get started maybe around optimizing some of the performance you know for for a lot of us you know if you come from the data engineering side you know you're used to running big Hadoop pipelines you understand you know how to how to tweak the performance and optimize the performance of these large long-running you know analytics jobs and we've actually had several people who've gotten into the project by saying hey you know I kind of understand what this algorithm is doing kind of vaguely I'm gonna measure its performance try to make it a bit faster and then you know three months later they wind up you know really understanding how everything works in the system what some of these important algorithms are what the data types look like mm-hmm um so you know it's a bit more computationally friendly you know it's it's a type of thing you put your computer hat on you start to make some you know modifications the code base to see if you can improve performance and just by doing that you know you can understand a lot about what's going on in the system no so you're basically start kind of no way using your skills in optimization and then can expand more into into the domain space and actual brings me to the question what got interested you in like why you you know got into this field and you know kind of what kind of people I would say by generalization can be interested in in the biology problems sure sure well you know the the big thing for me that that got me excited about it is just it's this opportunity to really apply computational computational technologies to something that's a greater social good mm-hmm um you know that's that's what something that really got me inspired about it I you know I was looking for Dana get really start playing around more with a big data stack came across this application domain dived in and haven't regretted it since I you know I think one of the great things about about this kind of quantitative area of biology is that it's it's somewhere where a wide variety of people can come in and start working on problems you know if you're if you're a strong developer you know we always need people who can go through and put together these algorithms get them running fast processing large amounts of data if you're someone who has more of an analytical mind you know more of an algorithms backgrounds a lot of these algorithms that we're using in genomics are fundamentally you know they're based on really core theoretical computer science you know maybe they're based on graph theoretical algorithms you know you're doing some sort of string edit distance comparison mm-hmm so that's another area and then fundamentally a lot of what we're doing is some some form of statistical inference yes so you know we have this huge community in the bay area that's developed this huge data science community a lot of that is some form of you know statistical inference machine learning and a lot of those skills naturally map very well to the problems that people are tackling and biodata jr. of course that's like all this things together exactly exactly interesting so you know I'll probably rip out by kind of hypothesizing um so let's see you know personal medicine offices is gaining momentum let's say magically you have a thousand developers your disposal heretical building of top-level skull and spiral developers who are you know eager to help right and through the community effort effectively you have you know part-time have some of the spare time instead of building another pet store a little bit rails they will help with Adam right and hopefully we'll get enough tutorials enough resources to explain them basically the space right so so in you can put them on any tasks in this pipeline right so what would you do this force where would you direct the effort where you call the attention you know what should they do sure that's that's a really interesting question and one you know one that I've been thinking about a lot recently and you know what's kind of interesting is one of the one of the big problems in genomics is a problem that anybody who works in a organization that has a big data lake or you know big data silos can can relate to and it's dealing with discovering large data sets and figuring out how to aggregate across those data sets well you know a lot of the algorithms that that we've been working on the first two years of Adam have been focused on some of these initial you know initial discoveries that you want to make you know you want to detect oh you know how is your genome and my genome difference right so we've spent a lot of time in the last few years working on that but in the future what I'd like to see you no more focus on is saying okay well you know now that I understand why you know how your genome and my genome are different how does that influence you know how the biology of our bodies works differently you know why does that influence some of my personal disease risk factors how does it change what traits the two of us have and all of this fundamentally is a statistical process that depends on having very large data sets mmm um you know like especially if you look there some studies that have looked at for example schizophrenia autism Alzheimer's disease there you know if you're able to go ahead and and look at you know not ten people's genomes not a hundred people's genomes but ten thousand people's genomes and have a much better understanding of what genomic links there are between that disease yo essentially what links there are between that disease and the genomics mm-hmm but you know one of the problems right now is even though you know even though we can build these algorithms to do these sorts of aggregations it's actually very difficult to manage the data sharing you know and a lot of this is for good reason you know fundamentally whenever you're collecting information about someone's genome or someone's you know someone's health status someone's familial health there's a lot of important personally identifiable information there yes so we see a lot of challenges coming up and you know kind of one in the area of you know if I'm a if I'm a doctor if I'm some clinician how can I discover who else has a cohort of Alzheimer's patients who else has looked at people with myeloid you know acute myeloid leukemia mm-hmm but then once they gone ahead and done that and that's fundamentally a metadata search problem it's a search and indexing problem something that as computational people have thought a very long time about it gets into this question of how can we do analytics in a way that you know manages these privacy concerns right and I and I think that's a it's an interesting problem I don't think it's been looked at a ton in the analytic space but I think it's a very natural extension on top of what people have doing there have been doing in the analytic space yes I think the privacy preserve and data mining I think it's a whole field right obviously you know with the volume of data tool you need to improve specialist algorithms get better matter that reminds me a problem we'll add one more question to this so you mentioned basically you know there are these tasks and then we'll multiple bodies obviously you know privacy is government or regulation but if the database and maintained by consortium so and for those who are not on the field can you maybe briefly outline and what are the other force is what are the key groups and we can supply the links to this video all right but you know what are the key bodies in the accademia what are the key body is he kind of big data genomics that you think it's useful for for folks to follow to learn more about Brad because there is a lot of Sanders where this knowledge has been kept a lot of different organizations and companies which basically want to solve these problems sure sure well you know there's a there's a variety of variety of groups out there you know there's there's obviously our our kind of large open source group yes there's also a similar open source effort that's coming out of the genomics England project this is a project to go ahead and sequence a hundred thousand people in the UK they're developing a similar open open source stack called open open sea be so that's that's another resource for people to look at you know there's a variety of other places you know there's it's kind of hard to hard to say a single place but there's a lot of interesting a lot of interesting discussion on the privacy and data sharing side coming out of the Global Alliance for genomics and health yes that's that's a big one and then you know the one of the wonderful things with with the field of genomics is that there's so many so many great people doing such valuable work that it's kind it's kind of hard to point to little centers of excellence yeah but you know I know there's a there's a a burgeoning community that's really kind of coming up in the in the open source and scalable tools for genomics world so I think you'll start to see more projects starting to starting to come up to speed over the next year okay so looking forward and for you guys will have more resources that the new committee side called open genomics at i/o which were putting together to aggregate this resource will have more links and and will definitely follow and playoffs progress and you know hope to have you know Frank's research link from their past present and future and looking forward to your talk today great well thank you for having me thanks right