BDSBTB 2015: Martin Odersky, Spark -- the Ultimate Scala Collections

BDSBTB 2015: Martin Odersky, Spark -- the Ultimate Scala Collections

Recording: BDSBTB 2015: Martin Odersky, Spark -- the Ultimate Scala Collections

thank you for the invitation to speak here uh at uh Big Data SCA I'm I'm very happy to come here in particular since essentially big data has been of course uh one of the poster Childs for scalar adoption recently like Mike said and uh I would like to explore a little bit how in particular spark and Scala are connected and how we can help each other and how we can work together to do something good so um Scala of course uh I guess everybody here knows Scala very well so it's a functional language uh it's objectoriented uh statically typed and it interoperates well with uh Java and JavaScript um but I want to essentially look a little bit beyond that and say well that's just essentially the point it is in the languages space so what is really the important invariance in the language that you say what's sort of the philosophy the the spirit behind Scala and I think there are actually two and the first invariant was always uh it should be a scalable language that's actually part of the name even and with scalable here I mean that it helps people essentially grow their own languages uh so languages Scala is not a language that is very opinionated that tells you you have to program exactly this way that's both its strength and some people would say that's a shortcoming not everybody wants a non opinionated language but uh it has to it has led to a lot of very very good things and one of them was Park so uh what what the language has to enable this grow ability is it has very flexible syntax uh flexible types it has user-driven um uh user definable operation high order functions implicits and so on and all these make it very easy to create your own domain specific languages on top of Scala let me just I want to okay so and that design uh had a spectacular success you could say because over the years there have been lots and lots and lots of very interesting domain specific languages on top of scal so uh spark is one of them uh chisel is another language coming out of Berkeley for Hardware design and the latest risk five processor which is a very neat processor has been designed from the ground up with chisel as many as as well as many other things uh there SBT like it to hate it it's a Scala DSL uh AKA is a Scala DSL the testing Frameworks are that uh and so on so a lot of dsls have come up and Spark is one of them so it's a domain M specific language it's implemented in Scala and it's embedded in Scala as a host language so essentially you can see spark as just a library running on top of Scala that gives you uh a new language interacting with both scalar and Spark so if you look at the architecture diagram then here you see you have the uh spark runtime uh that is uh implemented in Scala so it uses the Scala runtime which in turn is implemented on top of the jvm so it uses the Java runtime here and the spark runtime it interacts with both the file system and uh a cluster manager such as yarn or mesos but that's actually not the whole picture so if you look at it in a little bit more detail then you see there's actually a layer on top of spark at least if you consume Spark from Scala that's the picture so there's actually the sca repple the re valal print Loop sitting on top of spark and that Ral print Loop would essentially you would type a query uh uh the uh query would go into the reppel the repper would go to the compiler and say compiler uh please compile this query into Java bite codes which the compiler will oblige to do once that's done essentially you load the class file and that class file then would use the scalar run time and do its magic uh so spark in fact at least the Scala mode of spark is really embedded much deeper than just another Library into the Scala ecosystem it actually reach uses the scalar reppel and the scalar compiler as well now if you look at dsls actually for me there are two major reasons why you might want to do a DSL one reason is you want to you have an idea for essentially a new syntax for a language new operator so things like that and uh Scala has a very flexible way to define this syntax so a lot of people create dsls that way uh the other major reason is to say you want to essentially take the uh functionality of what you compute to a new dimension let's say from a single computer to a whole cluster as as is the case in spark and that you to do that you need to have essentially a non-standard compiler pipeline or execution pipeline not what you get in the standard scalar distribution for me actually I find the second use of dsls much more interesting than the first because the first is just syntax and syntax is actually always pretty pretty shallow whereas to take the idea of computation into a completely new place and do something that people haven't done before is really fascinating so what kind of a DSL is spark well spark is very much in the second category uh you could say if I squint my eyes a little bit and look at spark what is it well it's centered around scalar collections uh it uh essentially provides you immutable data sets called rdds for resilient distributed data sets and they equipped with functional Transformers and what are these Transformers well the most basic ones are things like map reduce flat map filter fold aggregate Union intersection that's sort of the typical things you start out with in spark and interestingly you have the same operators no matter whether you interact the spark from let's say Scala or python they're named all the same and that's very very good because essentially it gives you a common understanding and it makes it easier to switch the language so have you seen these operators before and the answer is well I guess most of you have they are just the standard operators of scalar collections they're exactly the same operators in scalar collections so one way to see SC spark a bit provocative is to say spark is nothing but scalar collections running on a cluster great um so is that the whole truth well not quite there actually several differences I think the two most important ones are that the fundamental nature of the collections in spark and Scala is different at least if you take the default Scala collections uh so we'll see that the spark collections are actually lazy whereas the Scala collections are strict and the other major differen is of course that the operators I've shown you they are some part of spark but spark has actually other very useful operators in particular around pair rdds uh where essentially you see a collection as having some sort sort of key and some sort of data associated with the key and these currently are not part of the standard scalar Collections and one of the outcomes of this talk is to say well maybe they should be U I think that's probably a good a good idea to have these so let's look at the first one first uh spark is lazy scalar collections are strict so generally if you do a collection Library you face a number of design choices first design choice you have are is do you want to be imperative or you want to be functional so one very successful collection framework uh that's imperative is the Java util the Java util collections where essentially the fundamental operations are you have a collection and you can iterate over it so a collection will give you an iterator so you can go through it and then the second fundamental operation is ADD you can build an operation you can build a collection by adding elements uh to it one by one and that's of course an inherently imperative thing you work element wise and the collection gets mutated by the ad um in contrast to that functional uh collection Frameworks have immutable data sets uh as as the fundamental data type and then what they give you is essentially Transformations that transform one collection into another using things like map reduce um filter um and the probably most U popular or widespread uh example of a functional collection framework is scalar collection immutable uh there there are others as well so once you have a functional collection framework you have essentially design choices uh what kind it should be and the first is strict uh that's what we have in Scala second one is lazy which actually seems to be a bit more widespread so C uh is has lazy functional Collections and so has spark and uh the third one would be lifted and that a good example of that is the new Java 8 streams so let's see what the differences are so here I have a simple example where I have some data and I map them with a function f yielding a set or sequence Xs and then I filter this collection and um uh that gives me Ys and then I take 10 and that gives me Zs so uh in a strict collection the map is actually evaluated only once here so I have evalate the map I get the EXs and then I use the EXs in my computation of Ys and Zs whereas in a lazy computation the map is not evaluated at all when I write it essentially all I get is a recipe to say if ever you want to get the result of this collection then here's how you do it and I need to result twice I need to result once for the Ys and once for the Zs so I will actually go back and compute the map twice the advantage of being lazy is that you don't need the inter immediate list excess uh because that's not uh that that never needs to come into existence so that means you don't need to build up these intermediate collection structures and that means that lazy can be faster than strict uh because of that can also be a lot of slower uh a lot slower namely when you have to when you do a lot of needless recomputation of these things so lazy demands that you are smarter about what you recompute okay so uh Let Me Maybe Just Launch into um into Eclipse to show you that um let me just put this up uh that we can see the example um all right is that visible okay okay let me just go back to that one I just want to do step by step so we'll get there okay so here I have an example which is actually due to Eric Meer uh Eric Meer had a very well read acmq uh paper which says essentially uh if you need to be functional you need to be purely functional so almost functional programming doesn't work and uh those were his examples actually I think what he meant to say and we talked later after it later words uh afterwards about it is that you need to be very careful to mix laziness with imperative programming so here you see a simple example we have a list and we have two predicate one says uh the element is less than 30 and the other says it's more than 20 and just to see what goes on each of these tests actually prints out uh What uh what it uh computes so if I go and now apply them let's say with two filter operations then I get what I would expect here so I have the list here then I go through all the elements of the list ask for each one of them are you less than 30 and then for the remaining elements here I ask each one of them are you more than 20 and if I then print the list then uh well nothing more happens because the list is evaluated so let's do um something slightly different uh Scala actually has lazy collections just like SP Mar uh but uh you have to demand them explicitly by uh calling view on a on a collection so if you call the view method on a collection it says from now on I want this collection all operations on this collection to be lazy and it's sticky so if you perform a map on a view you get back a view and so on so if you have the view here then uh you see nothing gets computed I just have a sequence View and if I do a filter I still have a sequence view the f here says there's actually a filter operation as the last operation on The View and still nothing changes if I do uh uh the uh the the second operation so when gets something computed well the first time I actually need the elements to do something else with them so here that would be when I do the print l so here now all the computation happens and you see furthermore it's actually interleaved so the order in which things happen is not immediately apparent from for a programmer so you have less more less more than you have two lesses than a more and and so on well if you think hard about it you probably will figure out why it is in that order but it's not something that is immediately clear to to to people here okay so one problem if your order is not the order of operations is not completely clear to you is once you have effects like side effects you're often in for a surprise so because of course for a side effect it's very important to know when precisely the side effect happens because that effect will interact in time with all the other side effects in your program so just to demonstrate that we do a very simple side effect that's an exception so uh we I go back to the normal program and then here we have uh a a new map on my excess list and it says well I want to essentially do one over element in the list each element in the list well that could give me a division by zero of course so I want to catch that here in this last line and say well if that happens then I just want to return the empty list not uh not uh do do anything there okay so we can run this and if we do um if you then print print the elements then uh well that's what we get so in the normal situation we just get the empty list why because well we indeed had a zero here so one over that zero caused the exception to be to be raised and that the the the catch Clause here gave us the nil so let's see what H what changes if we do a view so we get the same computation and well we still get the view here because we didn't really need the element in that list so we have a view okay and once we start to print whoops we get an arithmetic exception division by zero so even though you thought you caught the exception here you didn't because what happened here is not the actual map computation it was a recipe to give you the map computation the map computation happened here where you didn't probably didn't expect it when you went through the printon statement and that pral statement wasn't guarded by a TR catch so the exception escaped unchecked so these were essentially Eric's examples to say well if you have lazy collections be careful and essentially he argued well the C uh design to make the collections lazy wasn't actually a very good one because all these are essentially traps and pitfalls for C programmers but on the other hand if you if you are in a purely functional setting like spark is then actually being lazy is a very very good idea so that shows you so sort of the tradeoff to say well lazy collections are a very good idea if you're in a purely functional setting they are a more dubious idea if you are if you allow for mutation uh Scala has both uh and and uh I think what we because Scala is over the years becoming more and more functional so that means the mutation plays a less a less important role now I think we should reemphasize this lazy collection view in Scala so let me just get to back to the presentation okay so um what then is the last is is the later uh thing there the lifted thing so lifted is um one uh good example of lifted is what Java 8 does so lifted means that your collection operations actually are not neither strict nor lazy but what your map and filter and things do is essentially really Define a computation a computation only and you have to link that to a to to to a collection in another explicit step so in Java 8 what you would write is you have a colle ction and then essentially you get on the level of recipes which is called streams in Java 8 so you have to turn it explicitly into a stream and then on the streams you do a map uh so another way to see that stream is just another fancy sort of iterator and what the map gives you is another stream and but a stream is not a collection to go back into a collection you have to do a thing like collect uh and here the collect in in Java 8 you have to give the precise type of collections that you want to go back so you have xs. stream. mf. collect the advantage of that is that it's made very clear where you're on the level of recipes that means no reuse uh just just computation and where you're on the level of data the disadvantage is it does add quite a lot of boiler plate in particular for simple tasks and if you're like me then I probably write as many Maps as pluses in my program so these things matter so essentially having needless boilerplate for a map is is annoying it would be like if I told you well no you can't write plus you have to write integer do prepare for addition do add dot uh map back to integer you would say you're crazy no that that's that's way too complicated of course I don't want that but if you write as many Maps as you write pluses then essentially the same argument applies okay so what Scala can learn from spark here uh is uh I think two main things the first thing is I think we really should redo views for scalar collections so views are currently not loved a lot uh which has to do with the fact that uh the it's uh some of the design it has some rough edges in particular some of the views uh force you into become strict serously so uh because uh essentially views one problem with views is that they support the whole collection API and some of the operations uh you can't really do in a lazy way and that means that these views become suddenly strict and people are surprised and they then they don't trust views anymore because they don't know whether they they will be always lazy or not so I think we should actually uh change them and uh there is a scope for changing them because I think because of these problems not many people are using views right now so I I guess we can afford to do some uh breakage to have some backwards compatible breakage uh and design views uh pretty much like they would be designed in spark the other thing we can learn is that uh what's really good is to uh control persistence explicitly so spark for that has the cache operation where you say you have this view the computation and to avoid being uh things being recomputed you have cach or persist which says essentially take a snapshot of what you currently have and from then on going forward take the snapshot but not the computations that led to it and I think that's a very very good idea that we should copy for the base collections in Scala the other thing I think we should do is uh operations like reduced by key key Group by Group by key not only are they more efficient than the main uh than the scalar equivalence which is essentially Group by go into map do a map values or something like that they're also much easier to read and understand so I think that's another obvious good idea we should have essentially a notion of uh what a pair collection is similar to what spark does an implicit conversion from essentially collections over pairs to these pair rdd operations just for essentially collections not just for rdds okay um the second thing uh I uh that I want to talk about in the Scala language is the second invariant and that's it's really Scala is really about the types so most of the uh gray cells that went into uh the Scala designs were about designing the type system the rest is comparatively easy because essentially we were able to profit in the runtime from the jvm so not a lot of work had to go into into the actual runtime and essentially syntax and parsing and these things are always easy so the types um are hard because essentially there's a trade-off between safety and flexibility or ease of use you can make your type system super flexible and super easy to use by just having one type called any or dynamic so then essentially you can do whatever but the system wouldn't be very safe also there are there are type systems that are extremely safe uh up to the point where essentially you can encode total program correctness these are then usually Pro called theorem provs so examples are or Isabel and uh the problem is that they are they I think it's still fair to say that it wouldn't hurt to have a PhD to use them I don't say you are required to have a PhD but it wouldn't hurt to have a PhD to use them uh so they are they are not that super easy to use and of course goal in type system design is to push the envelopes to say well we want to essentially at the same time become safer and uh we want to be easier to use uh and because it's a trade-off you sort of choose where you get the Bulge out here so typical research in type systems lately has chosen to essentially push here in the vertical so essentially that's all the work that led to uh the theor improvers we have now or languages with dependent type systems like agda or Idris they are up here so to say well essentially we assume that people are very comfortable with complex type systems uh let let us put those type systems to use to get something that's safer and uh that that was this trend in safety you could say rust is actually another example great example of that where essentially for rust the challenge was uh can we get a type system that makes memory allocation safe without having essentially the escape hatch of a garbage collector so if you do only have maloc and free can we essentially make a type system that makes this safe and rust largely succeeded with that but you could argue at the price of some uh usability and complexity people reportedly say using rust is actually not so super easy uh but of course for many the trade-off not having to use a garbage collector is worth it so you could say that's a trend in type systems the trend in Scala has been a a little bit different uh in Scala actually we were much more worried about flexibility we were much more concerned about getting a type system that it's easy to use so the main um motivation here was to say well essentially we know hindley Milner and we know these type systems of ML and hascal and languages like that U Can we sort of get something that first interoperates well with Java and second would be appreciated by somebody coming let's say from python or Ruby uh so people who are actually not used to static type systems that was sort of the the the gamble on the challenge that we took and that's why a lot of the the essentially specifics in scalet type systems go more towards the area of flexibility we'd like to move it up uh in the area of safety in particular I'd like to be able uh in the type system to define or to to declare whether something is purely functional or not and that would then also make the views safer then you could say well use views if things are purely functional but once you have effects then it's a bad idea uh but when not there yet so that's essentially future research so um the why why do I talk about the types here well it's because spark as we know is a multilanguage platform several programming languages that supported I think currently roughly in order of popularity python is the most widely spread and scalar Java are and probably in the future there will be others as well so um one uh question to ask is well why would we recommend that somebody uses Scala instead of python with spark because spark has a perfectly good python API it has very good python tools so why Scara so for me I think there are two reasons the first is kind of obvious well Scara is native to spark spark is written in Scara so if you want to embed spark let's say in a bigger system that uses not just essentially the the reppel but essentially uses Spark as a system component then probably have a much easier time using Scala than using python for these things but the second one and I think that also applies to the data scientists is that I do believe that types help a lot so in my own experience which is mostly about scalar collections not about spark but I've argued they are basically the same thing I would never ever consider using collections without the types they're just too helpful to to do without so why is that well functional operations if you have functional transformations of collections then there there are no hidden dependencies there are no side effects no backwards channels where you you don't know what goes on everything is out there uh inputs of these operations are parameters and outputs are results so if you have a statically typed language then everything that goes in and out of a transformation is checked by the type Checker and it turns out that the type Checkers of of languages in these collections are surprisingly good so that means that uh typically what if you have a logic error or just a typo or things like that uh typically it will lead to a type error immediately uh so you don't even have to run the thing you don't even have to figure out what they do the compiler will tell you well this you you gave me this type but I needed this other type so logic errors usually translate into type errors and that means that you can write programs that are literally hundreds of lines with these collections or spark and once all the type errors are fixed the ch Es are very very high that your program will run if you don't have the static types that's of course not true that then essentially you run your program and you then debugging starts and says well why did I get this weird thing so I think in particular when you work with collections with M functional collections types are sort of the poster child that's the poster child of essentially smart use of types where types really really help that's why I think that uh I would also recommend to a data scientist to say look use use a language with a static type system with a precise static type system it really helps you good so um that's essentially the the Spiel for the why house spark and scalar relate now I want to talk a little bit about essentially what Scala can do to help spark further because uh I think spark is a like Mike says a killer app for for for Scala so we should think about essentially how can we get the best out of it by helping each other and uh I have four topics here that's the infrastructure in particular reppel spores fusion and staging so let's look at the infrastructure so here's the uh picture of uh the spark runtime again so uh I've already argued that spark reuses and repurposes a lot of the scalar run time the reppel concurrency abstraction serialization and so on so what we want to make sure is that those standard components work well out of the box and no Forks are needed uh one example for that currently uh spark uses a fork of the scalar repple uh because there was an incompatibility with the versions of jline used so essentially the the spark is bound to one version of jline for other reasons and the scalar reple used another so uh you couldn't you couldn't use one with the other and that's sort of a fairly um trivial engineering problem somebody just has to do it to make essentially this scalar rle work with foreign version of the jline so what typ safe did is to say well we make that a high priority thing and we put in we make that change and hopefully I think we're on track to actually be able to roll back the standard repple into the sky spark distribution and I think that's generally something where I want to appeal to essentially both sides to say well that's really something we should be doing actively because nobody wins by forking we really need to have the essentially the standard components because only by doing that you get the fixes you get you get the improvements down the line so forking is essentially always it might be a good idea for tomorrow or for next month if you're in a deadline but in the long term it's almost never a good idea um the other thing where I I believe spark can help Scala can help spark a lot is in the problem in the area of serialization so one problem here in particular is closures so closures use Java serialization and that can drag a very very large dependency graph so here's an example uh let's say you have a class C and uh it has some field data which is a drags a very very large collectional list or something else and then there's a sum uh field which is just a sum of the data so that's just a simple int and then you do at some later point and you pass to this do later um function a closure which says well I want to print the sum I just want to print that sum so you would expect to say well that shouldn't be a very expensive closure but in fact it is because what happens is that the compiler when it serializes this closure it will serialize a pointer to the enclosing Class C and that pointer transitively uh can refer to this very large data collection and that means that that very large data collection will be serialized as well with the closure so the the graph of this closure could become arbitrarily large and that's of course a problem when those closures have to be shipped over the network to essentially all the worker nodes in your cluster so a very very large closure means uh that that that performance will suffer so the idea here is you can avoid that uh as a programmer already now if you just make explicit water closure captures so a much better way to formulate this uh program here is to write do later like this to say well do later and then you pass a block and you say well let me come let me pull out the sum field here into a local uh variable s and print DS so why is that better because with a smart sufficiently smart compiler the compiler will actually figure out that that closure here doesn't have any dependencies to this class C uh the closure is from here to here right it's just this last line here and sorry it's just this last line here so uh the closure just refers to a local variable and that's just a single int that needs to be serialized so the compiler will figure out the we don't need a pointer to the class C and uh therefore the closure can be very light white okay so that requires a compiler that actually do does this uh is smart enough to know about the dependencies and that's something again that the Scala compiler team is working on to essentially get the compiler out there to to to to have a compiler that that uh avoids these gratuitous references uh there's still the question well how can you be sure because of course the transformation from that one here to that one here is maybe again not super obvious for a lot of programmers so again you want help and say well look this is a dangerous closure a lot might leak from that and there's a a neat trick uh called spores that can uh um give you that so A Spore is simply a macro you call it like essentially a Constructor uh so you say Spore and then of the same thing and all that macro does is it says well the thing inside a Spore will be a closure and uh the only uh uh references in that closure that the closure is allowed to have go must be to variables that are defined in the same sport so you can't have any outside references from a sport a sport is in a sense a a a closed system that can be shipped to some other node without having dragging a dependency graph that has to be shipped as well that's where the name comes from okay so spores I think are a great idea which would help a lot in all circumstances when a code has to be shipped over the wire and Spark is one one of of of the primary examples for that okay second example is Fusion so uh in the at the last spark Summit Jay Ood she had a talk showing that essentially a lot of spark jobs are compute bounds so TP spark jobs are typically compute bound because spark does a very good job in keeping uh data in memory using the rdd obstructions and uh that means that if you want to make sparks spark faster we have to essentially concentrate on the compute portion and one thing we can do here is fusing so lazy collections as we've seen avoid intermediate results but they have still a large overhead because essentially every operation is a separate closure so to do all these maps and filters essentially we go to the closure for each element and we go to the map closure then we go to the filter closure then we call them and then they do something what we would rather want is just a tight Loop that does these operations as Primitives and that's what Fusion can can achieve uh we had some very promising early results in the scalar Blitz collection library that used essentially macros to achieve essentially global fusion and that got speed UPS of I think a factor of 30 to 100 in many examples so that was really very very impressive and we are currently working on a whole program Optimizer that would have Fusion built in so once that is available I think it can actually do a lot for computing intensive jobs uh like spark and the third one is uh looking a little bit further out that's staging so staging is an is an old idea it has uh one of its uh recent applications uh came out of uh joint work by epfl and Stanford uh Stanford part is called Delight where the idea is you have a lot of applications sorry oops you have a bunch of applications and each of these applications would have a DSL u a query DSL spark could be one of them and for each dsls you would have a specific DSL compiler and those compilers would map onto a large range of Hardware the hardware could be very specialized it could be GPU it could even be fbga to really run your job uh much in in in in in the fastest possible ways and all this takes uh is made possible in Scala using staging so what ST aging does is essentially when you run a program or a query then the first thing it does is it doesn't actually run it constructs a representation of itself as a syntax tree or more likely as a data flow graph so it essentially gives you a graph that says well here's that's the way I work and then you have you you you have a runtime compiler that actually runs when your program runs or more precisely when your program starts up and that compiler can construct an optimized version for your program based on the actual data workload so the good thing is that runtime compiler actually sees what sees your data set it knows the size of your data it knows the distribution of your data and it also knows what Target platform you're running on so you say okay you're running on an Intel with Vector computer I do expressive vectorization or essentially you can also say well I want to help help with a GPU the compiler can do that uh so the targets then can be jvm native GPU fpgas and so on and I know that the amplab and data bricks are actually looking at staging as one of the techniques to make the the next version of spark faster good so I want to conclude with saying well how can we work each other that was it so far uh but uh the next thing would be to say well how can Scara and Spark help a good course and uh there has been uh great work by Alexi and others uh to actually set up an initiative uh where we teamed up between the user groups SF Scala SF spark amplab and Driver group to create a community effort around open genomics and the purpose of this community effort is essentially to make an open-source system and uh ecosystem uh to help fight cancer and uh we uh just put together a video that shows you how we proposed to go about that so I I want to conclude the talk with this video [Music] just oh sorry David Patterson is a professor at UC Berkeley and wrote a uped piece in the New York Times it was a call to arms that said if computer scientists can uh attack the problem of cancer then we have almost a moral obligation to do that cancer is this terrible disease we have the skills that could help fight it how can we as moral human beings ignore this the cancer patient in 2015 lives in a paradox uh there's never been more knowledge and more drugs to help that patient but the patients never had access to a smaller fraction when we started the atom project we were looking at a lot of the computational problems in genomics and a lot of them are patterns that map really well to a traditional ETL like setup Adam is an attempt to think up a scalable API for processing genomic data what genomics is is the process by which we take the DNA out of cells DNA and RNA all of the information and we analyze it and convert it into Data so that we can identify what has gone wrong in the cell DNA is 3 billion letters long and so what we do is we fish out that coding information and then we sequence it and the output of that machine is basically text file they're about 100 letters long and so then what has to happen from a big data perspective is that that raw data has to be reassembled into the original sequence what we do here in this laboratory is that we take tumor samples from patients we get a normal cell We compare the genetic information in the tumor cells and then we apply statistics and algorithms in our understanding about cancer biology and what we're looking for are differences in that genetic information we call those driver alterations and uh this has revolutionized Cancer Care over the past 15 years op genomics. is a group of volunteers which wants to help defeat cancer faster through open source Technologies and education for both biologists and computer scientists open genomics is a great way for developers in scullin Spar communities to give back we work on very important problems and commercially successful companies this is a way for you to volunteer and to help a really important problem of humanity we had huge enthusiastic response from open source developers that generally ask how can I help if you really want developers who are expert in their own area to help out um you have to tell them what's out there what the data sets are and give them something that they can play with so you know we we try to make a strong effort in the Big Data genomics project to keep everything open to keep our plans for what we're doing open and make it easy for people who are new to get up to speed and start contributing to the project I find it incredibly exciting to see scalar and Spark to be put to such an extremely important use saving life oops sorry lives it's very rewarding to see this technology already used in medical applications and you know in cases uh with healed patients through the patient basing experience we're building we're giving people direct access to their data and information that will Empower them to seek the care they deserve we're fundamentally changing the way that patients receive Cancer Care you know it's really exciting I mean you feel like you're on the Forefront of of Science and Discovery and you're also helping people which is awesome most people today think that the next big leap forward in cancer is going to be a breakthrough in the future we in a very revolutionary disruptive way believe that the biggest breakthrough the patient needs today is an organizational model that gives them access to the breakthroughs that have already taken place and getting the patients access to those breakthroughs We Believe will be the biggest Leap Forward uh that's taken place yet in the wargan cancer thank you good thank thank you all and I think we have a pause now so we can talk more about this and other things then thank you [Applause]