Cloud Science

scale.bythebay.io: Marius Eriksen, Cloud Science

Recording: scale.bythebay.io: Marius Eriksen, Cloud Science

you [Applause] okay so today I'm gonna speak about something completely different from from my traditional topics so first of all I'm now at a company called Grail what do we do we basically detect cancer early when it can be cured so the idea of said cancer is when they're detected early generally have about in 80% chance of survival after stage two or three that flips and so you have about a 20% chance of survival and so we believe that the key to solving cancer one of the keys to solving cancer is early detection so how do we do that a ton of data and a ton of data processing and so Grails approach is that we sequence cell free DNA from blood at very high depth and so this results in about a terabyte of data for you know a single little tube of blood and then we analyze these data to look for signs of cancers there's a lot of bioinformatics a lot of statistics machine learning and so on and so forth that's the basic gist of how we go about doing this so bioinformatics is an interesting sort of sub discipline and the the general sort of gist of it is it comprises a number of different tools to analyze sequencing data in various ways and so they're sort of classical example is sequence alignment and I'm not going to go into too much detail in fact it's not really needed for this talk but sequence alignment is basically a way to kind of reassemble the sort of shredded-up phonebook in the traditional analogy it's a very computationally expensive thing to do and a lot of bioinformatics is very computationally expensive it's also highly interdisciplinary and so there's a lot of application of advanced data structures and algorithms biology of course statistics mathematics and so on and so forth and so it's actually a really really fun area to work in so how do these bioinformatics sort of workflows tend to work well first of all we get raw data from a set of sequencers so this is where the sort of raw genomic data comes from and typically after that they're aligned to some reference the reference is sort of a reference human genome for example or if you're working in non human genomes whatever you know species that you're dealing with and then you tend to do a lot of different analysis on these aligned data so everything from you know quality filtering to trying to detect patterns in the data to trying to build statistical models based on some baseline data to doing quality control you name it it's all over the place so not only do you so this sort of represents what you might do for a single sample and once you have this sort of baseline data you might want to say apply some some set of models to it so for example in our case maybe we want to try to classify whether or not some genetic variant belongs to a solid tumor or not right and in other applications you might want to determine whether or not a particular genomic variant represents a biological variant or a technical artifact things like that and so there's a number of different kinds of analysis that you want to do on these on these data and then finally once you have you know a lots and lots and lots of samples where you implied these different kinds of models you might want to build classifiers so in our case you want to be able to build classifiers that are able to basically determine whether or not a particular sample may or may not have evidence of cancer and you can imagine in other research there are very similar kinds of problems so the sort of typical computing infrastructure that's employed and bioinformatics comes from this sort of academic high performance computing tradition and the basic gist of it is that you go to a vendor like EMC and Supermicro and you basically fork them a boatload of for cover a boatload of money and they give you a set of expensive compute nodes and you share data over some shared file system like NFS and then you might employ some sort of work queue system like sun grid engine to schedule work across that cluster right and as you can probably tell that's not a very sort of cloud friendly way of doing things and it has many other drawbacks as well for example you know it's very very difficult to expand capacity on demand very little is assumed about the environment of the computation and so what tends what ends up typically happening is that the various pieces of software depends on certain artifacts of the environment like what Python version is installed across the cluster or whatever anyway it's a thoroughly kind of outdated approach to to computing and so so our goal is to kind of bring this into the modern age the the other thing is that usually there's you usually you apply some sort of workflow framework to manage your computation on top of this sort of cluster engine for example there's a couple there's a cottage industry of these and so one one example is Apache airflow you might have heard of but there's there's a number of others like Luigi and things like that most of these are very low-level they require you to construct an explicit graph that declare data dependencies between different nodes in the graph each node represents a a computation to do and they don't assume anything about a data model or a computation model or anything it's sort of purely here's a graph you go execute it and what we found is that first of all they're very cumbersome to to deal with in practice especially when your workflows become fairly complicated as I just showed you they're very difficult to compose because you tend to kind of bacon assumptions and different modules it's hard to modularize in other words and I think most severely it in in Butler Lampson swards its highest he has the implementer right it doesn't give a lot of leverage to somebody providing infrastructure to do interesting things and you'll see what kinds of things we might do if we had that freedom so here's an example from Apache airflow which is a popular sort of work queue system and in this particular example you can't see the code that's sort of on purpose I guess because there was too much of it but really this is just invoking the date command and bash and sleep or something like that tying them together and and executing them and you know this is all yep this is what you have to do in order to do that which seems it seems a little excessive to me also these systems are only involved in the sort of orchestration again they might be able to run certain commands locally but if you want to distribute your work or paralyze it across multiple machines again they make no assumptions about your data model right then make no assumptions about your underlying cluster execution and so you kind of have to fill that in yourself and of course people have provided plug in some things like that to make it make that simpler but it's a lot of moving pieces and and what we saw it was something that's sort of closer to what we now call service computing right where this infrastructure is something that you don't necessarily have to provide yourself so this is where reflow comes in and this is our system and kind of our answer to to tackle these kinds of problems so the first basic idea is that we want to get rid of this notion of workflow altogether right it's sort of an awkward thing to do like what you're really doing is just computing programs and we're used to having search additional tools and programming that we can apply like you know function application list comprehensions data structures things like that right and if you can just a you know if you can just program these these these these things directly it makes it all the simpler and there's a number of different features that you can apply once you sort of control your own language that makes it very easy to reason about and gives the runway decided the runtime a lot of leverage furthermore once you control your own language you can also have a type checker which is very helpful in these cases because sometimes they run for hours if not days at a time and you want to be able to understand the avoidable errors before they happen right so another thing that reflow does is that again I've been kind of harking on this but it defines a model and that data model is one of referential transparency you know I'll get to what that means precisely in a second and the result of that is that it gives the runtime itself a lot of leverage to do very interesting things as we'll also see and then finally what we do so now we have a language we have a sort of runtime or a way to interpret programs in that language and now what what we want to do is to combine it into sort of single vertically integrated system that it's just sort of transparent and so you want to have the experience of say running a Python script it's just that instead of running it on your local laptop it's able to say run it on a cluster and easy to instantiate resources as they're needed deal with data movement and so on and so forth right and this is sort of what I mean by by server less or cloud native so what were our our goals here well first of all in terms of the the domain-specific language and I'll show you lots of examples in just a little bed we wanted a sort of simple statically typed functional language and and we chose a goal like syntax simple because that's what we were familiar with and the the goal which is you know somewhat tenuous is that we wanted to make sure it had just enough power for this kind of workflow computing but not enough to really be a general-purpose programming language right and that's of course a hard balance to strike but I think we managed to do it fairly well we wanted to be able to express things like compound data structures and compositions so you know think familiar things like structs and lists and maps and so on and so forth again everything in reflow is referentially transparent and also lazily evaluated and this means that we can again write straightforward programs that evaluates in a way that doesn't perform unnecessary work we also wanted a module system so that we could actually write testable Documenta Bilal's for each of the different parts of these workflows and then also include a documentation system so in terms of the runtime what we what we sought was the ability to sort of interpret the program's directly on the cloud and so when you run a reflow program if it needs more resources it should itself instantiate you know for example nodes and easy to bootstrap them and and transport the data and encode or whatever else is needed kind of transparently the tools themselves that we invoke inside a reflow so for example the aligner and the bioinformatics example I showed earlier should all be distributed by docker images this gives us a way of packaging up and about an environment together with binaries and then we wanted to make sure we could cache all costly reductions and so what this means is basically everything in reflow gets memorized ok and that means that if you run it again and something's already been computed it's basically no op right and this allows some very interesting properties as we'll see in a second we also wanted to make sure that we could distribute evaluation and paralyze it beyond a single node and so reflow the runtime has a work-stealing set up that allows it to do that and so if you write a program that could use you know the parallelism of hundreds of different ec2 instances it will be able to to actually harness that computation power and then in terms of usability it should really just authenticate the user once and bootstrap everything from there and so from a usability point of view it really should act like you know I'm I'm running a programming language interpreter it just so happens to be able to execute this thing on a cloud also we wanted to make sure that it was portable so we could support multiple cloud providers but also an on-premise computer and cluster as well as running things locally on your laptop in our case if you have a Mac we'd use docker for Mac if Linux you know use docker there and so on and so forth so let's show you a little I'm first going to show a little demo and I'll explain things as we see them and then I'll go into a little bit more detail about some of the aspects of what you're seeing so first of all we start off where you know I have some AWS access keys in my environment just a standard standard where you would do things you you install it and now we're going to write our sort of hello world reflow program I I guess I made a mess I changed my mind you know what I'm gonna abstract something so I'm gonna declare a function called hello you pass a greeting and what this function is going to do is that it's going to execute inside of the image of bun 2 while reserving 10 megabytes of memory and one CPU placing its output and a file called out this you know I could echo hello greeting and then redirect that to the output file and so you can see reflow kind of seamlessly integrates the environment of execution inside of a docker container with the interpreter environment around it the main will simply evaluate that now we're going to run it so what does this do well first of all it'll now determine that actually I don't have any ec2 instances standing by some launching one it'll find the instance type that is most suitable for my work and it'll even go out on the spot market and make a bid to make to allocate spot instances it now ran this and inside of the Abunda image again and then returned you know the the hash of the produced file in this case right and so that's how simple it was and I now with a few lines of code and literally just typing a reflow run instantiate a cluster uneasy to run the thing there and gotten the results back and this is obviously a very very simple example I'll show you in a second the kinds are more sophisticated things you can do with this language and so with that I'm going to go through a little bit of a code walkthrough and so one of the most common sort of bioinformatics tasks is to again do this sort of sequence alignment where you have raw data coming out of sequencers and you want to align that against a reference it's a needle in a haystack problem I have you know a billion different needles and I have a well-defined haystack and I want to place them precisely so I also want to show that as I mentioned reflow has a documentation system so in this case I have a module called a line if I call reflow doc on a line it'll show me the module parameters its declarations and so forth this the documentation documentation that's the query and what you'll see here is that we're dealing with sort of ordinary programming things right we have functions and you know integers and strings and you know what have you there's nothing really weird about it right it's just straightforward ordinary programming so let's look a little bit of what this align module actually does so first of all it takes a number of different parameters right so there are some metadata parameters that are used to just embed you know metadata in the aligned output it takes an alignment genome the alignment genome is the reference that I talked about it defines how many threads we want to use for a single instantiation of this alignment and so on and so forth and again I'm just using ordinary programming constructs so for example here I'm defining a function that takes two files and so this is a sort of technicality but reads from Illumina sequencers come in pairs so it takes two files and then what flow cell and Lane those files came from flow cell and Lane is sort of where and the sequencing flow cell particular read came from and in order to line that I'm going to execute this BW a memo liner inside of whatever BW image I have reserving 16 cores of CPU and 12 gigabytes of memory I'm going to place the output in a file called Sam right and you'll see that I'm doing some interesting things here so first of all I'm referring to these files directly inside of what's called an exit care so this is the code that runs inside of the docker container here and when I refer to a file directly inside of my next SEC when that actually runs its materialized as an on disk file with an effectively random path so you can't really rely on it but you know it's there if I were a directory like a lineman genome as its materialized as a directory and I can refer to things inside of that directory I can also interpolate you know even arbitrary expressions so in this case I'm saying I'm calling this read group function which just emits a string that contains sort of the correctly formatted metadata this aligner expects okay and so when so that's the first thing I do and then once I have this output I'm going to call this tool called Sam tools which converts what's called the Sam format into a BAM formats is basically a binary version of the same format and then finally I'm going to use a BAM module to sort the align data for me right and again like I'm just writing straightforward code like there's nothing really special about that so not things start to get pretty interesting so so I have this this underscore or lowercase align function so it's sort of a the workhorse of this module I mean abstracting that by this uppercase align functions and so also follow and go identifiers that begin and uppercase are exported other ones are not right and so I can call a line this upper case a line and it has a sort of understandable signature so given this read pair give me a bam file back and if the split fast Q option is set on the module it's actually going to split the input file into multiple different chunks right and sometimes thousands of different chunks and align each of those chunks individually right and then merge them all together now under the hood this does something very interesting because reflow has data flow semantics it's actually able to paralyze all those things that at a given time and so for you know really large alignments I can split this into a thousand different pieces and reflow will allocate you know a few hundred machines to work on this for you in parallel right and as a user you don't see that at all right it's just you again writing a straightforward program it's all so that gives you kind of a flavor for the kinds of things that you can do inside a reflow I have mentioned that there's a module system so there's a line module is you know basically module that I use inside in this case inside of a sort of top-level module where I'm you know fetching some some data from a public data set aligning it and doing some other things right and so these things composed in the expected ways you can write unit tests for them you know what happen right so so as I mentioned the evaluation is basically always paralyzed as long as there are no data dependencies between two nodes of the evaluation and so what that means again is that inside a reflow you're just writing straightforward code it looks like it's serial in any other context but it's fact can be paralyzed underneath the the hood all the computations again because reflow controls that concrete syntax and understands the precise dependencies between all the computations it can cache everything that it computes and retrieve that from cache if it's been computed before and so for example if I have an align if I've done an alignment and then do some computation in that alignment and I do in different computation on the same alignment reflow will simply just retrieve the cache results and it'll go much faster reflow itself is lazily evaluated meaning that something is computed only if it's actually needed inside of an execution at any one point and what this means is that you have I would say this enhances the ability to reason about your code and also compose abilities for example if I have this alignment situation for example and I pass in this reference that might be really expensive to compute for some reason maybe I'm computing the reference from scratch instead of using when that's been computed for me well by passing that in I'm not actually computing it right it's only being used if it gets used at the end of the day and so sort of one example where where that helps helps out a lot and the net effect of this is that the combination of the sort of lazy evaluation together with memoization together with dataflow semantics is that reflow is fully incremental and so if you compute something reflow is guaranteed to compute the smallest difference that is required you update your results from what's previously been computed and this is all sorts of interesting benefits that's we'll get into in a second so here's an example you're sort of demonstrating the value of incremental computation and again I'm abstracting the actual code but let's just say that everything here is expensive to compute so it's sort of standard you know data processing pipeline might have a cleanup step some sort of analysis step it might merge multiple analyses together and that's what this represents so in this case I might have a set of samples in our case it might be a set of different you know sequencing samples that we have and I want to perform the clean-up on all the samples I want to analyze them all and then finally I want to merge all the analyzed samples into some you know some common data structure that allows me to for example to do some sort of population analysis so when incremental evaluation gives us is very interesting so for example let's say I have this setup and I changed my analysis well reflow will recompute all the analysis for all these samples and then the merge right but not the cleanup if I add a sample to my population which is very common it'll compute to clean up and analyze step just for that added sample and then we merge everything right and and so on and so forth so you can kind of see where this goes right it's able to kind of compute the smallest thing that it needs to compute in order to give you what it is that you asked for and that's a very very useful property of these sorted systems so in terms of runtime so reflow does not actually depend on any sort of external components except access to sees to say the ec2 API it does not use an external custom Ana juror or anything like that it computes directly on on the cloud if you will in the examples that I showed you the evaluator itself you know in this case just ran on my laptop right and again it was able to instantiate workers perhaps across multiple ec2 nodes and make resource allocations inside each of these workers it maintains a cache in this case an s3 right but that's an implementation detail that's hidden behind a fairly simple interface that can be implemented in multiple different ways and so we have an s3 implementation but we also have a file system implementation if you know a few others and that's sort of the the gist of that the interesting thing is that while it seems like this is implementing a lot it's effectively sort of implementing parts of a cluster scheduler like kubernetes the evaluation semantics of reflow allows for a pretty drastic amount of simplification to that which would take great advantage of and I'll get into that a little bit as well so I mentioned that reflow paralyzes work through a work-stealing setup and the way that works is that I have a sort of primary resource allocation where I try to do most of my work but if this runs out of capacity in other words if I'm queuing work then there's a work Co that will go and allocate more nodes and [Music] assign them tasks basically and the way that works is that the the the stealer will transfer all the dependent objects again because reflow attracts all the dependencies precisely it knows exactly which data are required to compute any given step it transfers those dependent objects it computes the work and then transfers the result back and this is all done sort of note to note or media - by s3 depending on the on the configuration so one really really important thing so I want to mention that for all that reflow does is only around 30,000 lines ago and as you know go is not the most succinct language right and so it's really not that much code but it goes to great lengths to try to simplify its implementation by sort of exploiting the computing semantics so my favorite example of that is that because we have caching and referential transparency fault tolerance is simply just restarting the whole computation because if something fails like a node goes down for example or is reclaimed because it's a it's a spot instance Andes - I can just restock the whole thing you know more or less pick up where I left off right because I can just make use of the previously computed results the evaluator itself is completely stateless because it computes its state again for the same reason I can the the entirety of the semantics of the computation is expressed in the concrete syntax that you saw and there's there's no external way of influencing that and so I can always compute the the internal state and so I need to keep any additional state also we sort of liberally applied the antonin principle so for example the evaluator maintains keep lives to these resource allocations and if they fail they just restarted and so on and so forth and in practice that actually works really really well we end up with a you know simple and robust vertical compute stack that actually does a lot with very little again by kind of exploiting the inherent simplicity of the computing model and not trying to do too much more so again I sort of went over this again a little bit before but one thing that's really really important I think the the sort of main takeaway I think is that this combination of features is lazy evaluation caching and so on and so forth really results in a very simple and sort of computing model that's easy to exploit and the other sort of interesting part about that is that because of these features because reflow is able to the incremental computation and so on and so forth it can also replace a lot of the kind of super structures around scientific computations so here's an example it's often the case that you have a system that deals with like sample management and so it tracks here all the samples hair all the computations that have been running these samples here's the you know where the results of those computations reside and so on and so forth and whenever there's new versions of software out you know you have to go and kind of reconcile that state with with the sort of new desired State well in reflow that's very simple if I just know my set of samples I know exactly what it is that I want to computer which is just my workflow over all those samples and I can express that directly and the state itself is computed and so there's nothing more to do right I don't need these sort of super structures anymore and and and again because of referential transparency the runtime itself is given a wide latitude and everything from cache management to did a movement to you know retrying operations trade-off of computer versus storage costs so you know one thing that we're starting to do for example is to effectively store what the cost of a given object us and then decide you know whether or not that's worth keeping around or we computing because we can always recompute it and so it gives you a lot of interesting freedom in the runtime and again in the world of Butler Lampson we have I think we've successfully untied the hands of the implementer ie me so another sort of interesting thing with this is that we feel gives you perfect reproducibility in the sense that again these reflow programs actually fully describe the computation that you want right and so looking at say versioning a workflow or pipeline as simple as adding a git tag for that version you can always go back to it you can compare the results there are some tools that will tell you sort of what the difference between you know given get tagged s and and and the current master and so on and so forth and so a lot of the computing model also gives you a lot of power to build these kinds of tools around it or use existing tools in this case get to to kind of do the the hard work for you so reflow is now open sourced as of last month it's on you know github comm grille bio reflow and inside of Grail we know how for almost a year I've been using well I would say half a year we've been using reflow for all of our scientific computation all our bioinformatics but also a lot of ad hoc computing and so we have biostatisticians so I've used reflow to compute really really expensive noise models using you know 20,000 cores on AWS no no sweat at all right first for a single computation a lot of exploratory analyses so you just want to test the hypothesis really really quickly it's very simple to do with reflow and I think overall the the kind of original original premises have actually held and the model ends up having you're working really well and especially caching and so one thing that we do a lot in in the kind of computation we do at grill specifically is that we have these pipelines and they tend to change all the time but some parts of them are slower moving happily alignment is one of those which is also one of the more expensive things that we do and so we're able to edit rate really really quickly launch new versions of the pipeline all the time but it ends up being very very cheap to run because it's reflow is able to reuse most of the computation from previous runs right and that's saved us a lot of money in time in computing so that's that's it for my talk I'm happy to take questions I encourage you to check it out it it is truly an open-source project we have release binaries there's a tutorial even and it really is as simple as as I made it out to be in the sense that if you come with some AWS credentials you can run whatever your AWS account allows for you can easily run a thousand node computation in a matter of minutes and so so with that I'm happy to take any questions [Applause] that's people is in this remind me of lucky ideas and basil or blaze a shirt properly I think you're very familiar with and so as long as lines and so you say that's reference to transparent which is how do you manage that without like I mean the sooner have to trick their make being like like side effective things preferentially transparent is of course like London's right simile they can make these objects like that you see do this thing how are you like syntactically like kind of like understanding when these actions are there actually lose it not like an actual file that can be read so that's kind of one question and along the same lines if you look at the difference between so silver pants other builds is to the contract music support of things to is a clinic place it allows for fully dynamic computations and there's a lot of arguments in favor of that basil does not allow for that there's also arguments in favor of that click very it may seem like you can do some query and so I was wondering originally in army kind of like do you allow like fully dynamic grass the county like unwind as we go and like what do you think about like the bezel argument against that and I just so one last thing is I'm pretty sure I'm sure your identity who's been a thousand inferences how do you avoid like spending too much good you'll be giant computation like what about garbage collection what about like just like breaking yes so so so let me go in reverse chronological order so classic Twitter so if for your last question reflected does implement garbage collection and so when a node becomes idle after a certain amount of time that collect terminus itself right and it tries to pack work so that it utilizes smallest number of nodes right it also has file of garbage collection right because it can track files precisely and so once something is uploaded to cache and it's guaranteed not to be needed anymore it can it can ditch it right and so does both garbage trucks and both of data and compute nodes the previous question was dynamic graphs versus static ones or no how do you avoid spinning up a thousand nodes while we have limits built in and so you can at least accidentally do it right in terms of dynamic versus static reflows flow dynamic so you actually saw this with a split fasiq' or the splitting example we don't know how many splits are going to be up at a time right and we can compute those inside out reflow for the kinds of workloads that we do with reflow dynamism is absolutely required it just because there are a lot of things that depend on the data depend on the data sizes depend on a lot of things like that that you want to be able to exploit in terms of a build system I I'm not as well-versed and and in that domain but I could see an argument for making more static simple because the domain is more constrained and compilers tend to behave in fairly static ways like usually you don't have to determine which modules to compile based on the output of some other compilation step right because you have static type checking always differently the first question was yeah let's catch up after yeah yeah first question which sounds like a lot of the complexity was in the caption so something exact so it can be fully kids the cash key can again be fully computer without any state and it's because we control the concrete syntax and we can do enough analysis or without AST right so you can't have another programming language influenced yeah those are those are expressed directly in the program right so for example your doctor image just expressed in the program the input data are expressed when so here's an example right so let's say let's say I cache one of these alignments right the cache group of that alignment comprises the the digests a shot to the sex of all the input files the the resolve you know docker image you know and it can track precisely how to reconstruct that environment right and that's the key that's used to cache it right and so it is precise and in fact one of the one of the problems that we have in practice is that it's overly precise so you know if you change the character in your command right it's gonna invalidate the cache key and so we're now starting to come up with ways to kind of modulate or or give a little bit of control to users in terms of caching as well right but by the fault that's it's precise I'm happy to I'll be you know in the corner over there happy to the chat [Applause]