data.bythebay.io: Gopal Ramachandran, Protecting data scientists in healthcare with type safety
maybe the place to start is to tell you guys a little bit more about wellframe so we are a Boston based Venture back digital Health startup focusing primarily on Care Management and for anybody not really familiar with Care Management Care Management is our attempt to try and take care of complex Pat patents when they're not in the hospital see if this thing wants to so this is our goal we want to bring the highest standard of evidence-based care to all of those in need and I want to drill down on a couple parts of this the first is the evidence-based piece so whenever a patient is in the hospital a lot of the decisions that are being made on their behalf are being made according to guidelines and those guidelines are based on clinical research studies which look at a specific clinical scenario and figure out the optimal action and then measure the sort of short medium and long-term outcomes that you can expect after of taking that that action this is very helpful until the person leaves the hospital and then they no longer have these guidelines to help them navigate to making Health decisions the other part of this is all of those in need so you might have seen some of the talks earlier today from our friends at lra and a lot of those approaches really focus on depth over breath I think here we're trying to reach the widest possible audience we can we have everything from 18year olds with severe and persistent mental illness all the way to 85y olds with congestive heart failure and pretty much everything in between so how do we do this well the first thing we do is we take those guidelines and we try and interpolate them to give us day-by-day little tidbits and distill them down to what we think are kind of the atomic elements then we take those and we give them to our users and then we can combine them across different diseases diseases diabetes hypertension to give them a relatively custom approach but at the same time give them those granular day-by-day instructions they need this is in the form of a task list which they can get on their smartphone this could be medication reminders physical activity goals educational content about their disease then the other piece of this is the clinician side so we provide a dashboard for our clinical staff which can be doctors nurses care managers and this allows them to see all of the data that's coming off the phone so that they can keep up up to date on what's going on with their patients as well as message them through a secure hippoc compliant messaging platform this is what Care Management looks like today so if you're a complex patient and you're not in the hospital chances are you'll hear from your care manager about once every 3 months this is really just not efficient enough we have too many people and too few clinicians so how do we make this more efficient well what we want to do is we want to take those periods those three-month windows and we want to make sure because we have relatively little visibility into what's going on during those windows we want to be able to see everything that's happening so we're starting to get data off the phone we also can have them messaging back and forth but really the key to making a care manager efficient is just getting every piece of data we can we would ingest that and we would make sense of it and then that would allow us to help the care manager triage their patients this from our just messaging alone and with a little bit of use on the phone we can start to get sort of a 3 Improvement but that's nowhere near what we need to actually cover all the patients that require these services so with that in mind we started saying how do we get more and more data to leverage to use against this so we started in the kind of obvious place which is let's just get the data that users are already putting into their phones we can get message data we can see what tasks they're completing and we can also get a sense of which activities they're performing on the phone actually lead to better outcomes on their part so then we we wanted to extend this and start to see where are they spending time in the app are they really reading the content they're we're that we're giving them who's doing messaging all the little details to kind of tell us what their experience looks like we did the same thing with the care managers when care managers are about to message a user we want to be able to figure out what is the data that they're looking at that paints that clinical picture that's going to allow them to do an Outreach in a meaningful way and of course we need to do things like API monitoring and you know just sort of standard stuff but all of this right now really doesn't help us make the care manager that much more efficient because most of this isn't clinical information this is just essentially refining the product improving the experience improving stability around the API and like our backend performance so then we thought okay what we need to do is we need to plug into the clinical information to figure out what the Healthcare System already knows about these people what happens to them while they're in the hospital what does our primary care provider know so we started to try and plug in and integrate there well we always joke about this in the office that if you want to break into the EMR industry it's only going to take you a few decades to get your MVP up and going and we got a little taste of that when we did this because there were so many different data types that we had to manage to just start to get that 8020 of clinical information we needed for Downstream decision- making you know the first thing we tried was Pharmacy integration a lot of our patients have 20 plus meds we don't want them to have to enter them manually because that's super prone if we can ingest all of that for them hopefully we can increase medication compliance substantially start figuring out scheduling and all of these things and we realized how owner onerous this was the first time around but then we started trying to do this more broadly and then things kind of went haywire so it's worth taking a step back I can sort of give my biases because when I was looking at this I was coming from a genomics background and you know genomics in retrospect is a very classic data science problem it's basically everything is in batch there's no concept of streaming your data set is just sitting there ETL is pretty straightforward you just take that whole thing drop it in a big machine or your cluster and you're pretty much done and the data is super homogeneous there isn't really a whole lot to manage so I mean obviously the first thing we all think of is scale but I knew from previous experience we were nowhere near the scale that you would need to be at to actually start breaking just because of load what was pretty apparent was the number of data types we had to manage and just the amount of bookkeeping we had to do was causing a lot more problems and this is something that we hadn't really thought through and the last part is genomics at the end of the day is a scientific Endeavor if you have to make simplifying assumptions about your data types you can just write off some of it and say I don't know what's going on there it's noise it's just corrupted data Healthcare we can't do that because you know this is all about clinical trust and we're making decisions that affect people's lives so we need to dot every last i and cross every last T and make sure that our data is exactly what we think it is so the big question was kind of how to handle this well the obvious thing to do is we're starting to get more and more data let's just let our data scientists loose and that was kind of where we started and that was with the understanding that at some point they're going to hit a a scale bottleneck and then we'll get the infrastructure guys involved and I think that this kind of division is something that we're all used to data science lives in one place it's one group of people infrastructure is a to separate one and this divide becomes very very important so today I'm going to be talking about basically infrastructure and upwards um we actually have a lot of work that we've done infrastructure and downwards all the way to the cluster level uh Jeff Warren who's our lead infrastructure engineer is in the audience today so if you have any questions please feel free to come and grab us afterwards he's also writing a series of blog posts for mesosphere on our experiences with mesos and dcos I think it's also important that also in r Ros spect looking back on this I realized that this division actually works really well when scale is your only bottleneck right you let your infrastructure engineers build out a platform data scientists just use it they never really need to talk to each other and things work beautifully so where does data science start well data scientists they're great at finding needles and hay stacks until you hit that in for a bottleneck they're pretty good just on their own and they're really fast if your ingestion works well and you can get things into a data frame they're pretty much golden after that and you're they're just going to start turning out a steady stream of insights if your ingestion works I think it's worth you know data frames have very quickly in the last few years become sort of the common currency in data science and we all know that they're very performant I remember the first time I saw the spark numbers where they showed that data frames were much much faster than rdds and in a language agnostic way and that was pretty convincing and they're obviously super super convenient if you're doing data science and a data science workflow the issue is they're not type safe right and I'll come back to this a little a little later and I think that this speaks to kind of where we see data science fitting into the larger organization so how do we do how did we start with data science at wellframe so we're doing everything with python Panda data frames which is a pretty standard setup we're doing everything in a one-off way way so we get a new data source like app usage data and then we just start doing writing a lot of scripts that basically ingest that and this is heavily manual so now when we get the next set of data be it say a dashboard usage we're going to reinvent the exact same type of pip pipeline the bigger question here is kind of like what was our goal with this what are we hoping to get out of it now this is where I kind of want to take a step back and say our goal is to make those care man manager is more efficient right but at the end of the day what data scientists are producing is they're producing insights and those insights could be reports and graphs they could be academic papers but at the end of the day we need the the data to be and the insights that they're gleaning not to just inform features but to actually be the features what very the other problem we quickly became aware of was the diversity issue we were seeing too many data types this was all being cleaned and QA manually and this was slowing us down our data scientists were spending all of their time just trying to get the ingestion working and they were getting totally drowned in that and they were spending they weren't having any time left to focus on finding the insights instead of finding the needle and the Hast stack they were spending the whole day just moving the Hast stack around so this is when the infrastructure guys got involved and the first thing they did was they went and looked at all those things that were being repeated by the data scientists over every different piece of the pipeline and they quickly librari that away this became orders of magnitude more efficient both in terms of developer time and in terms of performance the other bonus that this yielded was this was being done in Scala rather than python so we take in a bunch of raw data it goes through this library and outome a bunch of Scala case classes which is basically One Stop short of having a DSL on which to build very robust features which which kind of leads into the part that I felt was really the most appealing which is okay we know we're having a bit of a bookkeeping nightmare in terms of taking in all the data and we don't want to have to manage all of that complexity by hand and one thing that isn't built into a lot of data analysis is regression testing so as we move forward we want to make sure that things aren't breaking that we left in our wake one way to do this is to just go and pepper assertions through your code and then hope that that will sort of plug all the leaks but we all kind of know that that's going to break down going forward and this is where the type system really kicks in because the type system will take all the invariant that we care about and will force them continuously going forward so at this point in this week I'm sure this looks pretty familiar so the first thing they did was we put they took Kafka and put it between the data sources and everything Downstream we had some Scola consumers which then yield case classes but the question is where was this leading us so now we could safely ingest all of our data the QA was like Rock Solid we know exactly what we have but the problem is the data scientists don't know Scola so they can't take advantage of these case classes that are coming out of our library and it's a bit of a stretch to ask them to just completely give up on python because python has much better support for data science than Scola does okay so now we've solved one problem but it seems like we've come to an impass and what we've done is we basically ran up against that wall that divides data scientists from infrastructure Engineers right and the question was how do we get around that because we still want to build features and we want to use all of this data for Downstream purposes right we don't want to just ingest it and let it sit there so actually I had this conversation with Alexi when we were talking about this a few months ago and he told me that what Nitro is doing I believe is they're hiring data scientists that either know or are willing to learn Scala when I heard this the first thing that came to mind is where do you find such people that sounds like a unicorn if ever I've heard of one and so if if he was willing to help us find these I would certainly you know be in favor of that if anyone in the audience fits this description please come and see me afterwards in the absence of finding unicorns we had to come up with something different and so for us we started to use spark not as a means to handle load but as an abstraction layer to separate our infrastructure from our data science this had the plus that the data scientists got the data frames they wanted it had the downside of well that type safety that we fought so hard to get an ingestion is now getting thrown away the second we put this thing in a data frame and the one thing we' done in the meantime is like okay we're ingesting all this data what are we going to do with it let's put it in a nicely schematized database and just dump it there until somebody can figure out what to do with it okay so now we take spark we give our data scientists the the python data frames that they want the P spark data frames okay but now where do we go from here because now we we're back to this question of if we really want to build robust features we're going to need the end to end type safety we've Sol the ingestion question but we still haven't gotten to how to build features so I think a lot of this is kind of what we're used to because we're used to data scientists being siloed away from production engineering and what usually happens is a data scientist writes some code they find an insight and then that Insight is delivered to someone like a PM the issue then is the PM takes this to an engineer and then asks the engineer to build it and then the engineer writes some code which is probably pretty similar to what the data scientists use in the first place and then they go and deploy this feature so the obvious question is what is it going to take to Short Circuit this the first thing is more tests data scientists definitely don't write as many tests as production Engineers or infro Engineers okay fine I think we can get over that one ruring code okay you can't script as much you need to start thinking about coding for reliability and maintainability down the road okay fine we can restructure the code but this was the sticking point they were working in Python how are we going to get them to work in Scala or something like something similar this is kind of where we started now admittedly this is not the best representation of Panda's code you're probably going to see a lot more slicing and a lot less chaining if you actually see pandas being used in the wild but the point is that it's not unimaginable that you could get your data scientist to code something that looks like this right and we actually already had because we were asking them to use P spark data frames and so now this is starting to look a little more familiar because it's spark and Spark is basically heavily influenced by the scolar collections Library so then the question becomes why can't we just take this and turn it into a A scolar or spark data set we get all the type safety we want but at the same time we'll get the performance and the semantics that the data scientists are used to from using data frames all they need is a couple little type declarations peppered in there and then they can take advantage of all the ingestion case classes that we produced and just code directly into that code base so this is what it's starting to look like so now we dump everything into storage and then we have our data sets and then our data sets can then pull from that storage and produce features there's one big problem here which is this is all in batch right so what that means is are they going to be pulling our storage all the time and then pre-computing everything I mean that sounds pretty horrible but we had done this with the the knowledge of spark 2 so we knew where Spark 2 was headed and we knew that data sets would be a part of structured streaming so that we could then allow our data scientists to code in what they thought was batch mode but then would transparently work for streaming so this is kind of where we're at now which is this meets all of the guidelines that we want we have end to end type safety we have ingestion that we believe is actually working reliably and our data scientists aren't required aren't reliant on other people to be able to turn these things into production features so before I stop I I want to make one comment which is a lot of this talk today has been about what I would call sort of bi style data analysis right it's very SQL heavy and in terms of what it gives you back is it gives you back a clear picture of what's going on today right it's a way to go through your data and pull things out I think that the obvious other piece of what's going on is prediction machine learning and there are the core currency instead of being SQL is linear algebra right and you know the the thing I'm wondering about this is will this actually work in that setting and I think it's going to require something more and the reason is is because if you think about linear algebra from a typed point of view it's it's essentially opaque to the compiler what's going on in there it's just a tensor of floats right and to be able to get the same level of clarity that we have in a setup like this is going to take something additional and that's kind of the next thing that we're looking at so this has met the first series of requirements we have but sort of going forward towards prediction it's going to require sort of an additional step so I will stop there and take [Applause] questions it's also worth mentioning again if there are any scull engineers who love to do data scientists were hiring so in Boston in Boston Alexi if you have enough of them in California we could be convinced to open up a West Coast office I didn't but I might know something I mean I I was just ask you know how do data scientists kind of feel in this structure um there was some initial resistance especially when we moved from Panda to py spark data frames I think as soon as they got a taste of being able to have very homogeneous access across our data sources they were really happy because what they want is they want one enormous data frame that has everything they want and then they can just go to town but getting that is the hard part for them and I think that spark made it very easy for us to do that and then they were on board now sort of coaxing them towards Scala was a little easier because because of how similar spark and Scala looked but it's still sort of a work in progress I think that right now we've lied away a lot of the complexity so that they can work in something that looks just like pure spark without having to embrace the rest of Scala and hopefully they'll kind of expand out from there do you use ml we do use we're starting to use ml lib more and more but and that's exactly my point about using machine learning and the sort typed point of view so if we want data scientists to be able to turn out features directly we have to be sure that we know exactly what the algorithm is doing from a type point of view right and that's hard to know because we basically take in some clinical information you you know you project it into some space that doesn't necessarily make immediate clinical sense and then you have to map it back to clinical reality right and it's very hard you lose like any notion of what's going on in the middle there and that's what makes the linear algebra part very challenging uh you mentioned some kind of database storage for intermediate results uh I'm curious why not just serialize data frames into par uh I think this is basically a legacy issue so we were making all of these decisions in motion and a lot of it was what we had available at the time so we already had you know we had a lot of investment in MySQL and we had a lot of investment in Cassandra and elastic search so it was just too easy to put it in there right away because you know a few minutes to cook up the schema you want and then all of a sudden you've got a place to put it well thank you the thank [Applause] you