Text By the Bay 2015: Ola Wiberg, Unlocking Our Health Data: Transforming Unstructured Data at Scale
foreign ERG I'm a co-founder and currently the VP of engineering at the human API so today I'm going to talk to you about unlocking our health data and how to transform or work without data at scale so I'll probably focus a little bit more on the scale part rather than transformation because I'm a data engineer not a scientist so I'm not a mathematician so this will be a little bit more focus on architecture and system design and yeah comparative probably some other talks to go to during this couple of days so human API we started about two years ago we launched the first product in the summer of 2013. um here based out the Redwood City here in California so what we're doing we're building basically unified Health Data platform uh to enable the next generation of Health applications basically by making it really simple for developers to integrate Health Data just working with one API so yeah what problems are we solving so Health Data today exists both in all the new systems there is a lot of Health Data Systems out there hospitals like Legacy systems so there are a number of difficulties working with that data just accessing that data for example is is pretty hard then there are a lot of new systems that are built for health tracking devices for example these systems are a little bit easier to deal with but there is not really any standard across these systems so we're making ideas for developers to to work with this kind of data basically just working with one API so some of the challenges with these old and the new systems so I briefly mentioned all systems are hospitals labs clinics Medicare so it's a lot of unstructured data or CM structured data uh and usually it's it's a lot of data for specific points in time so you go to the doctor they take a lot of tests they their procedures so they they collect a lot of data about you for one point in time and there are also a lot of Standards probably too many standards over the years so it's basically the same as working with no standards in some ways it's too complicated and new systems so today a lot of people carry a fitbit's jawbones Apple watch is coming out what is it this weekend or it will come out in just a couple of days wellnessfx you can go to the lab have your blood test taken and you can have the results sent to you without involving a doctor so those are type of newer systems so this the data that comes out from these systems are generally better structured but everyone has their own structure it's not a standard here as well another thing that's common between new system is is often a stream of data it's like you have data coming or generated throughout the whole day so it's not just these points in time it's it fills in throughout the whole day and as I mentioned no standards and just to briefly touch on the type of standards in health I.T so there are a lot of terminology standards code sets for example so Lauren guesses code set that determines how to name test results or it's a standard across test results and ICD-10 is a World Health Organization classification system just for medical conditions for example and nomenclatures like snomed for medical namings and then there are information exchange standards as well hl7 is the most common so there are many different versions of hl7 so it can be serial communication XML or the the latest one is called fire fhir so if it's for fast Healthcare integration resources so that's a promising standard XML and Json basically it's a it's going to be easy to work with systems if they Implement those kind of standards but we have to deal with a lot of different different types of data coming in and classifying it based on yeah different standards so what type of data when I talk about all the new I basically say the oldest systems that have been around for a long time so it's clinical data medical records vitals recorded by nurse and this data usually comes or could come at XML over https or TCP I mentioned before it's deep data for specific points in time modern data's personal health tracking so there's a lot of things that you do on your own like you track your data at home you track your Fitness you track Vital Signs by having devices at your home your weight your blood pressure blood glucose a lot of this data is available like Json over restful apis so it's not only problems with dealing with this data it's a lot of valuable data so both in older new systems they're really useful once we can make computers actually understand and work with this data so yeah and the new data is filling a lot of gaps from the old Legacy data because it it fills in this this full picture of a person's health profile over time not just specific points in time so we're building a system that enables deep learning from this desperate data sources so how do we do it so as you can understand we have a lot of different data sources coming in both push and pull sources it's many different types that I mentioned and we want to process this data in near real time so basically when the data comes into our system we want to be able to process it as fast as possible many times it can take a long time before we get the data but once we have it we want to be able to process it in their real time and we also want a lot of flexibility and speed of development so developer data scientists can work on testing things working on things without interrupting the flow of production environments so we're building an infrastructure basically with the microservices we containerize them using docker and with we deploy these services using messes and Marathon and we have a data backbone where we push all the data through which is based on Kafka and we use spark to work with the data so I'll go through some of these things in more detail as we go through the presentation here so the foundation for the data infrastructure is messes and Marathon I don't know if you're familiar with these Technologies but it basically it's an abstraction layer on top of the hardware so it's like a cloud operating system pretty much so we have a lot of AWS servers that we pull together and messes manages those resources for us so basically how that is done is we installed message Master we have zookeeper to manage it or manage configuration and then we'll fire up a lot of metal slaves which and massive slaves are the ones that actually run the services and then we use Marathon to manage the services so if you need to start up and scale up a service Marathon triggers a job and starts the service where on a slave that has resources available and when it started we scale the system and it's usually just a few seconds and so if you need to add resources it's really easy with that with this technology and the foundation of the data pipeline is Kafka so basically we have a Kafka cluster we have dockerized apps ever have producers and consumers for the Kafka topics um so basically the apps are listening for data of incoming data or collecting incoming data push it into Kafka and we have different types of consumers working with that data whatever it may be and we also have replay capabilities so if you need to re-parse the data or work on it we can easily trigger just replay based on Kafka because it persists the data there depend a period of time that is configurable but this makes it really flexible to work with with the data so looking at the data pipeline a little bit more detailed so basically we ingest data XML HTML PDF for whatever format it is basically go through this is a very simplified view of the the data pipeline um it's we do some simple organization cleanups and extractions so basic NLP stuff we do normalization and then at the end we have a structured data API so for this demonstration we're going to look a little bit more detail on data normalization specifically so as an example of what we need to do is using classifiers to normalize the test results so test results can be blood tests urine tests whatever tests are done from the doctor these tests are often they can be tight by hand the name or it can be used different kind of those systems that I mentioned earlier for different standards so when the data comes from many different systems it's hard to know how the tests map and what the naming correct naming should be so we are yeah we're predicting the class basically identifying the correct name with standardizing on units and normalizing values across the different tests um and we accomplish most of this by using Apache Spark so why Spark so spark provides a complete tool set and makes it really easy to work work with large data sets across clusters has a great stack for doing steam pro stream processing which fits well into our Kafka pipeline and it also has a lot of built-in tools for to do machine learning um and it deploys these loan methods which we are using for our base infrastructure and it's it's really fast compared to a lot of other tools that are available because you can do it uses rdds resilient distributed data sets across clusters and it also has the native Scala python interactions and it has a shell so it's you can actually work with the data just in command line which is real nice and developer friendly so some basics of Spark It's it works well in the the message setup so you can actually start up workers like I showed earlier where you had Marathon starting services you can have services or spark workers started up like that as well as you need to process the data or transform the data so basically I have a driver that launches the rdds and keeps the spark context and then the workers are actually the ones that are working on the data so they can read the data from the stream from file system and load it into memory and do Transformations on it like filter maps and unions and then the driver can call some actions on that on the workers to collect results do counts and broadcast data as well to those workers so Sparkles as oh sorry the spark also has machine learning library which is more of a framework than the library because it provides basically everything you need to do machine learning at scale it leveraged rdds to actually work with the data across multiple workers so it's really fast if you need to test things on large data sets and it also includes common common algorithms that are specifically optimized for parallel parallel environments so for classification and regression specifically they have libraries for binary classification multi-class classification and regression analysis and I'm going to demo some code here where we actually use a specific logistic progression with lb fgs algorithm so let's look at some code I know does a lot of you work with spark already so quite a few people so I'm just going to show you a real simple example here of Sparks so this is spark using Scala not python so can you see this okay so basically if you're familiar with Scala case class just shows you what if what a clinical test is basically a test name label name and a label value we have a test result normalizer class which basically does everything I'm going to demo here 's a simple mapping it's just 21 values for some the test results we want to normalize so of course the real number of results are in thousands but for this sake of this demonstration it's just 21. so this glucose test hemoglobin level hemoglobin A1C as examples um so the way spark works is just start up a or instantiate the spark context basically just take you some configuration give it a name and create a spark context and here I wrap that context in a SQL context which basically give you for this demonstration is mainly done just to be able to load a Json file with easy so I can load Json data from a file and just perform some simple functions on it so in this case I'm loading the training set so we have a training set here as well as just very few a small sample here with 90 90 lines of tests basically mapping the various test names to a normalized test name so this is generated basically manually at this point because it's just a small small set here so below that um into Data frame the data frame is a class where you can basically it just gives you some convenience methods around working with that data as well so you can load the the Json data into a data frame and then map across it to generate the actual clinical tests so the clinical tests from the training sets we take those and we create like to turn frequency hash based on that so Spar comes with algorithms for that as well so you just instantiate that the class and then do the transformation here with the test names and I return the vector and label values and so a map over it again so in this case just to generate the the label points and I catch that training data in memory and then I can use this algorithm to actually classify the data or create the model science then say that I tell it here it's hard-coded the number of classes as I showed earlier there are 21 classes for this demo I take that and I run this algorithm on the training data and generate a model so that's all I need to do to have a model to work with I can then save that model in this case it's commented out but you can save that model and then reuse it so here we're just gonna run run through but you can actually save this model um use it for streaming purpose or for other purposes at a later point or as in this case you just use it to do the predictions so the prediction is done on the model so just call predict and here at the last stage you actually do the pass in the values from this array so we take all these values we're gonna classify them and see if it works so I think the resolution is a little bit off here just a second can you see that so first let's just build it and then to run it will run this command so basically when you run a spark program just submit it to to the cluster basically so using spark submit and then you tell where the master is in this case it's on localhost and I tell it to use all or four cores on the system so just send that command and it's going to run around the program and basically here here's the prediction results so WBC is classified as white blood cell count white blood count is white blood cell count so it's working so that it pretty much demonstrates how how simple a spark program you can actually create to do some yeah some simple machine learning work and it's a it's a lot of algorithms available and you can also use you don't have to use the algorithms of course that are part of spark you can use other libraries and plug them in and run them on the workers so but it has a lot of things already built in that makes it really simple to get started so another thing if I mentioned it before that we use spark streaming so once you have built this model or trained a model you can plug it into a streaming context and actually use that model on real-time data as is passing through so spark streaming it extends the core API if it can ingest data from Kafka that we are using or other things like just TCP or crmq and they can publish it directly to dashboards or store it to the database whatever you like and it's really simple here's just a quick example of how we would set that up so you basically create a streaming context and use do you tell in this case to capture till to create a stream and once you have a string plate that you can just read the data from the Stream So in here Kafka stream.map we get the name and then you can load the existing model so like in the last demo we were creating a model so you can load that same model from the file system and then do predictions on the stream stream data so you basically have real time processing there so the way it looks like this so you can ingest the data you can stream it so in Kafka you can stream it in multiple topics one that can be consumed by the by your machine learning service and basically build models to train the classifiers and save new models that you can then reuse in the streaming when you stream normalized data basically so here the two streaming Parts here is it's all Kafka the spark takes care of the machine learning all this is running on on mesos basically one one system yeah so what's next yeah this architecture has showed us to be very flexible it's also stable platform that can actually scale I don't know endlessly but it seems like it and we're really early in this stage of building out this platform but it has proven to be really good for productivity for the company so it's it's a good choice and we feel happy about it and their Spark is a very powerful tool so if you haven't looked at it yet I would recommend taking a look at it because it's very capable of a lot of things so that's it for my presentation so if you have any questions so the the library dependencies basically if you create a Docker image that has those dependencies you can just use that when you deploy so it hasn't really been an issue is that what you mean just have to make sure it's there yes but with the dockerized environment where you can actually specify those dependencies but yeah it's the g4train dependency fifth are normal questions so I think that's it oh there's one more how to actually store the data or yeah so your problem most like the store I'm using hdfs the hdfs we use Cassandra as well so but hdfs is a really common one yes yes hdfs and Cassandra are good options okay thanks