BDSBTB 2015: Ola Wilberg, Building a Health Data platform for the Future
Recording: BDSBTB 2015: Ola Wilberg, Building a Health Data platform for the Future
so as I said my name is all away berg i'm the cofounder and vp of engineering at gym in api and today i'll be talking about building a health data platform for the future so it's probably a little bit different from some of the other talks that you've been listening to during this conference so i'll talk more about the infrastructure how do you actually build the infrastructure to support a big data product versus more code specific example set or talks as you have seen earlier today so so a little bit about jumann api what are we trying to solve as a company yes so our mission is to drive availability of health data and medical data and today if you want to build an application using a clinical data for example it's pretty difficult task to actually get the data that you need so we are here trying to solve that problem to basically be the easiest way to get health data from anywhere so currently we have integrated with more than 2,000 data sources we have 10,000 developers on the platform in about 35 different countries as of today so you can view human api basic lessons new type of health data exchange so look at it we have a lot of different sources thousands of different sources of data and that data can come from tracking devices like your your Fitbit jawbone those type of devices that you wear or glucose meters that you use to take to read your blood measurements at home or it can be wireless scale those type of devices or apps that you are tracking so moves for example track your activity your location how many steps you take how much the cycle but it's also a clinical data so if you go to the hospital and your create a medical record we can get that into the system and you can share that with doctors we can also get data from labs so of course this is a lot better than fax machines people still use a lot of fax machines and health data today to transport the transfer health records so sending paper copies around by mail or fax so we try to make that a lot easier by creating a simple to use API to get access to that data for application developers and also of course normalize and standardized that data to make it easy to programmatically work with that data and so we would enable by doing this we enable basically a complete living history of a person's health data so that's the long-term vision of course today we only have it a little bit of data a few years of these type of devices but in the long run you can build a whole history of your health basically on our platform and one thing that we see is really important is that the consumer should be in control of this data so you as a user you are the one hotels both data to get and who to share it with so if you seen a doctor and you need to share that data with a specialist you should have full control and insight into where that data is going versus today you don't really know the data is shared without explicit your explicit consent so about the technology is whatever what does our system do so yeah we process a lot of data and basically there are simple tasks that we need to maintain so basically having on-demand workers to collect process store and present that data so we have many many sources of data coming into the system but we only have one API that serves that data so the challenging for us is that maintain all these collection services and get the data from many different places and processing it into a format that is usable for application developers so how do we design a system to handling this at scale so some of our goals with the system is it should be agile so we want to be able to develop and move fast so have a fast develop build deploy cycle and we want self-contained services so if we deploy something we want everything to be part of that deployment so make it easier to deploy applications and it should be repeatable whenever you deploy that application it should have the same result and also scalable so a lot of these integrations need to scale up and down based on demand batch jobs or or load different kind of load and we also want a language agnostic platform so I know a lot of people here are scholar developers we have Scala Java that we also have no Jas and some Python it can be any language and we're going to have a platform that actually supports and a type of language so how do we build this so the main building block of our system is basically talker microservices so all services in our system run on lightweight containers so some of the advantages are is that it's self-contained I don't know actually have how many of you are using docker ok so this about half of you so some of the advantages there are the self-contained so you can specify all the dependencies that is needed to run that container and you can configure it at Build time all runtime so it's very flexible in that way and it's very quick to scale up new services as you need it or take them down if you don't need them and you can run this like thousands of these taco containers the pretty easy with what we are using so yeah how do you manage to services so we are using methods to run those Tokyo containers so mace assisted technology basically I mean it's the foundation of our infrastructure so you can see that's a cloud operating system an abstraction layer over your hardware on on the cloud basically so it pulls all the resources on on Amazon's we're running on amazon so it pulls all the resources we can just add servers button meses abstract the hardware way and just gives us the CPU and memory that we need to run the sort of the containers and of course it supports talker for container rising and it has great service isolation so you can dedicate specific CPU and memory to the containers so it doesn't use up more than the needs and mrs. system contains of masters servers which are the one that keep track of how much resources are available and it has the mess of slaves which are the the servers are actually run all the containers and has frameworks bits are basically applications that helps you manage those the running containers to running docker containers you so just to visualize that a little bit on this slide you see there are a couple of slaves and the slaves basically report to the master how much CPU we r memory that is available on that on that slave and this small little green dots in the slave those are the doctor containers so they're running in the slave and they publish what they have available to use so four gigabyte of RAM and 33 CPUs free so they published out to the master and the master make that available to the framework so the framework knows what it can use to to publish new tasks pretty much and the frameworks can be a number of different type of applications but what we use is for example for deployment we use something called marathon so let's say you want to deploy service that need one gigabyte of RAM and hat the CPU you tell marathon you need to to run that task so how does that happen so marathon is it's a meses framework that manages to running containers and it basically does things like sets the environmental variables and the science the requested CPU in memory and starts up as many as this as as needed to run that service and and to manage that deployment process we have actually developed a command-line tool we call it happy Co so that is for deployment and automation so basically anything that we have encountered as problematic they have built tools around that just to make this process really simple so we can just type a command to scale up or scale down a service or deploy a new service so let's see a lie it it interacts with marathon also metals and Jenkins which we use to actually build the project and create the docker containers so just to illustrate how that works if we have this the infrastructure running with the missus masters they're controlled by so keep zookeeper and we have a number of slaves here nine slaves and marathon that is the the framework and as you see that there is a service a that is available where it's running on three different slaves so what it does is it try to to run the services on on different slaves just for redundancy purpose and if you notice that we need more resources at this point we actually can just issue a command to marathon and the tell marathon that we need to start up another instance marathon checks the available slaves and see where resources available and it finds that slave five has some resource available to run the service and it tells it to start up a new serve as a on slave ride and now we have a scaled-up that service and this this happens in just the matter of seconds so it's really fast to scale up and down services like this so this is a little bit blurry for this resolution but the marathon comes with a web UI so you can actually control the services through through the web but also of the white box in the corner is shows our command line tool that we have developed and how you can the glory with simple commands and specify CPU memory and number of instances you need so now when everything is running another problem we had to solve with this platform is minimize downtime so the API that we are serving is on AWS we cannot use AWS lbs and some of the requirements we have is of course that we want to be able to deploy often like I mentioned in the beginning so should be quick to deploy and it should also be able to scale out easily horizontally for the public services not just the internal workers so lb is a little bit problematic that way because it's kind of tied to one specific port and mrs. brings up the services with random ports if you start up a service it just assigned support usually random high number port so what we did to solve that is to use the combination of engine X console and quantity so console is basically a key value store and it's used for service discovery and configuration management and comedy is configuration tool that is used to basically update config files and execute commands with a simple simple tasks and mrs. actually publish all the events that is happening on the system to like as a web hook and for example containers coming up and down or published as those events so we are using that for the text layer control or to configure the system so when mrs. creates a new service we run that too on command and start up a new service so as you see in step one here we have V Creighton container it assigns a random address and it published that event so we have an internal web book service that listening to all those events so as soon as we get that event we update console configuration with those service details and conf d is listening for changes in console so as soon as confidence is that changed in console it just quickly reconfigures engine X and then reloads so that service or that containers now available on the public port so that's a little bit about the containers containers and how have a managed uptime and quick reconfiguration or scaling of the system so do how to actually handle the data used by the services so some of our needs is that yeah microservices needs to be able to communicate and to communicate directly with each other it's time consuming and hard to manage and the needs to be general purpose solution and we also want to be able to enable easy data pipelining to work with the data to collect parson persist and should be able to handle large volumes of data and preferably be durable and ability to replay so I think this sounds quite familiar to a lot of you so what we ended up doing is using Kafka pretty much for everything so that's is pretty much yeah the day that it's the data pipeline for us a data back end so we put the logs in there we have metrics and heart beats for Ford applications and the containers go through Kafka also application events that can be passed to analyzers and data pipelining jobs we put the data in there as part of the processing before it's passed onto persisters and this also gives us a great feature that we can use like we can replay things so Kafka allows it what Jay was showing us this morning you just point to a specific place in Kafka and the Catholic a timeline and process the task from that point and it's also great for auditing purpose so just a quick picture but we're running at the time so I'm just going to skip through and recap yeah meiosis our foundation we run da Christ microservices building blocks marathon is the orchestrator to manage the tasks and calf guess the main backbone for the data flow flowing through the system soon that's pretty much it so thank you and also we're hiring like everyone else see if you're interested talk more about that see me afterwards stuff thank you we maybe have time for just like one or two quick questions is there anyone up here okay this one in the back over here so you mentioned that you were using mesos for mostly the data center operating system idea I have you are you using it at all as like a spark execution engine or have you looked at that sort of stuff or they haven't used it as that but we're looking at it all right one in the front I'll be right there yeah hi I wonder is any of this tied to the US market or could work in say Europe yeah yes he was asking if this is tied to US market but we have customers in Europe and other places as well as not just us but there are different laws and regulations different parts all right and maybe one more so does it mean you spin up a separate instance for each customer of this configuration of it no so it's more instances for different services so the integration the data sources the many of those around dedicated instances but not the customers so the customers consume the data through the API which is basically as one I mean many instances but is 11 API so you have security on the API yes okay great well if there's any more questions I'm sure you all I would be happy to answer those offline so but thank you so much all out maybe we can join me to thank him one more time