Cancer Screening using Deep Learning

data.bythebay.io: Daniel Golden, Cancer Screening using Deep Learning

Recording: data.bythebay.io: Daniel Golden, Cancer Screening using Deep Learning

[Applause] thanks so this is actually a small topic change from what was originally listed on the schedule we uh made a slight change in company plans right after I submitted the abstract so I think on the schedule it's uh the talk title is referencing uh cancer diagnosis and lung cancer and but instead I'll be talking about something at least from a data perspective that's similar which is segmenting hearts in cardiac MRI using deep learning and specifically I'll be talking a little bit about how we deal with some clinical data that we're using which is distributed in a sort of An Inconvenient way and how we work around that so let me give you a little background first on how arteris works as a company and our current platform for the use case of cardiac MRI typically the first step is the patient will receive an MRI in a clinic or Hospital cardiac MRI the data is then sent to our Cloud where it's stored and processed and reconstructed and uh the cloud performs a few different functions in addition to reconstructing and storing the data we use gpus in the cloud on Amazon AWS for rendering the data so when the clinicians are using our web app uh all of the rendering is done on gpus in the cloud and the user can use a Very Thin Client such as their phone with very good performance and also we use gpus obviously for deep warning so this is either as a pre-processing or online step of doing inference in order to detect uh features for example in the data so to specifically color what we're using deep learning for it helps to have a little bit of background in cardiac Anatomy what I'm showing here is a cross-section of the heart in blue is the oxygenated blood uh that is blood that is going to the lungs ready to receive oxygen oxygen and the red is oxygenated blood that has come from the lungs and is ready to be distributed through the rest of the body so initially deoxygenated blood comes from the body as a whole through the vnea superior and inferior into the right ventricle and as a calvat here standard cardiac uh views in MRI the right heart is on the left on the screen so they're revers here it's as if you're looking at the patient face on so deoxygenated blood comes in here to the right atrium flows into the right ventricle the ventricles are the actual pumping parts of the heart the Atria or staging areas pumped from the right ventricle into the lungs comes back into to the left atrium left ventricle and the left ventricle is responsible for pumping it to the rest of the body through the aorta so in a lot of different types of cardiac disease and dysfunction the ventricles are the region of the heart that are affected the pumping regions so these are a particularly important area for uh performing segment a on and for quantifying the volumes the specific type of cardiac MRI that I'll be talking about is called ssfp which is a specific kind of MRI pulse sequence that stands for steady state free procession which I put up here because I knew I would not remember the acronym live uh it looks like this and just as a side note if you look up arteris online you'll see a lot of references to 40 flow which is a novel type of pulse sequence developed in collaboration with art Stanford and GE Healthcare I won't be talking about 40 flow here it's a that's a newer type of cardiac MRI but because a lot of Institutions still use this Legacy uh method ssfp we are working on products to support that Legacy data type so I'll be talking about this kind of MRI uh right over here I should say a typical ssfp acquisition consists of slices uh along the short axis of The ventricle so if you remember remember from the previous slide maybe I'll jump back for just a second the heart has a long Direction and a short Direction so these slices come back one more time are Cuts Like That through the ventricles um in the center here the rounder region is the left ventricle this is the part containing oxygenated blood the uh Crescent shaped structure over here is the right ventricle which has deoxygenated blood and root to the lungs and then the darker regions surrounding the left ventricle is The myocardium which is the fairly thick muscle surrounding the left ventricle that is pumping the blood throughout the body so one thing that you can see here that's going to be important later is that although the left and right ventricles mostly pump together they're slightly out of sync which means that the time point at which the left ventricle is completely contracted can be slightly off from when the right ventricle is completely contracted and vice versa the time point when they're maximally d ated may be different and the reason this is important is because when measuring cardiac function the radiologist will typically only segments and determine volumes at maximum contraction and dilation so the time points at which we have segmentations can be different for the left and right ventricles and I'll explain why that's important next so the next the The End effect of these time points being a little bit different is that our data is not not really in in the optimal format so there are three kinds of segmentations that we're interested in that card radiologist typically perform in order to measure volumes within the heart there's the endocardium of the left ventricle endocardium means uh basically the blood within the left ventricle there's the epicardium which is a contour that surrounds the entire ventricle including the muscle and there's the right ventricle endocardium again just the blood in the right ventricle and typically Radiologists do not segment the myocardium of the right ventricle because it's very thin and difficult to see in these scans so as a consequence of the fact that Radiologists typically do not measure all of these Contours in the same slice and also the Epic left ventrical epicardium is typically of lesser interest it's only interesting in certain more esoteric cases we do not have all three of these segmentations on all of our images in fact although we have left and right ventricle segment uh endocardium segmentations on most of our images we have uh epicardium segmentations on far fewer about 20% and then when you take the images that have all labels that's only about 18% of our total data volume has all labels so this poses a problem if you want to use a very simplistic segmentation neural network architecture because most images do not have all the labels you care about and we want to avoid sort of the onerous process of making different models for each of the different labels because all those those models would take you know three times as long to train and they would share a lot of information or they would have a lot of information that could potentially be shared that wouldn't be shared we wouldn't be exploiting the similarity in these problems but let me show you uh some uh an architecture that we implemented to do this the naive way and only use the data for which we had all three segmentations On Any Given image which was only about 20% of our data volume and then I'll show you some results of that and uh way we improved upon that so in this case we have uh a standard convolutional neural network architecture we have more than three layers I'm just showing a cartoon over here uh where we have the data coming in from the left and then uh output predicted masks coming out on the right and I just want to actually take a step back how many people in the room have worked with convolutional neural networks in the past so that's about a third or a half okay so for for the rest of you I I apologize some of this might be esoteric but I'm going to show some pretty movies so we can pay attention to those all right so uh the input into this model that's being trained is an image which is a a single cross-section a single time Point static image uh of the heart and then a ground truth mask that is indexed where every pixel is assigned one of four labels background uh and then the other the three classes which take too long to say um and then what the network will do is it will produce probability maps for each of those three classes so for every pixel there will be um a probability given to each of the four classes and those probability sum to one and then when training um this data comes in the model makes a prediction it's compared against the ground truth loss is back propagated Etc and then the at inference time the model actually uh can create these masks which will be um again probabilities but can be you can take the argmax function and you end up with these you know static masks like that them again the problem with this approach is very straightforward it's a very standard classification architecture except with segmentation so about classifying each pixel and the problem is that we only end up using 20% of our data which fits into this format so here's the result of segmentation when we uh use this architecture with 20% of our data and what I'm showing here are probability masks colored according to the different classes and then derived Contours from them where we've thre holded the masks and then just drawn a contour around and like the first thing that's going to strike you is the red Contour which is the right ventricle is very poor if we were we think the goal of this whole procedure is to deploy the results of this model in our app where a radiologist will click a button and these uh Contours will be created automatically and you can see the right ventricle in this case is not Deployable this would not be usable in a production app and here I'm not talking about an FDA clearance issue because the radiologist has the opportunity to correct everything before for determining clinical care it's really a ux issue where the radiologist would essentially have to start from scratch with a right ventrical segmentation like this the left ventricle segmentations look better um which is partially due to the fact that the left ventricles a simpler shape but still there's some noise there and some corrections would have to be made so now I want to talk about the architecture how we changed the architecture to allow us to train on all of our data now uh again again well the architecture has changed here so that instead of outputting four classes we essentially interpret each class independently but still within the same architecture where we predict on each class versus the background of that class so each pixel is either right ventricle endocardium or not and those are all done separately for the three classes so in this case the training data is separate masks for each of the three classes and loss is calculated separately for each of the three classes and then the loss is summed across the three classes and back propagated again the the the only difference here um from a practical uh standpoint is the output the internals of the network are the same and the key Advantage here is that if a given image is missing one of those three classes we simply ignore that component of the loss the network when when doing training the network will still make predictions for a given training batch even on a lab for which we do not have a ground truth mask but we will simply discard those predictions when calculating the total loss sum the loss from the two classes that we do have ground truth on and back propagate that and in doing that every image that has at least one ground truth Contour can be used in the training process so here are the results when we train it using all of our data using that architecture I just described you can see immediately the right ventricle Contour looks much better the uh it much more closely uh follows the actual boundary of The ventricle and uh the left ventricle Contours look better as well so the huge difference here is really from a user experience standpoint either way you could deploy the model you're not going to get in trouble with the FDA because the radiologist has the opportunity to fix anything that went wrong but the radiologist needs to do much less work in this case to get to an acceptable segmentation and an accurate volume estimation so that's a big advantage of this technique and and why we really want to make it very accurate to ensure they have to do as little work as possible and I just showed one example case there but if you look at a bigger picture the whole point of these segmentations is to measure ventricular volumes so what I'm showing over here is specifically for the right ventricle which is outlined in red in the previous Slide the relative volume error so we have ground Truth for um in this case we're using a a a Val validation set with something like 100 cases and we have ground truth there for what the correct volume is and then we've predicted a volume um and what you can see here over here on the left is the model where we only use 20% of the data the more classic classification architecture and on the right in green is the model where we used all the data the somewhat more sophisticated architecture and you can see that we have a median volume error of about. 3 that means about half of all cases have 30% or more error in their volume or if your glasses half full guy or girl 30% have man this is too too complicated I think you get the idea um 30% error in half of cases and when you go to the the model that uses all the data about 10% error in half of cases so you've reduced the median error from 30% to 10% which is really a huge savings I mean it's I think it's pretty obvious from this plot that this is a big advantage and has worked is going to work very well so that's the the the last content slide I had and I just want to say the conclusion here maybe this is not a surprise and it's not something you needed to be told that more data is better but the real conclusion from the effort that we went through to exploit that data is that the effort is often worth it that data you our original version of the network was sort of locked away and it was very was challenging to access so by putting in a little more effort modifying the network architecture to accommodate it and to exploit all the data we got a much better result which is really nice because that means Radiologists will have a much better experience using our product uh that's the end of the talk and I just want to mention that we are hiring if you are interested in computer vision in the medical domain we' got some really fun stuff for you go ahead and send me an email we'll get in touch talk more and thank you all very much for your [Applause] attention thanks Dan I'm gonna abuse my position as microphone holder um and ask a question is um uh I noticed that in the looks like you pretty much nailed it with the second model but in the first model the boundaries look quite smooth in the space dimensions but kind of flickery in the time was it did you train on individual images or on the sort of 4D volumes U maybe I'll just go back to that I think you were referring to this video or to either one this yeah yeah that guy so this model was trained using uh two-dimensional images only so in this particular model we have only used a single image at a time in training and inference we're also experimenting simultaneously with using adjacent images uh in space and time and also 3D convolutions but this does not exploit that thanks any other questions can you talk a little more about the stack can you talk a little more about what you used you know for CNN sure a little bit more about uh the the framework and model architecture sure so we use tensor flow for both our um training and inference so the whole goal of this is to integrate into the app so tensorflow is a good choice for us because of how much attention has been paid to using it in products so we want to use this in a in a real product a healthcare product so that robustness was was very important to us um and then in terms of the actual model architecture we use a variant of the semantic segmentation architecture posed by long and all in 2014 which was I think the first paper that mentioned using a Comm a fully convolutional neural network for doing segmentation uh so we have no pre-or postprocessing of the images the segmentation is wholly contained within the neural network I have another question um when you didn't have the labels for particular parts of the heart um and you added those those uh that data um you didn't train those n notes or or um how do you you know you you show that X so that means you're not training uh the connections to those noes or you're just pruning them out so the uh when when uh training and this uh data comes into the network this ends up being set to zeros which is okay because it's ignored anyway and we essentially have a flag which is sort of just how we arrange the data to indicate to uh the last layer of the network which is right at the loss function that we do not have information there and then within the tensor flow graph we actually just delete the components of the loss the loss ends up being one it ends up being a vector that has the same length as the total number of pixels that came in and we just say that the components of that Vector that correspond to that missing mask delete them take the average of the remainder and back propagate that does that answer your question cool uh thanks for a great talk Dan um and if there are any more questions hopefully you'll be around uh afterwards offline yep cool thank you [Applause] thanks