Extracting Medical Attributes from Clini...

data.bythebay.io: Sanghamitra Deb, Extracting Medical Attributes from Clinical Trials

Recording: data.bythebay.io: Sanghamitra Deb, Extracting Medical Attributes from Clinical Trials

so firstly I'd like to thank uh people at data by the bay for such a great conference and you all for staying for one of the last talks in the conference um my talk so I am not a Pharma or a biomedic person as you could probably tell from my background so forgive me if I kind of throw terms all over uh because it took me a long time to actually understand Concepts diseases and symptoms and how you differentiate them Etc and uh my talk is going to be on the details of how to use machine learning and natural language processing to extract medical attributes and also I'll take a step step back for a more General solution so let's see uh so what I'm trying to to do is I'm trying to my goal is to extract medical attributes such as age groups side effects ethnicity the drugs that are being studied uh from clinical trials as well as from FDA drug labels and then also figure out how they are related for instance a drug and a disease can be related because that drug actually treats that disease it can also be related because that disease is a side effect of taking the drugs so also to dig through and find out the relations between these entities and not all of them will be connected but uh some of the them will be and the main motivation for this is we have heard quite a bit about personalized medicine today and uh there the idea being that a 200 lb man with fever should not get the same dose of Tylenol as me uh with a similar kind of uh symptoms so something like studying clinical trials and try getting all these um entities out and figuring out the relationships between them actually gives you access or gives you uh an idea about personalized medicine without having to do more trials uh and that's like one of the goals for doing such an extraction so first thing uh that I look at is FDA drug labels and these drug labels are the drug labels that comes behind your Tylenol box uh except that they get updated very frequently and new drugs come in and typically the one you'll find behind your uh medicine uh pill box may not be the latest one for that product so it's it's a it's a very um Dynamic kind of data and the first thing that I look at is the relationship between drugs and diseases and the relationship is that it whether it treats it and a very similar can be done similar thing can be done for side effects and some of the other uh attributes and these are some of the examples of the data so um if you look at it is indicated for treating respir respiratory disorder caused due to Allergy now what I'm going for is I shouldn't just uh extract respiratory disorder and Allergy because this drug treats respiratory disorder and not allergy so I'm going for the exact symptom that the drug is treating and then there are like more uh simple sentences and uh more complex sentences uh here and a lot of the time the sentences are not even very understandable in English terms and I kind of did not put them up so that you get a feel for uh what to extract the other thing that I'm looking at is clinical trials and clinical trials are a little harder than FDA drug labels because um typically FDA drug labels have somewhat a structure that is uh you can identify that your uh the content you're looking for is within these three sentences for FDA drug labels I had to do more uh topic modeling and other kinds of NLP to just extract the sentences that contain the information uh also my goal in the FDA drug label uh sorry in the clinical trial data was much wider I kind of want to extract all of these uh metadata I want to uh extract the age of the subject being studied the gender uh the side effects the duration of the study uh Etc and um the hardest part here of course is that uh it's very hard to find training data once you move out of the simple drug disease regime and uh this is another one with just side effects uh and the questions I'm trying to answer uh one of the things uh I'm hoping to find especially out of clinical trials is off label drug uses we heard a little bit about this today as well and um it would be quite interesting to see uh new um new treatments of the same drugs that existed so maybe like some uh drug for depression actually cures hair loss uh that would be pretty great uh the other is database completion so there are a lot of databases like umls for instance a talk that happened uh about an hour or two hours ago um that contained data which kind of get updated there are also multiple databases there's umls there's snowed there is bioportal and all of these get updated at different times so the idea of doing like a drug disease extraction is also that if you can gather all of the text data and if you run an extraction every night you actually have the most recent uh complete database and you don't have to worry about how do I like how do I get my head around all these five databases that exist around the world uh the caveat here also is that you have to get around how do you get all of your text data in one place which is again non-trivial and the other place where this can be useful is when people start clinical trials a very common question is what have has been studied before or uh are these drugs like on on what conditions are these drugs studied or are these compounds in the drug studied in some way and most of the time people don't know how to uh quer query those results so converting like all of this text Data into structured format would help in to generate queries like this so and uh in order to get the training data I actually took it from umls and the first time I looked into umls it's um it's actually very daunting especially if you're not from the Pharma Community there are Concepts there are uh disease names then there are viruses and you don't know what and how to get it so actually at the end of the day I had someone who worked in Pharma pull out 8,000 um 8,000 disease and drug names for me uh for this particular uh problem and uh the course of action was the first extract the sentences that contained the data uh in the next phase uh use po tags to actually get the candidates that might be diseases and for pox spe specifically I had used nouns and phrases or byrams or trigrams around nouns and then extract features so if you read the sentences if you read like sufficient amount of the sentences what you realize is that uh the drug uh the disease that is treated by a drug is typically in kind of a some kind of a syntactic or syntactic format in that sentence so what it it told me was that the context of the word was very important in figuring out uh whether was a disease that the drug was treating and also like the position seemed like it would be important so these are the two things I picked out uh I used both like a window that I created myself but I also used uh word to ve um vectors for these disease uh phrases and then the next part that I did was so I have these candidates uh that I generated and then here is like my data for my training data so I kind of created a mapping between the two and so this was a process of automattic automatically generating the training data and this also had some error and I'll speak a little bit more about it once I show the results and once this process has been done uh I take I typically feed it to a machine learning uh Mo model and uh the model predicts whether this is a disease that a drug treats or not to go about doing it so this is like a typically uh lemmatized sentence and these are the nouns and the noun phrases that came out of it and you see there are lots of uh nouns apart from diseases there's maintenance therapy intensity episode and uh when you create a training set what you have is like I call this rule-based prediction but B basically the these are the candidates so you say all of these are my candidates and then you use the training data to generate which one is actually a disease uh now a typical thing that happens is for every sentence you'll find typical one disease that the drug treats and a lot of nouns that are just there so there are just zero so if you take the model just by itself it will be a very disbalanced model you'll have very few positives and mostly negatives so it's important to go in and kind of balance your training data so you have equal numbers of positives and negatives to build a good model uh so another thing that I I had done is to build the word vectors and I use Jens some for that purpose uh and I also initialized the vocabulary with pre-trained vectors which uh gave slightly better results and since I um was working with word vectors I decided we needed to have some um man plus woman kind of thing so I I added some uh depression related diseases it's not as much fun when you have depression here but ADHD plus manic episode can be bipolar disorder then respiratory disorder per plus allergy could be common cold and they don't make any sense in this context but it's just just for fun um and uh this is pretty much like after all the feature uh feature engineering and all all like the data cleaning and most of the work goes in that like the data cleaning the feature engineering the tokenization uh this is the code that actually does the machine learning modeling and I have a completely python stack uh with the exception of the Stanford NLP parser uh which kind of gives really nice limitied uh output Spa tags and dependency parsing of the data um and so I feed the data to a pipeline like this and um typically I do something like a grid search or a hyperparameter tuning to get the best possible answers and the best possible um machine learning module um and I I get the answers so this is like the to the previous slide was just uh this box and this is like the typical machine learning NLP uh pipeline line uh you do feature extraction um you do uh create label data uh you have tokenization initially you feed it to your machine learning pipeline do hyper parameter tuning and then um base your answer on the metrics any questions so far so here is one of the results uh nice one and the drug name was lithium carbonate uh and it cures bipolar disorder and and manic episode and these are the words that were like not related to disease at all and so it predicted uh to be zero so one is the ones it treats I also want to show some examples where it goes wrong uh so it goes wrong for two reason one reason is here where uh the prediction itself is not good the other reason is sometimes the labeling of the data is not very good in the training Set uh so for instance in here in the training set basically it was labeled as uh not a disease whereas this is like a cream that is used in thir degree um Burns and the um model predicted some of these as um as correct but this was still went into it still contributed to the score going lower and this this can be improved more uh the automatic creation of uh training data uh so the next thing I'm looking at is uh the exploration of different uh machine learning models and one thing you notice is that most of the machine learning models give pretty high uh Precision recall um and F1 scores and one of the reason for that is for this specific data I had very good um training set and for the other uh metadata that I'm trying to extract the main hindrance is the training set and that's what I'll talk about for the rest of the time uh the other thing that we see is that pretty much all of the techniques do similar and pretty well so I am sticking with logistic regression because in future I kind of need to deploy something like this on larger amounts of data or on data with big PHA companies so doing this at Accenture one of the goals is also to create like a proof of concept uh that can be reapplied to different kinds of data so one version of the code will be like having a spark version which does like tokenization logistic regression produces an answer and for that purpose I typically try to go with the simplest model that works uh so now the clinical trials data and you could approach the clinical trials data exactly similarly you could have uh nouns noun phrases or use different kinds of pause tagging uh to extract words and then you could just say that you know is this word uh refering to the age is this word refering to duration of the CL uh of the drug being um administered is it refering to the disease itself the problem with something like age or dosage is that I start with no training data and so that's the problem that I'm working on right now and um what I'm trying what I want to do is I want to have a training data uh that's like this for every sentence where whatever I extract uh will have like uh a one in one of these verticals and I can use it to extract um the attributes and I only have TR training data for um for this kind of data like the um the relationship between drug and side effects drug and disease and also drugs itself and for the other ones I have to figure out either manually or semi-supervised way of generating the training data and this has been basically the hardest part of the project once you figure out how to once you break down your problem in such a way that you can uh put it in a machine learning algorithm it's fairly easy to find a model that will give you a relatively good answer so one of the things that I'm trying right now uh is um I'm uh just like uh the previous part I'm creating candidate Set uh of uh nouns or even other PA tags uh I'm sticking to nouns right now uh and then I'm taking something like hand labeling a 100 of them after hand labeling a 100 of them them I'm generating a set of rules and the rules could be like for instance oxa occurs in drugs quite frequently um or and I have some more rules uh which I'll go into later and so I generate a set of rules and I use those rules to get a data set uh that has 95% accuracy and once I have that I kind of um look at this data set and I I change the rules so that the accuracy keeps in increasing the other thing to kind of keep in mind is this is an iterative process but you should not iterate more than two to three times because then your training set might become very close to the rules and you might just be keeping you might just keep fitting to the rules once you go to the machine learning part so iterate a few times not more than three and the best way is finally when you have a sample if it's if you have enough resources which I don't currently but if you're working in Pharma you might um is to go through like reduce your data down to maybe 5% of your total data and then find human resources to actually um guarante 100% accuracy in the 5% of your data and because if you've reduced it so much you have a very small data set which you might have human time for uh but you have 100% gold standard data for whatever you're trying to find out uh so some examples of rules are for instance if you're going for age the sentence should contain numbers uh or the first one is dosage so it should contain numbers it should contain terms like milligrams mg and other forms of uh dosages and these are things you can find out by exploring your data uh contains a word do but also look at other synonyms of do as a matter of fact you could do word to on top of your medical journal put those in and see what the other words are and have a rule that contains those words similar things for age uh contains numbers contains year old um and and um and to start the process you don't even have to know all the rules you could start with like three rules and you could reduce your data down once you reduce your data down you could go and explore the DAT data look at the data do engrams on that data and figure out another set of rules that makes your um that makes your makes the error go down further so to begin with you could have just 70% accuracy your rules could give you and then you could keep improving from there and uh typical set of codes um that are there for uh generating rules uh the next thing that I want to talk about and uh this is a program um or a software from Stanford that I have been working with a little bit it's called Deep dive and what it promises to do is you throw all your data in it could be Json it could be PDFs it could be um text files and it will and you also need to know your candidates and you basically need to do all the work till before the machine learning part and you feed feed that data into deep dive and it'll create a structured database for you now I started this project actually six months ago and I thought okay this is Magic and I can throw in all the clinical trials date out and out will come the age uh dosage Etc so it's not that simple it's actually you have to put in quite a bit of work uh but the part that it makes easy is uh if you're not a machine learning engineer you don't have to worry about should I use logistic regression should I use random Forest what do I do with Hyper parameter tuning uh it gives you an answer and it gives you the probability of getting that answer uh so that's the re really neat part is you don't have to worry much and initially when I started working with this uh it had a lot of overhead in the sense that you had to uh know your data structure ahead of time and create the entire architecture and then start the problem uh luckily these days they've made it uh easier for prototyping and they've put everything like you can do it on an IPython notebook uh so again the typical way of using the a lighter version which I have been uh using a little bit lately is you create the candidates either through nouns or maybe you have a set of candidates stored somewhere uh you have some labeling functions you feed them in and uh the model works on it gives you an output most likely you won't be happy with the output uh then you go back and look at the output and uh they also have a really neat um UI to look at the output uh so the UI actually I'll um just go directly to the UI so the UI actually um marks the ones that have been um tagged by the code as your um as your attribute and you could go in and kind of do this is wrong this is right like for some of them and that data gets that feedback gets stored into it and you could use that feedback to retrain the model so I have found one thing to be really cool here is even though even if I don't use Deep dive just for looking at this data this UI is really great and the other thing that I wanted to show was uh the IPython notebook is also very convenient so basically you can load the code up and all of this is kind of available open source and these IPython notebooks are also available so you can uh load the code up and uh load your data into it and there is some formatting but not not a lot and once you load the candidates you can actually look at the candidates in the IPython notebook itself and then you can use a code to write your labeling functions and use a code to get your answers so um this is pretty neat they've only done this IPython notebook thing for the past I I don't think it's even a month old so it's very much on the uh it's very much being developed and if you look at it like I feel like a few months from now it might be much easier to use than it is right now um and uh so this is like the workflow for deep dive and this is a typical workflow that I have also used uh in my machine learning modules except in the models and features there I have done the uh machine learning and the hyper parameter uh tuning and there are two things when you come back and change the rules there are two things you do one thing you sometimes you realize is that you're missing out when you look at the data you're missing out most of the C candidates in that case you need a rules that Encompass everything and then the other part is where you realize um that the accuracy is not good enough which means the rules need to be more complex or maybe you need to add three more rules and uh this is like one of the neat um renderings from uh the Deep dive light as as well and it's just a D3 on their back end but it basically talks about a gene that causes sex reversal and yeah that's it and so in conclusion um NLP relationship extraction and machine learning is not very difficult if you have very good Training data and the for most projects I concentrate most of my time in gathering training data and finding gold labels uh another thing in Pharma that has uh I've only worked in Pharma for like four to six months and what has struck out to me is that there is a lot of data but it's very very spread out so what would be really beneficial is if someone like me who doesn't work in the field could go to one place and find all the data or there would be like easier apis for getting the data uh and most of the conclusions that we make should be made from the entire volume of data and not from data silos because we're kind of missing out when we don't include all of the papers uh yeah I mean that's that's what I've got uh at Accenture we are hiring all kinds of people data scientists engineers so uh if you're looking for a job um email me and uh I'm also writing blogs about this Pharma stuff so [Applause] yeah [Music] questions so uh may ask um where do you see the sort of lowest hanging fruit in terms of the the first things that these bringing these methods will will will allow so the lowest hanging some of the lowest hanging fruits are data that has not been tapped in the sense of uh doctor's notes or laboratory notes or even uh published clinical trial so FDA drug labels are actually not the lowest hanging fruits it was a good kind of proof proof of concept to start with um but these are the places where that there's data and it's not a lot of work to actually extract that data Out and because there's like a community out there I feel like if we share the training sets if we share more of the information it's it's it won't be very hard to get things like um the like dosages or age groups and these things don't exist so when I spoke to the Health Care people uh at Accenture U they were like yeah these are information we need and they they're not in a database thanks well thanks for a a really fascinating talk and sharing your research um hopefully you'll be around uh if people have more questions later sure great thank you very [Applause] much