data.bythebay.io: Jeremy Corbett - Case Law and ML on Spark
Recording: data.bythebay.io: Jeremy Corbett - Case Law and ML on Spark
thank you so yeah today um just going to give a some broad Strokes about um H how our ingestion pipeline works on spark um yeah so let me me get started so uh to frame the talk a little bit um our CEO spoke in this room at 11: talked about the business a little bit but we're a uh rev is a legal research and analytics company so we signed a deal with Harvard uh last fall that allows us to scan all of the US's case law and um present it to the public for free um that's uh we're very excited we really think this uh this information should be available to the public and we're also excited to be able to have exclusive commercial rights to it to build analytics on top of it to sell it to our our nation's largest law firms so two products we have uh available for people today are judge analytics if you have a legal matter in front of a judge we want to give you the ability to see everything they've written in the matter which you're going to be before them and some analytics on the citation ation patterns that they have on those matters we also have a uh a search product and a reading experience for for case law so what is what is case law so I really like this quote from uh justice Oliver wend Holmes the life of the law has not been logic it has been experienced so Congress will pass laws those laws will be codified into two statutes and then a judge is the person who actually represents the Law for um a particular matter so legislators will pass things and then a judge will see the actual facts of a matter and decide how those laws should actually be applied so instead of it being sort of like this really uh pure philosophical thing it is actually like our cultural experience that goes into how um our laws get shaped so in a more computer sciency form case law is a graph the nodes of case law are legal opinions that judges right and the edges are their citations so right now in our Corpus we have about 7 million opinions at the completion of the HS project which should take about two years we should have about 11 million documents right now we have 54 million legal citations between those those notes so just uh just to give you a little bit of a a couple screenshots of the product to give some context for what are spark um build pipeline needs to support uh this is a a screenshot of our our search visualization so as I said there was a um case law is a graph and when we when you search for a term we try and show you the relevant citation Network for the um search results so uh we'll segregate them by level Supreme Court um Federal Court and state court and hopefully give you a a more abstract overview of how this area law has developed over time then we also have the the judge analytics product so if if you follow um any of the Apple Samsung litigation that's at the district level been in front of Justice or judge Co um here I've we've selected the topic filter of patent and you can see in in opinions that Co has written on patent law these are the most um frequently cited cases you also get some some language on um the particular cases that have been cited um and and some pieces of text from the opinion she has written to get the sense of why why does this judge lean on this particular case when they're explaining their reasoning so in in the judge's product as well we we uh um have some different uh machine learning features to help people get a sense of what a case is without actually reading it it's it's kind of fun like this is all um very most legal writing is pretty readable Pros but the first thing you hear from attorneys is if there's any way I can not read a case um please make that happen for me so we try to do that um so we will uh we have topic models we'll try and say what a case's um topic is we have U key phrases things that we think are important and can help um get to very spec you know more specific search experience for the user um motions so we'll try and identify if a motion is being ruled on and what the outcome of that is um and then we have we'll try to get decision and summary sentences so we use text rank to try to figure out what the matter of the case is and then if we can identify in the opinion where a judge actually makes a decision um we will we'll try to deliver that that piece of text to the user Okay so so um lots of stuff needs to go on and Spark to support this product so we need to um get our source documents from various data providers and get them into mongodb in our opinion model we need to extract the case law and statute citations and resolve them to the actual documents that that they're citing we need to do uh D duplication detection um we use techniques um from the plagiarism detection space but there's no Universal identifier for case law it's uh the publishing oligopoly sort of establishes how where they appear in the books but there's no um Universal way that people agree upon for saying this is this particular case we extract the authors and try to resolve them to the actual judge entities in our database we uh use LDA topics and um those support uh the text rank we do to try to get the case summary motion outcomes is one of more complicated systems several ner pieces in play then after that is done we'll use some um some parsing techniques and then various classifiers to try to decide um which part of the case is actually talking about em motion key phrase extractions mostly NE but several anyr components in that pipeline um we do some semantic clustering of um um Snippets from case citations that that uses K and X means under the hood and then finally after all these steps are done we need to export these things to mongodb solar Cloud Cav uh c c is a little more obscure of the three uh so it's an inmemory um graph uh storage system so we can store our entire citation Network in memory and do some realtime analytics on on the citation Network for judge analytics and other things that we do so uh I I showed up at Ravel in late 2013 when I got here we were on Apache Pig and that was exporing to Cassandra so 2014 I think around like the spark 17 maybe 1 n time was doing a little bit of proof but concept work on spark and um finally around late 24 14 were actually switch over to spark exploring to mongodb no no particular hate for Cassandra at Ravel law but um it do we're very much dealing with documents so a more document Focus store just really helped us out for our quering option uh for querying and other things that we do so why switch away from Pig um so here's some code like I I could rant on this quite a bit but you'll notice there's a lot of like type ceremony so you're going to be getting things from various sources we have very uh deeply nested data structure so a lot of calls to flatten and then saying What fields of the things you're flattening it very very much has the feel of a u sort of dynamic programming language where when you are exploring or dealing with the the outside world you are having to very verbosely explain the the type information of the the data you're dealing with so pig is mostly SQL but you can make these things called userdefined functions once again very much like a dynamic language you need to describe the types of the things coming into and going out of the system so um that that ends up becoming a a very major maintenance burden over time also you uh even when you do have some type of information you're frequently throwing it away into tles so uh yeah if you've ever had to deal with Json much in a type language um you'll sort of know the pain I'm talking about but uh Pig it's it's very quick to get it set up and get started getting results as soon as you have much of a a code base you um you can become very Paralyzed by uh the amount of type ceremony you're sort of doing in your code so uh got this quote from Robert Harper I can thank him for ruining my my rails career with this argument uh a dynamic language is a straight jacketed static language that affords less rather than more expressiveness so in a type language you can have various types and know properties about them in an untyped language you have one type and you know nothing about any of them so uh Google recently moved off of SAA I really like this quote from that that blog post um as your as a language grows the rationale for using a domain specific language rather the general purpose becomes more diluted so being in pig um having to really support the types versus being in a typed language like Scala which has a lot of tools that are useful for for us it's night and day experience um yeah so spark makes you happier so here's one of our our steps in our build pipeline very straightforward we'll take an rdd of opinions maps all of those to find a topic and then below it is a function that is scoped directly to one opinion so I can use the top to run this across the whole Corpus the function below I can easily use for local testing or other experiments I might do um the fact that I'm going to execute distributedly on spark is sort of like a minor detail in this actual method it's very much focused on doing one thing right and then um using something like spark for a nice horizontal scaling strategy so when we move to spark uh we have a lot more confidence and we had a lot more speed moving forward in the build pipeline um we were able to add a lot more n models and classifiers because it's a lot better to develop on top of this um with the exception of closure serialization issues spark is just distributed scalla collections there's nothing um that special about it you really don't think about the fact that you're running on spark a lot of the time it's very straight forward crank out some scol code and then run out a run out a cluster all right and so um to people are provocative I really think that machine learning is software um data science and engineering can move it and operate at different time scales But ultimately data science hopes to create software that is shipped in production so I think a lot of the lessons we know from software we can apply to data science and build pipelines um very much so don't overthink things don't pre prematurely optimize profile these things while they're running to get a sense of what are the things you can attack to make them faster if needed so the the new version of the uh the build system I like to call the sence pipeline so we're still on spark and but instead of um dealing directly with during the build process we use parquet as an archal format between steps so um yeah very I think this project was a very good problem to have for our company we've added a lot of systems and uh this to the uh the build pipeline but it had become more complex and um very cumbersome to run so as over time we added more and more ner and classifiers it's not just these individual systems but the interplay between them that is very valuable so um nice problem to have but we needed to do something to decrease the time it was taking to do belts so this is a at first this was not very painful but every step in the old build process would go read from mongod Tob and then write and some sort of update to mongod Tob over time there are a number of issues that happen with this so um one is that it's very hard to reproduce States unless between every build step you do a bison dump you have no real way to say what was the actual input of this job without restarting the entire process the other thing is is that um the original storage ENT from did horrible aext wired tiger uh brought compression to the mix and and uh really helps things out but under wired tiger if you update the documents after insertion then you um are B you end up basically doubling this the database size they don't really do a good job of providing a vacuum like facility for um update databases that you're updating and uh we're the Corpus isn't huge we're easily doing pretty fine with a single cor uh single node setup we could have scaled that out that would have come with a lot of overhead but with par the you have IO that scales on read and right side with the cluster size naturally that and you don't have to think about the database Administration side yeah just uh to diagram this out instead of having this one um sort of cluster of of a hardto reproduce State we have any of our parket archives we can use for an experiment for the next build so if we want to do 10 variants of one step we can easily do that and have a uh a really nice operational story for how we'll get there so the uh the results of this work um we're able to take the the build time down from 10 days to 2 days on stay on pretty modest cluster sizes um we are able to run a lot more experiments on top of uh the ner that we have in our system and we have a much better uh development story so before it was very hard to say what the state of manga was at each step now with these paret archives we can just pull whatever subset so the Corpus we want down locally do local development spark notebook and then we're ready do um deployment on a cluster um on Amazon so uh little short on time we are we are hiring in this um for two spots they're related to machine learning and um data engineering so if you're interested in this stuff please please get in touch um then like I said am running short on time but I have a little spark notebook here um showing off a little bit of our code so really like work we're going to spark notebook um if you're spark shell on the repple can be nice but like being able to easily shift between um graphs and code works out really nicely um add our assembly jar start a uh SQL context and load some par up here I've got uh all of our opinions from this particular subset with um topics and then I filtered down to these sentences that have key key terms in them so from here um if I add a little more time for today I would say we could look at um try and look at some co-variance between topic spaces and the key phrases but the the the nice thing about this work is we can easily look at our Corpus and also see what the machine learning system has set about each part of that corus um yeah any any questions uh so we're we are for the most part A Stanford uh Stanford any Ariz uh store yes and uh in terms of searching um uh if I search by fi I okay how about supposed those huge area and even though you come from the citation depend way to define res for but some of the um relevant case may come earlier okay so uh how can I find the missing maybe I just descri want to as a as a junior attorney May naive in ter of the case I was invol describe my case in more naive way yeah okay so so uh yes so in our search ranking algorithm like the uh we do some pretty vanilla stuff in solar but then we also do a a reranking using the citation Network so if you're an attorney searching your the our results are very naturally weighted to highly cited case law which for the most part is the most precedential and influential you mention using I was just CU why you use ah okay so um we there are oh oh thank you so uh why I use caver versus Graphics so uh there are some stuff that we do do batch side with graphics with caver we actually do have the database in our app servers for like real-time analytics for the user so no uh there are some algorithms that we actually do want to explore and use in the graphics package but um just Cav very really hits a a runtime need that we do have oh sure okay yeah so for the um for the document level topics uh we do we have looked at the the terms and given um sort of areas of law labels for them there's yeah it's very much like a human in the loop type process and the other the other A is for the uh the summarization so we have models that um are more for the sentence level and then using those vectors we do text rank to try and figure out what is the most highly relevant sentences from the opinion Jeremy thank you very much thanks