SF Scala, Andrew McCallum:: FACTORIE: A Scala Library for Machine Learning & NLP

Recording: SF Scala, Andrew McCallum:: FACTORIE: A Scala Library for Machine Learning & NLP

from their papers and patents and web pages newswire even press releases tweets and blogs we'd like to accelerate the prog the uh progress of Science by doing this and you can imagine some ways in which this would be helpful by helping people find papers to read and to site or find reviewers or collaborators or people to hire I'm always trying to find new post dos and I would love to have a tool like this to help me do so um or even policy makers to understand Trends and uh the landscape of science okay so to do all of this we need to do some information extraction from text into large databases and let's let's look at uh at how that's done look at it just in pictures without equations all right so we start with some text and one of the first things we need to do is find mentions of those entities like waye and shinoa University right and then we can also find relations in the midst of this text like way Lee attends shinwa University and then uh way Lee might appear in many places in the text but she might also be in some structured data that we already have sitting around and we'd like to incorporate that as well and now there are many appearances of way appearing everywhere and I'd like to pull them together by resolving them or doing entity resolution or co-reference to know when we're talking about wayy is the same person and when actually wayy is a different person and we try to do that from using all kinds of machine learning and uh uh um uh methods and then here I have what we think about as the truth about entities and relations I put this into my database from which I can answer queries and the Machine Learning Community takes great pride in having made a lot of progress just in the last 10 years over doing these things much more accurately than uh was available before um but it's 90% isn't perfect and although 90% might be okay as an individual item the errors snowball through this chain right once you make an error early in the chain it's very hard to recover from it later and so the end accuracy is more like taking 0. n and raising it to the third power which doesn't look so good so in response to this I and others have been interested in reasoning with more uncertainty and and uh sort of joint reasoning throughout this process by reasoning about probability distributions over possible outputs not passing just the single most likely outut and reasoning with you know sort of top down and bottom up reasoning um so that later more Global thinking can help correct errors in my in my low-level um uh thinking and so you can think about this is reasoning about all these together and we call it joint inance and it's we've we and others have shown that it absolutely can make a large difference in the accuracy of these systems most of this past workk has been a joint inference at the early steps of this pipeline where we're reasoning jointly about say the data that falls in one sentence or maybe within one document but I actually think that more there's more juice to be had by reasoning at the um at the far end of the pipeline because we that's just where we have more data that we've marshaled all together from which we can share evidence and uh um look for more patterns that can help us remove errors and confirm patterns but you know joint inference was hard enough at the early stages of the pipeline as it is and now I'm going to try to reason with uncertainty about more random variables than I can even fit in memory and that presents a whole new challenge and so this has us thinking uh about these problems in a bit of a different way um but first let's consider some of the interesting issues that this brings up so how am I going to represent and inject uncertainty from extraction into the database I'd like to use the databases contents to help me do a better job of information extraction this is like the way that we all use our innate Common Sense knowledge or knowledge about the world when we're interpreting lowlevel language and third and maybe this is um uh the most Salient and practical terms you know extraction is not one shot it's not like I'm going to have my 200 million documents extract from that and then I'm done tomorrow I'm going to have another 10,000 documents and the new evidence that I receive tomorrow should sometimes cause me to change my mind about something I uh extracted yesterday like Oh I thought these were two different people but now I see with this new data they're actually the same person and so I would like the databases the database not just to store the results of the truth in the end but help me manage the whole information extraction pipeline L storing intermediate results that make it easier for me to go back and back patch these situations so with that in mind uh we've been thinking about enlarging the purview of the database to look more like this in which the database stores all of those intermediate values and the only thing that you really inject into the database is not the result of some external truth Discovery process it's just the raw evidence and it's up to the database internally to discover its own notion of the truth so I think of this as aligned with epistemological philosophy which says you know you don't get to observe the truth right you just observe your evidence and it's up to your own reasoning to uh to discover the truth so we've taken into calling these epistemological databases and um before moving on and getting to factory let me just briefly describe some implications of uh of what this looks like um starting with uh human edits so you build a large knowledge base whether it's built automatically or by humans inevitably it's going to have some errors in it and when your human users look at it and they notice some errors they're going to feel better if they have the opportunity to make some corrections and the classic way to manage this think about Wikipedia I suppose is a human edit consists of going into the place where the truth is stored erasing the old value writing in a new value and maybe you you keep some edit history so you can know some Providence so you can unroll it if you want but you know this is really unsatisfying because users have edit Wars and um also sometimes the human can make a correct edit but then the state of the world changes 6 months later automated extraction notices the change but the human didn't and so sometimes the audit meth method should be allowed to overwrite the human and so with this in mind we've taken to thinking of a human edit as just like an additional piece of evidence it's like a little mini document that says uh you know on the state uh Scott said that this thing was true and that evidence just participates along with the rest including the textual uh data in part of this truth Discovery process in which we can balance all these pieces of evidence reason about truth and about human editors reliability all at once um we also then do a lot of work on never-ending inance so the truth isn't discovered just once externally but there's some sense in which inference is constantly bubbling in the background inside the database always being reconsidered with more evidence more time there will be also times when our database has some holes in it and we'd like to fill them in with additional data and we think it's probably out on the web somewhere but we don't quite have a Google size resources to extract from the entire web so we'd like in some targeted way to go out and do queries to find exactly the raw data that we think might be able to best help us fill our particular holes and this actually turns into a reinforcement learning problem that we've done some work on and then furthermore and lastly if we're going to do this at Large Scale we have to think in parall uh with parallelism in mind and we have found that just using map ruce as a blackbox is often a really bad idea and have seen some great gains by reasoning in a tightly integrated fashion about the details of machine learning and the way parallelism works and and uh I've gotten some big winds there uh um you know as has have graph lab I know there's some folks from graph lab here uh um tonight all right so uh with that now let me uh jump into uh into Factory uh so Factory is a tool that's going to help us do a lot of these things as well as uh other stuff and uh so Factory is like it's probably like the fifth major machine learning uh toolkit that I've built over the last couple of decades um I won't bother to tell you about the first two um uh the third one that I built was actually at this startup company uh in the late 9s um it uh was the key the core intellectual property of the company it consisted of about 300,000 lines of java and it did classification and clustering and confidence and fining State transducers and various things like that and and when the company went up in Flames it got locked in a closet and never to be seen again and uh I decided to go to the university I mean I actually like the startup life uh I had a lot of and I you know might do it again but um uh I missed working with students and doing research and uh so I wanted to go to the university and I arrived at UMass and I knew that to do the kind of research I cared about uh we' need a lot of infrastructure and that's when I set off to build Mallet um uh it's grown since then to about 130,000 uh lines of java it's pretty widely used both in industry and Academia and it does classification sequence labeling and a little bit of topic modeling and um if those are the things you care about then that was great but our research was heading Us in even more General directions that weren't covered by these cases and around 2006 or seven I was also feeling a little bit tired of the verbosity uh of java and so I uh talked to one of my programming languages uh um fellow faculty members uh who recommended that I um talk with Martin ODI um uh and this got me interested in schola so in yeah so this was around late 2006 I started working in Scala um started working on the precursor of factory got something up and running that was already supporting our research and about uh six months wrote email to Martin around the middle of 2008 and um yeah he says thanks for your mail I'm excited that Scola finds some following in the machine learning community at first it seemed surprising because I thought that machine learning demands very conservative resource usage in particular memory and that therefore C++ would be more suitable but I think what we've all found is that number one in machine learning having a kind of degree of flexibility and composability uh can be um really important because people often don't build cookie cutter models but want to put them together in different ways and that's secondly something that runs in the jvm is not you know not necessarily that slow and and furthermore actually the biggest speed Speed games can also be had made by algorithmic differences and not necessarily uh um uh low-level considerations and that we've certainly found that um ourselves in some ways that actually describe later okay so Factory uh it's not that big right now I don't know about 60,000 lines of the Scala uh it's used at several universities and a few companies uh um not as widely as Mallet yet and it unlike Mallet it's designed to handle arbitrary Factor graphs I'll I'll talk about Factor graphs in a minute but they're essentially the generalization of both directed and undirected graphical models or graphical models um I think them as the lingua franka of uh of machine learning most kinds of machine learning can be expressed as a graphical model it does all kinds of General entrance and optimization in these models it can be used for all kinds of applications but since we ourselves do a lot of work in natural language processing it has extensive uh pre-built natural language processing tools it also does some topic modeling and uh actually in a way that's faster than mallets now and also um the kind of um word Vector building that the Google brain team has been doing uh um some of those the simpler versions of that have been implemented in Factory as well um okay so first I want to describe make sure that we're all on the same page about what a factor graph is because it's the key uh um uh basis on which the rest of factory is built so uh let's start with a short tutorial on that and let's uh let's start with a a simple task and that's just uh item labeling or classification we say we want to do spam filtering so what does our data look like we have a collection of email messages and uh because we're going to be talking about problemistic model so all of our data is going to be captured as a random variable this is an observed random variable we're given this we know what its value is it's some collection of text our other random variable is the label that we're trying to produce this is the predicted random variable associated with each spam messages I'm trying to say is it spam or not spam and how am I going to accomplish this and The observed email messages uh message and uh and produces some real valued score where higher scor say say I um I find these values uh um uh more compatible and these black boxes we'll call factors right so so again they're a factor is like a compatibility function a function that takes as its arguments the values of the variables to which it's connected through these edges and it's responsible for returning just a real value and uh so this is a factor graph and probabilistic inference then consists of going to the variables whose values you're supposed to predict considering all their different values finding the ones that gives you the one that gives you the highest score and then that's the one you say would is the preferred one according to the model all right so here's one simple one let's consider another task Chinese word segmentation because you know Chinese doesn't come with a spaces between the words uh if we want to do natural language processing we need to find those boundaries uh ourself so our observed data are again the Chinese character is in sequence we're trying to predict which character is the start of a word and which one is not the start of a word in the middle or the end of a word and we could build a model like this that measures the compatibility between individual Chinese characters and whether we think it's the start or the middle or end of a word but this actually is not so great because they're really strong dependencies sequential dependencies among the labels um whether I think that um this thing is a not start label may depend on what label I gave um uh to the previous character and so I should add some compatibility functions like this these correspond to the transition probabilities in h barov model yes sorry CH maybe not the hard language for segmentation um so let's see I'm not an expert in these things I did at one point build a Chinese word segmentor uh um I can I believe you that Tai uh um is harder yeah absolutely yeah um okay so here's a more complicated Factor graph inance now just became more difficult because before when searching for the highest scoring configuration I could look at each variable independently and um just cycle through the two options find the one that that scores the best and then move on to the next but now they're all interconnected and although one particular choice might be locally the best for this one character it it might not be best in the interconnected chain and so naively I need to search through all possible combinations of the entire sequence to find the one that scores best and there are two to the fourth of them here and in a sentence of say length 20 there would be two to the 20th that sounds pretty bad but luckily there are efficient Dynamic programs that uh will find the exact solution uh in much less time than that okay so now you know what a factor graph is um so however we care about models that are much more hairy than this so like we look at problems like information integration like we have 300 hospitals and uh that all have databases in their own schema patient databases we'd like to pull all those together into one schema so that we can do some analysis of pattern analysis of patient outcomes um so we need to align The Columns of these tables with each other we need to align the rows of these tables with each other and we need to decide what's the canonical value that we're going to put into the cells once we get all this alignment done and we also build graphical models for doing this for doing say the co-reference in canonicalization and we build other graphical models for doing the schea matching and then actually you can do a much better job if you don't do these sequentially but do them all together in one big joint model in which you're kind of simultaneously doing row and column matching at the same time and you know it gets really hairy and and this brings up all kinds of really juicy scientific machine learning problems and parameter estimation and insurance but besides this even just the software engineering is hard and my students were finding this we'd have some new idea about some problemistic model we wanted to try and it would take two months for us to implement it and get it debugged and get the results and of course the nature of research is that you try one model and then you have another idea tomorrow and you want to try another model and it was really slowing down our research so this got us interested in how we can make this process go faster and in the uh the vering sub area of machine learning called probabilistic programming languages um so these are tools that whose goal it is to make it easy to specify some rich complicated problemistic model using the full power of the kinds of things we're used to having in deterministic programming languages good data structures and control mechanisms and abstraction and you defined your model and then the idea is that after that inance the inance implementation comes for free it's just provided by the library that knows how to work with your customer model to do the entrance in the right way um and uh um Factory is an example of this but I want to put Factory in the context of several other tools because this is a rich and uh and growing area and I'm not even going to list all of them but uh just just a small sampling of them so a quite old tool that's been widely used by statistician is called Bugs which stands for basian inference using Gib sampling and as it's an ad Advantage it handles arbitrarily structured graphical models um but it was never really designed for large data it was designed for statistician small problems so it's kind of small small and only works on small data um there's an academic tool called alchemy that comes from University of massach of uh of Washington uh that's con has a very convenient concise modeling language um although I haven't used it myself but from people that I've talked with they're finding that it sort of has unpredictable efficiency um and has been tough in that way there's graph lab and I know we have people from graph lab here and so this is a delightful super scalable parallel distributed uh uh um set of Primitives that are based on um computation that sit on in the nodes and on the graph and uh um on the edges of the graph and um and has some great advantages The Primitives however sort of like The Primitives of map produce don't are defined separately from any machine learning uh uh considerations and so they're not especially tightly intertwined although I understand that that the tools there are getting U built up with a an increasing number of machine learning models and then there are other quite different tools like the Stanford natural language processing toolkit which in Academia anyway is one of the um the standard tools that are that are used for natural language processing and it's it's been extensively developed over a long period of time it has a very rich set of natural language processing data types and models um but it's kind of an ad hoc collection of multiple implementations all separated from each other um you know without some unifying principles that would make it easy to build say a slightly different NLP model by composing pieces of other models together so Factory one of the goals of factory is to try to get the best of all of these options um to pull them all together so um I really want to make sure not to just give you a positive sales pitch and to tell you about factories some of factory shortcomings as well so let me uh let me describe those also in terms of some other tools there's another academic tool from Columbia called Stan that does beijan inance using sampling and it really emphasizes and does a great job of real of inance for on real values real valued variables and um Factory really although it has the capability to represent them and as a few tools for handling real valued difference our emphasis so far has only been on discreet data we could develop out this area in the future but it's not done right now um Stan really focuses on this Markov chain Monte Carlo uh kind of sampling methods to do its inference and uh to my knowledge does not have variational inference or message passing inference okay then there are like mahot uh that has Hadoop that will scale uh um in very nice ways and Factory although we've applied it in various scalable distributed ways say we've run it on distributed across a thousand machines to do some uh um some integrated inference problem it doesn't yet come with pre-built tools that make it easy for you to do that somebody uh a student of mine rolled their own distributed arrangement in order to make this work across a thousand machines um and mahoot is going to do a better job of that on on the other hand well let so I don't know what you want to say about that uh um draw your own conclusions pyit learn is a delightful toolkit for machine learning and it's super mature has gorgeous uh documentation and Graphics built in and Factory is just barely 1.0 has no Graphics integrated and and actually you know the documentation in Factory you could use quite a lot of work actually um uh psyit learn is in Python and c and it's kind of uh implemented as a bag of methods that don't necessarily their implementation don't necessarily have too much to do with each other voral wabbit is a fantastic uh um tool for uh large scale classification done by John Langford it's written an incredibly tight efficient C++ and you know Factory is just less efficient than that uh part of that is due to C++ sort part of that is due to just John langford's Brilliance in uh um some of his tight coding um foral wabbit is really built for binary classification and that's what it does as its core it provides some tools for doing things like multiclass classification but it's actually it it takes multiclass and divides it into multiple binary classifications to get there and we're striving to solve a much larger speed of problems okay so let's uh let's get into further details of uh of factory with just a yes please thank you sorry you also cover the differences with let's see I guess thank you for it more accurate there are a few very special cases of things that Mallet does that are not currently covered in Factory those include things like um uh finite state transducers with a very large number of states that may have very sparse transition matrices uh and Factory doesn't uh keep track of that sparsity and leverage that at inference time uh but that's actually something I want in some of my own work so it's something I'll probably be adding in the next few months yeah thank you for the question I'd be happy to have other questions too um so it Loosely stands for Factor graphs imperative extensible it's implemented as a library and Scala as you know we chose to implement it as a library not as a new separate little language because this lets us write our models right in Scala it lets us integrate the code that we write for data pre-processing and evaluation and uh um and Diagnostics right in the same file and language in which we write our model specification and it lets us leverage object-oriented design for defining how the model works and how you can I don't know extend or modify through inheritance uh um various parts of the problemistic model it's object-oriented in that uh all of these things are implemented as classes and as objects that can be changed through inheritance variables and factors and inference methods and learning methods um are all objects uh it's flexible and that it's uh unlike some of the other tools I talked about earlier it's it can handle both directed and undirected graphical models both variational or message pring style of inference and sampling styles of inference and it's um scalable in in our experience anyway so far like we've we've run models with billions of variables ones with a super exponential number of factors that are actually never instantiated uh before inference starts but only like instantiated on the fly on demand and just the little places within the model in which that's needed it's been plugged into mongodb backend to run on more random variables that can fit in memory and um although it's maybe not as fast as V web it it has it's reasonably efficient in classification sequence labeling and a lot of other standard uh tasks and it's released under a license that you can use in a corporate environment the Special Sauce that I think about here is that some other tools um like Microsoft research in.net which I I didn't describ earlier but is also a fantastic package for inference and graphical models um that confer net is incredibly powerful but inference INF for.net is just a black box it's actually handled um as a compilation process to uh um it sort of compiles the model into code specific to your inance request and you know my experience or understanding is that uh if you want to modify the way entrance works here because you have your own ideas about how to do entrance a bit better you know forget it it's pretty much just penetrable by its original Architects whereas Factory is designed hopefully to make it easy for users to descend gently in layers of abstraction like if you don't know any machine learning uh you don't know how a topic model really works but you want to use latent darly allocation there's already a pre-built one that you can just call from the command line uh as there is in Mallet um if you want to use a model that mostly pre-exists and just change a few little things about it you can probably write just like a little 10line script they will uh to get the components you want and plug them together in a slightly different way uh without um without too much difficulty uh if you want to make you know um um you some more significant modifications maybe you write a 100 line script and get what you want or if you want to all the way at the bottom you know Implement your own new inference algorithm that somehow Works in concert with the other existing inference algorithms um you know you can you can get there again by you have to learn some more that's a bit more complicated but it's not like falling off of a cliff um okay so when you write a model in Factory uh what does it look like it comes in certain distinct stages and in part of our architecture we designed these to be very distinct in order to get a certain kind of flavor of modularity in the way we as machine learning people Define our models the first stage is to Define templates for your data or think of these as classes of random variables they're going to hold the data on which about which you're going to be reasoning and um here you can actually often use data structures like you would in deterministic programming so in in ways that you would think would be kind of strange for a graphical model you can have random variables for things like sequences or um or sets uh um or things like this okay so then you load your data into these and now you've essentially defined your problem in a way by loading the data in you have way to express both the input and the outputs the next stage you define templates for the factors or these this is the model right this is how you're going to express which values uh in concert with each other do I think are the ones that would score most highly or the ones that I would prefer and this is very distinct from the above data representation making it easy for me to Define my problem once and then if I want to brainstorm about different alternative models I can just change the localized code for that without changing the RoR one um and uh um yeah so I write this part and then I select some inference method then there's a large suite of uh different inference methods already provided uh then I select some learning method about which I'll talk a bit more and then I I don't know I just write code to read in the data create the random variables and then inference and parameter estimation are often on liners okay so I want to show you some code and I'm almost there but uh let me actually show you the names of some of the classes along the way to we give you a broad sense of uh of what the architecture looks like or what's available all the words that I'm about to list are all the names of classes uh um in Factory so there's a variable which is what holds your data and a variable has a value and has a method for setting the value and many variables also have a domain so domain is something that can be associated with a variable that since he says something about well what's the set of values that are allowed in this variable um then there are various kinds of labels or actually traits sorry on variables uh a so like a a labeled VAR is a kind of variable that um also has a human defined um true Target value and we would need to use this during uh during training uh um U to know the True Value that we should be using in our objective function there are things like diffs and diff lists that help us um almost like a transaction make marov CH Monte Carlo change to the world and then if we want to uh reject that proposal we can undo and roll back to where we were before there are factors as I've described before there there are things like families of factors that um are repository for share sharing many pieces of uh that would um things like parameters that would be shared by many factors there are templates which are ways to create factors on the Fly According to some certain recipe there are models which is a source of factors uh given their neighboring variables there's a marginal which is the result of inference like a distribution over just a smaller set of variables uh marginalizing out the other variables in the model and then there are inference methods and then there are three things that I'll talk about later optimizers examples and trainers and so we have various kinds of variables for holding everything from just integers to tensors like you know vectors and matrices sequences even random variables whose values can be Scala pointers which is what a ref variable is um okay so let me talk about learning and then I'll show you some code um so here's the classic very general equation for learning Theta is the set of parameters over which we'll operate um and I want to find the parameters that score the best According to some loss function uh and some regularization so sets of parameters like our parameters may be spread across many different factors and many different factors of families and there's a weight set is something that pulls together all of those parameters so that they can be treated as a whole by the inference and learning algorithm and preserves their mapping back to the factor families from which they came and this is what enables Factory to um yeah to operate to apply essentially the same learning code to many different kinds of graphical models of many different structures with different numbers of variables spread across different factors in different ways okay then all right we we want to we need to search across the set of uh parameters that will minimize our loss function and that's done by optimizers based on uh climbing some gradient and there are various methods of gradient hill climbing like conjugate gradient a pretty classic one that you'd learn about in school to things like lbfgs which is a quasi Newton fancy gradient climbing method that can work better as well as um some very fancy High performant online methods as well okay then we have the loss function um and these are encapsulated by what we call an example which is a little wrapper around a a little piece of data think of it as like uh um I mean in the case of IID classification it would just be the one email message and the one label that I'm trying to predict it as being spam or not or it could be a whole chain for an linear chain CRF for doing Chinese word segmentation so that data is wrapped together with a model and with a loss function that can deliver a gradient the job of an example is to give a gradient um that will eventually go into the optimizer and the thing that sits between optimizers an example is something called a trainer which is essentially for responsible for essentially scheduling the different examples and how they move into the optimizer so for example in batch training you would take all of your examples get all of their gradients sum them all together and then in one big chunk give that one gradient to the optimizer that will take some step in that direction but often it can be faster to do online training in which I get the gradient just from one example give that to the optimizer even though it doesn't really reflect the global optim optimal Direction overall it still tells me some direction to go that will be beneficial somehow and I may zigzag my way to um to the best place but I can make a lot of progress rapidly before I've even finished going through all of my data um and Factory allows us to mix and match these things in a way that's very flexible that's going to come up when I show you code okay so here we go let's look at some code and um I'm going to follow through a suite of different kinds of models aiming to show you how easy it is to mix and match some pieces and let me say also that this is I'm showing you building a model in a very raw sense and in a lot of cases if all you really cared about were classification or linear sequence labeling or topic model those things are already pre-built and you wouldn't need to get into any of this complication it would just be provided as a oneliner but here's what it looks like at a more raw level okay my first stage is I need to Define some Rand some classes to hold my data so uh I'm going to build a binary document classifier here that's just going to say yes or no to spam so I need one random variable to hold my document and here it is and it's going to be a feature Vector variable so it's going to store essentially um right you know counts of words that appear in the document and then my other kind of random variable is a label and it's a Boolean variable it can be either spam or not spam and because I'm going to need to train on it it'll be a labeled variable I'll know it'll be able to hold both its current predicted value as well as what its true value should have been okay and that's all I need for my data so what's next um like I somewhere else I'd write some code that then knows how to read my raw data in off of disk and then create all these random variables and that's what I I will have done here I get I'll get back a collection of labels and each label is actually connected to the document um uh that's associated with it okay now I need to Define my model and the key abstract method in model oh sorry so the model is going to need some factors in order to produce these scores and here are the two relevant factors I would want the uh the compatibility between the spam label and my feature Vector that's the email message and I'll also I didn't draw this before but it's often also helpful to have an additional factor that doesn't even look at the evidence it only looks at the label it provides a sort of bias well in the absence of any evidence you know most email messages are I don't know spam or not spam depending on your stream um okay and then these are defining families uh of factors that say here's one that labels that neighbors just a single variable here's one that neighbors two variables and they are factors that will produce their scores by doing Dot products this is what the dot part means here and then the other abstract method I need to produce is one that given a variable will return the set of factors that touch that variable so here's a function that's given a label and it returns a list that includes just two items one is a factor from the bias family and another is a factor from the observation family and I've just given them their proper neighboring arguments and now I'm done I've finished defining the model so this as an aside here and this will come out more later I can Define the structure of my model by writing procedural code here which which gives me incredible flexibility to implement some pretty complicated model structures that can be value conditioned and and and all kinds of things okay so now I want to do learning uh I I guess I'll start by deciding to do batch training and uh send my gradients into lbfgs quasi Newton gradient climbing method and then I'll ask that trainer to process um some examples that I'll create and I'll create a an example whose gradient comes from the maximum likelihood criteria uh using this label and uh and this knows how uh um uh um using the model to get what it needs in order to produce that gradient and then I can do inference say well okay on some piece of data Let's uh let's see what the model has to say about this and um yeah okay and and this already is is everything that I would need to uh train a bomic classifier so let's see I've already made some change let's say I wanted to have faster learning by doing online training I can just uh here before I had batch training and I can just change this to online without changing any of the rest of my data or my model and the rest of it will just work uh so that could be faster even better than stepwise gradient descent might be to take that gradient and put it into adagrad uh um um which is a more clever method for gradient he'll climbing or or maybe another method called adagrad RDA which uses an L1 prior which causes many parameters when it would not hurt accuracy to automatically go all the way to zero and actually we use this to make our part of speech tagger only 10% to the size that it would have been otherwise without losing any accuracy at all um and again I just had to change one little place in the code to make this happen uh let's say I want to train even faster and I want to do actually do training in parallel across uh um many core all at once um there's a method called Hog Wild that can make that happen and again without changing any of the rest of my model definition of the rest I can make that change um let's say that instead of doing binary classification I want to um classify documents uh into 10 different possible classes I can say that I want a categorical variable instead of a binary variable and uh and not change any of the rest of my code and uh and and it knows what to do with it um let's say that I want to uh get my Radiance from a method that knows how to be uh to do inance across a wide variety of models including ones that uh um uh that might be loopy or have a very complicated structure and the sample rank is a way to do this um again I can make that one change and then let's say that I want to actually operate not over single variables but over sequences as we would in the Chinese segmentation uh task and so I guess here I've said that um what I'm you know my data items consist of sequences of labels not individual labels and but I still actually didn't I mean I've changed the model pretty drastically and the rest of my training code um just magically uh worked in this environment this is not a great Factor graph for that model in the same way as the first Chinese word segmentation model was I have no Mark of transition factors in here so I can add them by saying let me have an add an additional Factor here that captures my mark of Transitions and then have my factors uh method step over the sequence of labels and um and create factors connecting successive uh labels in the sequence um and then I could do inference in all kinds of different ways again without changing anything about the model by Gib sampling or by specializing the Gib sampler with uh um with my own specially subsampling method or by mplp which is a message passing uh uh inference method or by belief propagation and again this is all done by a mix and match including just wild crazy you know much more comp licated models like ones that don't just have a chain of factors along the sequence but actually these extra long arcing edges um that turn this into a loopy graph uh maybe in a way that for the sake of time I'll uh I'll skip over um okay so now I want to get into uh what we do um in NLP and then I'll wrap up um first let me just say that we have pre-built tools for classification for sequence labeling lots of parts of natural language processing what all describe in a minute for topic modeling like plat der iocation that's actually faster than Mallet um for uh word embeding so for example we can learn semantic capturing real valued 300 dimensional vectors associated with words by training on the entirety of Wikipedia plus all of Giga word which is a large set of uh newswire articles on one machine in less than six hours um um so I guess with that in mind maybe I should tell you just a bit about efficiency so here we are comparing against some other tools for um just simple IID classification so in a way this is not leveraging factories graphical model infrastructure much at all um but at least you know we want to be not bad in this face and in the face of some of these other tools and um the results are that we're kind of within one time uh um within uh um the speed of a lot of these other tools um except for Leon bou's uh um stochastic gradient descent which um yeah which is just uh crazy uh I mean I know that's that's that's crazily engineered um but you know we we never expected actually I mean it's also written in C++ and their rest and you know being running on the jvm and and really having this very GLE infrastructure we never expected to be uh you know faster than the fastest um here we are in comparison with some other sequence labeling tools these are all um essentially linear chain conditional random fields and uh it really surprised us that we were even just within a factor of two of CRF Suite which is a C++ tool that again was just engineered to death to be super fast and one thing that I find interesting here is that although the time for one for one pass we're about half the speed of CRF of a CRF Suite in the total time for training we're actually dramatically faster than they are and in a way this is not just a a simple direct fair comparison because really it's a convergence issue we converge faster and this has to do with just the way that we set up our learning but in a way it just points out that having tools that make it easy to reconfigure your machine learning to tune it in various ways can really help you do things like get convergence uh issues right and that these can make more a bigger difference in the end um okay so in NLP we do we have representations for documents and sections of documents and tokens and Subs spans of tokens within a document and sentences uh mentions like noun phrases like David talked about and entities for doing entity resolution parries and as tools for doing all the things in the right and I might give a short demo of some of these at the end if there's time um a few months ago we decided to compare ourselves and accuracy and speed against the Stanford core NLP toolkit and we did not expect to do as well as them because they've been in development for a long time and they're really tuned and written from the beginning especially for NLP and we were pleasantly um surprised so in part of speech tagging we found ourselves to be more accurate this is token accuracy 96 versus 90 uh um in the High 96s versus the low 97s and in speed about four times their speeds we're running at about 110,000 tokens a second in part for part of speech tagging um in parsing we found ourselves to be a bit more accurate and dramatically faster uh like almost 2 and a half thousand tokens per second uh versus 47 here um and in Neer we found ourselves to be more accurate not currently faster although that's something that we're working on and we also have a new research result that's actually a state-of-the-art in a particular Benchmark uh data set uh um that we've been working on just recently okay so I'm just about done let me uh just give you a little sense of uh um of of speed here so I'm going to I'm going to train a linear chain conditional random field on um the problem of uh of named entity recognition So within a newswire article finding the instances of all people organizations locations and miscellaneous uh named entities and this is on a dat set there was part of a competition that was run in 2003 um I participated in that competition in 2003 and uh um uh using Mallet at the time actually and I remember to train in this data set with our it it took us about um about eight hours to train our model so let's see I think I should have about eight hours left here right before you all have to go um okay let's let's see what happens so uh um so let's start this training now so it's going to first read in about oops oh no how can I make this bigger uh about 200,000 tokens of training data and about 50,000 tokens of testing data extract all the features okay just first it's first passed through the data of training and parameter estimation on I forget is about 400,000 features it's now made it second pass and is already up to 94 okay that was its third pass um and okay it's finished okay so we finished training a named entity recognizer on 200,000 tokens of label data in 18 seconds got 95% accuracy on the test set and then uh and printed out a bunch of Diagnostics along the way so here are a few examples of um of named entities that it extracted um from just the first section of the data so like uh um petj Blackburn is the name of a person and Germany is a location um and EU is an organization uh and so on and actually so in this Set uh we actually got them all 100% correct I guess in in the previous pass we were getting a few wrong but by the time we finished the last pass we got them all right so that's a sense I have other demos that I could show you but I I think I might be running over time and I want to make sure I get to answer your questions so let me stop here take your questions and uh you can uh I could show you more demos if you like thank you yes I'm sorry could you talk a little louder yeah so we have we have no visualization either for results or for the graph itself you know we're um uh uh you know uh we have not designed Factory I think for um for use by people who I don't know don't already well know their machine learning and know what they're doing although actually no your question is separate than that you could know what you're doing but still want to visualize it and uh no but I'm sorry we don't have any uh any visualization that would be a great sub project I would love to uh to merge in the project that did that sorry thank you the question was do we have any visualization of the structure of the factor graph uh um and we don't yes well what's next okay yeah so we're continuing with Factory for quite a while one of the next major things that I want to look at there is um infrastructure for parallel distributed processing so like I said I have a number of students and others who have successfully used Factory in large scale thousand machine par distributed uh um environments and gotten great results but they rolled their own infrastructure to do that and uh um and I think that there could be some great advantages to building parallel distributed infrastructure really baked into the same uh the same library that knows intimately about the probabilistic model and can reason about how to distribute the data and the computation at the same time as it's reasoning about the structure of the protic model and in a number of um pieces of research that we've done especially my student Samir Singh who's actually doing a post talk with Carlos gestrin at udub right now um found that reasoning carefully about the problemistic model uh to decide how to do the distribution could make multiple orders of magnitude in difference in speed so for example we've done a lot of work on um uh entity resolution or or co-reference um say running on you know like a half a billion mentions uh um distributed over a few machines uh and um yeah and reason in a very tight-knit way about the protic model and how the data is distributed to get some big gains um so that's one area um you it's possible that if we have more users who um yeah I don't know are more like companies and less like graduate students that more visualization or some other polish would be helpful and maybe this is an opportunity for me to say that uh yeah I'm I'm I'm just I'm super thrilled that factory is getting usage by as many companies as it is already and we always happy to have company users and um and and in and if you ever use Factory yourself don't be shy send me email I would love to hear from you hear about the problems that you're applying it to and uh and yeah we'd be happy to try to help you as well yeah High question building live talking we including a a virtual a sales virtual assistant from virtual assistant for for sales people like Siri you know for sales people and say I've used new with speech recogn and sent and I use can I use Factory to build that virtual assistant you know according to my requirement yeah so that's a pretty high level question I mean it's going to depend and how much you're going to cast all the rest of that problem as a probabilistic model and whether it's well suited um um to this in particular I mean I could imagine some steps of that modeling problem that would be well cast as a machine learning problem and um uh then I imagine the answer would be yes I'm sorry the question was in in in building some Salesforce Auto uh automation methods would Factory be a good tool and um I can imagine that it would be um yeah I'd be happy to talk with you uh afterwards about the degree of fit yes it's it's a common pastime of functional programmers in schol and other languages to look for patterns forar alph probably right yeah understand Factor correctly other contexts um we can multiply factors which would form a multipli yeah from the code that you shown I don't know that that identifying that explicitly would make much difference but the good that's that's not something that I thought about really uh um before um let's see sorry the question was a lot of people who work in Scala or I'm quite taken with the the monoid abstractions and um would that interact well um with Factor graph structures and um not in a way that's obvious to me off the top of my head but that sounds like an interesting question uh yeah yeah um I uh I don't know if there are people here who are interested in some of our thoughts about Scala itself uh um uh there's certainly a lot of things that we love about Scala that have really helped us we've made extensive use of traits um to multiple pieces of functionality into one place like a a random variable that holds a token well it's a categorical variable because um the token is holding a string and those fall into the different categories in the dictionary but it is also a variable that is the link in a chain of a sequence because it's part of a sentence and we want it to be both at once we've made use of implicit parameters as well especially for these diff lists so like when you set a variable if there's an implicit diff list around then we'll pick it up to automatically build up this transition structure or this uh transaction structure sure we've made use to function objects to create some nice syntax foring making uh generative uh models we've used implicit conversions in a lot of places that have been really nice and we've made extensive use of the cake pattern in building to parts of our models and part of our inference methods and um so on some things that we would still like from Scala is maybe just a little bit more efficiency and I don't mean jvm issues I mean some other things um I mean uh a few years ago in the Scola mailing list I remember some people talking with Martin about making things a bit more efficient and part of the answer was well you know we want we don't want to do premature optimization in areas that won't really matter and I feel like a little more optimization would no longer be premature where sometimes kind of scared to use the scholar collection library because it has some surprising inefficiencies um um I mean and there are multiple places where even just the old Java collection uh classes are actually significantly faster than there's no reason for that to be true it shouldn't be true um it would be really nice if these things didn't box anymore and I know people are working on this but my understanding is it's I don't get it for free right now I um uh and then just like some more predictability it's like we might think that okay I just want to call the Signum method on this and that's like an incredibly that's a single instruction right on the CPU but if you look at what Scola does to this it boxes it and it's just this is just crazy and so the uh um and I understand how this accident could have happened but uh yeah some predictability would be nice okay let me uh collapse that I would love to have Constructor arguments on traits if for nothing else other than to get a class tag uh for um a type argument to a trait uh there are numerous places where I really wanted to use that and this was even previously dis uh discussed I think sometime in the late 2000s actually but has still hasn't arrived um and then another maybe this is a bit esoteric but something that I really would have loved in certain parts of the creation of our class hierarchy for factors and for models is a linearization of inner class inheritance so in this first sub bullet you know if you say trait Fu two extends through one and then um you know the meaning of super. Fu inside of that is determined by the linearization of the traits the order in which many traits might have come but that same linearization does not apply um to class extensions so if I in inside some inner class say you know Fu extends super Fu where this is the inner class of some super class this superfood is not linearized according to uh the trait ordering um and uh and this just really bit us and uh yeah anyway so that may be a bit esoteric yeah so I think that that's uh that's all I have to say about that other questions so I'd be super Happ yes thank you question so uh so spark is now extremely popular in in Scala big data and yes they came up with a graphx library and they claim to be able to implement any other gra formalism in few doz of L so I think they they came up with a way to implement graph lab approach in 4 Lin and so forth so did you look at that and how would that compare to to factory right okay so I've not looked at this but uh um um but I should and I'm uh I guess I gather that um the the guy who conceived of spark is supposed to be arriving at MIT sometime this coming Academic Year and so I'm looking forward to having him as a neighbor uh and um I guess without knowing the details um you know having abstractions for computation sitting on a graph is a very helpful building block for doing inance in graphical models but there's a lot more to go on top of that and um yeah I would probably be interested in determining whether the kinds of future work we want to do in Factory for better parallel distributed infrastructure um could be built on on top of this um yeah thank you nice question oh yes is M only option okay there is no um special privileged position for mongodb as a choice uh um with Factory um for in a lot of our own work we actually use mongodb for serialization but serialization code say for different pieces of factory is not is you know is unlike ja not unlike Java not baked into the classes themselves but actually written externally sort of like in the Scala pickling Library uh and um so it would be very easy um to build parallel infrastructure um as an alternative uh toongod DB that did something else instead and I know that there are some people some who are using Factory extensive ly who are instead uh building on top of a cumulo uh um uh instead yeah and I'd love to talk with anybody who would like to BU I mean I would be actually eager to have um you know publicly available uh advertised uh um a code with you know various different nosql backends um that factory could interoperate with yeah thank you all right I'd be super so there's my email address I'd be super happy hear from uh uh from any of you um stay in touch thanks so much