BDSBTB 2015: Sameer Singh, WOLFE: A Declarative Machine Learning Stack
Recording: BDSBTB 2015: Sameer Singh, WOLFE: A Declarative Machine Learning Stack
thanks for the introduction Alexei can everybody hear me not a great hi yes I'm Samir and this is joint work with Sebastian and Tim who are over at London sorry for the change in the schedule maybe some of you are here expecting a different talk but sorry about your stalker so quick introduction I do research in machine learning and NLP in an intersection and specifically I look at large collections of documents and try to see if we can extract some information out of it that sounds kind of way but what we actually think about is putting that stuff into a database and this is my research summarized in three parts but what this means is that I have become both a user and a researcher of large-scale machine learning and so during my PhD that Alexei briefly mentioned i worked on factory which was a machine learning toolkit for a certain class of machine learning models and it's has a quite active user base and they've been developing it since i left but today i'm going to be talking about this new toolkit that i'm working on called wolf and the main goal here is to have a language that is universal in the sense that it can represent both current and future machine learning techniques its expressive in the sense that it's easy to write and modify models in the language and it's modular so you want to plug in pay play you won't pick your favorite algorithm and be able to use it within wolf and I'll also briefly talk about morrow which is a software engineering side on trying to create ipython like notebooks in Scala ok so I'm going to start with a brief application that many of you must be familiar with just to sort of motivate wolf and the kind of stuff that we do so here are four characters who like watching movies and here are five movies and what happens is you know each of them goes and sees all these movies and gives the rating you know red is bad Green is good and then the second one comes along and gives us separate set of ratings the third one comes along and the fourth one comes along and you have a bunch of ratings for a bunch of movies but you see that there are a bunch of missing holes there and you want to finish this matrix you want to figure out what does the professor like does the professor like monsters inc or not does Professor like finding Nemo or not and so I mean it should be obvious from just this example is that the professor seems to be pretty close to blossom there and so he's likely to like the first movie and not like the second movie and this intuition is kind of captured by matrix factorization which takes this matrix this sparsely observed matrix of ratings and kind of approximate sit with a low rank matrix and so you get something like on the right here where you have matrix u & B which are much smaller than the original matrix but they are much denser and they kind of represent the similarities between users and movies and so if we want to write it out mathematically what machine learning in this scenario is actually solving it's just a summation over all pairs that were observed where X IJ is the observed rating and you kind of want to see how close it is to your approximation and if it is pretty far you want to sort of change your approximation to make it as close to the truth as possible okay so that's great this is a very useful application and you can also use existing tool kits to solve this you can pick scikit-learn and it has matrix factorization but if you are here you're probably not going to pick scikit-learn you'll probably pick something in the JVM maybe Becca or mahout or maybe malate which used to be around and if you're doing scholar maybe you look at spark and my lab or factory or the bunch of other tool kits right and that's great they can solve this problem they are pretty efficient of solving this problem and they usually support a whole class of algorithms so you can you don't just have to find one technique you can sort of play around with a bunch of thing so this is great you know you pick up one of these two kids and it solves the problem for you but there is a problem the problem is that machine learning doesn't really work until it does and the point is that if anybody has doubled in machine learning you kind of have to repeat repeatedly bang your head against it and at some point you don't know what you exactly you did but suddenly everything is working and you want to get there as soon as possible so the sort of things that you usually have to play around with for example our stuff like feature engineering you have a bunch of features you want to design your own features and mix and match you have a choice of algorithm you know and you want to run all of them and compare all of them and they perform very differently from each other but then also you read some paper or somebody tells you about this new method that you also want to try out and there's just no code available for it and then there are lots of other components inside each machine learning algorithm like regularization link function loss functions blah blah blah blah blah that you all you can all modify and sort of play around with and some certain combination gets you great results other combinations don't and the problem is that it's not just three different categories they are actually orthogonal to each other so you have to try all combinations of all three things and as a simple example of what you might want to do when you're doing machine learning is suppose you have an example like this where you have ratings for the three Powerpuff Girls but for professor you only have a single rating which is that he doesn't like Incredibles now that might make you think that he is kind of like blossom but one rating is kind of pretty noisy but maybe you know that professor is really good friends with bubbles and that they should basically have similar rating behavior sometimes they differ like in this case but otherwise they don't now thinking about it mathematically all you need to do is have a penalty to your objective function that says that you know don't don't make them very different I think for whatever reason that they should be similar just make them similar right unfortunately none of the existing tools that I outlined before can handle something even that simple as this now this is not cutting edge research I just thought of it in a few minutes but it's a pretty useful thing and think like these keep cropping up when you're doing machine learning and so this is the problem when you're using existing tools you might come up with a feature it's possible that the tool doesn't support the existing feature but even if it supports the existing feature it doesn't support it with the certain other feature that you're interested in and then for every tool kit that actually supports everything you want you have to kind of learn the API and there's a whole range of variance in there you have to port all your data structures and your models into it and then in the end you're just still hoping it works it's it could be all wasted effort and in the end a lot of times you just go like I'm just going to implement it on our own and that's often a bad idea okay so far it's pretty clear i hope that machine learning we can't use machine learning as a black box right you can't just throw some data at it and expect it to work but we also know quite well that machine learning is quite complex to be a white box you they have you know decades of research going on and people still don't understand what works what doesn't so you don't want to allow the user to tweak everything and what we were looking at with this work is is there something in between is there some power that we can give the user but still kind of hide away some of the nasty details in the sense what we need is something that is universal and complete by which it should be able to support existing techniques as much as possible it should allow arbitrary calm modifications to the code but it should still be expressive in the sense that the user it should be easy for the user to set this up on their own task and it should be minimal they shouldn't have to do more than necessary and also because machine learning requires iterative development the toolkit should support some kind of iterative development so with this I'm going to lay out a rough cartoon of what the rest of the talk is going to be like I imagine that's the user initially we're going to talk about language that the user X uses to express his intention then an engine that takes that intention and tries to do something with it and then finally some sort of visualization feedback that lets the user know whether what he thought is actually taking place or not so going a little bit deeper in the language I mean the user is declaring the behavior that he expects from the machine learning algorithm so like in the earlier case we want the two ba bubbles and professor to have similar behavior that's the intention and this kind of decorative definition can capture a bunch of hole models as we will see later and then given this intention the engine is there to basically perform those actions efficiently and scalable and you might require a bunch of algorithms to be able to do that and then finally the user has to be able to make sense of what's coming out and so you want to be able to debug the model somehow which requires visualizing the output so let's start with the first part which is the Wolves language it's going to be a decorative and functional definition of what machine learning is doing so let me have another jeezy code on here machine learning is twenty percent inspiration and eighty percent optimization and what we want to do is have the user focus only on the twenty percent and try to automate eighty percent as much as possible right so we named this wolves toolkit after this guy Philip Wolfe who is kind of the founder of mathematical programming and convex optimization and this whole slew of optimization research and he has a bunch of algorithms that are used very widely in machine learning but he himself is not a machine learning guy so our wolf language similar to the idea of mathematical programming is very inspired by math right so what you're writing is essentially Matt what makes it nice is math is universal you can pretty much write everything but what makes it tricky is you can easily write impossible and inefficient declarations as well but the idea is that the user focuses on what they want machine learning to do and leaves the heart of it to the wolf Angela but it's important to point out is that this is not a DSL we like Scala so much that we feel Scala is pretty close to map already we just had to add some few more operators and the good thing is that since what the users writing is scholar it kind of integrates very well with existing code bases and pipelines and all those kind of things and you know you can use compiler optimization or in scholars case the lac that off okay so let's talk a little bit in detail about the language and I'm going to be focusing mostly on the language in the stock so the the first thing the user defines is the universe of things that you're performing machine learning over right and that's represented as a set of things then we have functions over those universes so a common function that we use is something that takes an item of type newest and maps it to some double and then we have these operators over these functions that take these functions and the universe of things and kind of do other things with that and what I'm going to try and convince you is that all of machine learning algorithms can basically be composed of these things right so let's kind of step through them one by one and see what they would look like so first let's look at what universe of things is basically the space of elements in the universe that you're performing machine learning over is a set of things and so we have some default ones which would make sense like bullion's and stuff our finite categories so you have bullion's set true or false but we can also have infinite and uncountable sets right and this is where I'm going a little bit towards what the user wants to do and not so much about what the user how the use how we should actually do it so we have sets that kind of represent the set of all integers or even worse doubles which is the set of all real numbers and then on top of it you can have collections and you can have functions so you have the first cases all possible sequences of boolean's which can be infinite in length and so obviously it's not finite and then you have maps which is all possible mapping from bullion's to infinite integers so that the latter one is kind of like the family of functions and then you know we have abstract data types as well so in scholars case you can have two pools and we can have all possible tuples but case classes since they're so useful you can have a case class like person named in age and then you can have all possible person objects with all possible strings as the name sorry women and all possible integers as the h and so these are just be kind of defining what the set of things we want to do machine learning over and then we can also use the regular collection methods like filter on sets but in this case we kind of use it there because it's more mathematically so we can say all persons where the name is Alexei and so that's a smaller set it's still infinite because the age can still be from an infinite range but you're just kind of defining your space this way okay the second thing you do is define functions over these universes so most of them take off this forum where they take an element of the universe and map it to a double right and so suppose you are talking about a bug in domain and you want to have a probability distribution over the boolean domain you can write something like this which means you're just basically flipping a coin but then in machine learning you have loss functions and stuff like that so in this case it's taking a double as an input and it's doing X square that's your loss function for some case and the idea is that you're composing these functions from the set of operators that wolf is defining so we have a broad library of functions that you can use you have the standard plus minus blah blah blah but also log in exponentiation which kind of have special meanings in this case these are different from the regular ones but then you also have collections so you can have apply and map on these collections but you can also do sum over all these connections or product of all these things and these are kind of similar to like four left for right but they have some special mathematical meeting in this case and you also have vector operations because we do a lot of vector algebra so you have cross product which is the X you have the dot product you have the l1 norm l2 norm and concatenation of vectors and so on and so now given these functions we have a way to do we provide some operators to actually do something with these functions so I'm going to kind of draw one of these examples where on the x-axis here you have the universe of things so the t is the label but this is the set of tea and somehow you have projected it to one dimension and on the y-axis here you have the function that you're working with so some function that takes each element of tea and maps it to a double so this is a pretty easy part to understand one of the major operators we have is the sample operator which treats this function as a probability density and tries to sample from it so in this case you know maybe you call it four times and the is the answer it will give frame so this is the kind of the semantics of that function we also have an art max which in this case will go over the universe and find the maximum point and so you have an expectation which where you give another function and we'll compute the expectation of that function according to this to your function and finally there is this log z which is basically the area underneath the function and so in this cartoon case you can imagine is pretty easy to compute it but since I've been talking about infinite and uncountable sets all these operators are not well defined all the time which is fine as long as what the user is trying to express can be solved fulfil try to solve in know and so the idea is that the composition of these functions and operators can define machine learning out right and so we will go back to our recommendation systems example and see what it looks like in wolf so just to remind you we have the universe here of movies and people and then you have you and B which are the parameters of your model and those are in some sense part of your universe as well and what you want to do is find the U and V that sort of solves this objective function right and usually when you look at the matrix factorization papers there will be a bunch of more terms but you can imagine how that will fit in here as well so the first thing to do is represent your additional observation matrix just as a case class right so I'm just going to have case class X which is one observation which has I which is the person index and Jay that is the movie index and some score right then we have another case class representing the model parameters which are both you and B and so here we call it W for weights which has you as a sequence of vectors and be as a sequence of lectures and so the domain of both of these will be infinite and countable because they have doubles in there so then what we do is we write our loss function which goes over all your training data and data here is just a sequence of X in just a regular sequence and what we are doing is we are summing over all of them and for each of them we are computing basically this score we are doing the square of the rating of that observation minus the U of I which is w of x dot I and then dot product with VJ which is w dot v and u X dot J right and so you can see already that what you're writing here is exactly math just kind of converted into Scala and to some people this might be even more readable than maths all right but then just get your actual w you have to have this minimum thing also right now I've only defined the some part and that's basically aardman over dome of W loss and the dome of W here is I haven't got it on the slide where it kind of specifies all possible space of vectors sequences or vectors in fact and so now you can see how it basically looks very functional and very nice and this would be a good way to look at machine learning but now let's go back to our example of this case where the professor just had a single rating and this is just the loss and the math from before and what we were proposing of doing was having these two have similar behavior and we are going to do that by adding this kind of penalty term to the objective now in this case we have our loss function which is some function and let's just add another function called preference which doesn't depend on X anymore there's only you so you only have W as a parameter and all it does is subtract the two and then does an l2 or witnesses this basically defines this operator now your new loss is some of both of these and so now instead of doing argument over just the loss you're doing argument or the sum and this way you've just introduced a new feature that does not exist in the existing two kids or even in literature okay and so now this was a simple example of how matrix factorization can be presented this but we've been looking at a whole family of models and you can take label propagation based networks and I'm not going to read to all of these but you can basically go through and imagine all of these models to be written in this certain way right and what excites us a little bit with the recent advances is that this kind of stuff can also represent neural networks right which is nice because especially for Scala they don't seem to be that many toolkits that can work with these kind of things okay so this was the language and so far it's all nice and you know you can write all these models but what actually can was do right and this is the part where we are regularly updating both and getting better and better every day and kind of covering more and more models but I kind of want to talk to you about the ideology behind how we can support all these models and so the engine here refers to the infant's and learning algorithms that are efficient and scalable they kind of taken these programs and do something with it so the engine takes is input the decorative definition of the algorithm right and so the way to think about it is not that there's some function that has returned some value already but that what you're looking at is the expression tree of what the user has written right so it'll be some expression tree or functions and operators that the user used to define what he wants to do now we can run these functions in many cases you know if you're just summing over 10 numbers it's still a valid operation and basically we call it the brute force method which works sometimes but most of the time it's either impossible or completely intractable especially when you have doubles involved that's not possible to do these things and so what our idea is that we take this expression tree and apply transformations to it so that it becomes efficient and sometimes it means that you're being innings in accessing an inexact you're being approximate you're introducing an approximation that didn't exist in the definition but you're doing that so that it's actually tractable and since it's machine learning nobody really cares about correctness they just want accuracy to go well so that works for us so I can't go through all the transformation that we have but I'll give you a simple idea that sort of gives you a flavor of what you're doing and maybe I could be easy to sort of extrapolate that to the rest of it so for example let's take a simple both expression where the domain here is the cross product of two integer ranges right so you have 0 to n and you have 0 to M and that cross is taking all possible pairs right and then what you want to do is you want to do an arc max over this domain FA plus FB right gb so what this means is you want the pair that gives you the highest fa plus gb and it's obvious that if you just kind of go through this since it's a finite domain you can run through it and it gives you 0 of n times m steps and that's fine if and nms smaller but what you can easily do in this case and it might be obvious to many is you can kind of transform this expression by pushing the arc max in and kind of having two different art Max's with a split domain right so this works because your domain was a cross product of things but now you can have two different arc Max's both of which are going over a much smaller domain and so it kind of takes a linear time to solve this all right as this kind of gives you an idea of the kind of transformations we are looking at we are looking at the expression tree replacing some functions with approximate ones but even doing some exact transformation that lead us to more tractable algorithms and so if you're interested in more deeper machine learning stuff this is an example of CRF that we wrote and then you have an arc max over all possible sequences when you are working with a CRF and it's okay if you can't follow this but the idea here is that we're looking at all possible sequences which is in some sense a factored space and we're looking at the definition of CRF which also will look like a factored model in the same way as your domain and this allows us to do belief propagation or message passing in this particular scenario more junction tree or a bunch of different algorithms on the other hand you can have another arc max expression which looks fairly similar except that the functions and the domains are very different you have doubles here but it's actually not that bad because it's a continuous space and it turns out that what you have on the other side is also a smooth function over this domain and that suggests that you can use gradient descent and so what we do is we automatically automatically differentiate this function and try to find the gradient and then follow the gradient to get the arc max and if that function is non convex it will be approximate and all these disclaimers exist ok so this is basically what the wolf engine does I want to briefly talk about the scalability aspects we are currently using efficient single core implementations because you need to get there before you start thinking distributed and then we kind of are working towards supporting user design scalability what this kind of means is that you specify both code in wolf operations inside Park Blocks so that it's easy to distribute but then moving on we are also looking at automatic scalability where we detect in dependencies in the expression three and figure out ways to split it up over multiple machines and sometimes if there aren't any in dependencies we kind of force them which will make it more approximate but still faster I'm still useful for what you want to do one final thing that I do want to bring up is that in the engine we don't again treat it like a black box the user if he doesn't want to care about optimization he's free to just use it as it is but you can also inject knowledge if you do have it so for example if you want to use max buttock BP for solving a specific art max you can easily add an arc max by and give it the parameters you can also do which actually makes quite exciting is have different models that do sample by using different algorithms but then have a composition of those models that kind of uses each algorithm for its specific case then possible this is not always possible yeah okay so now I want to talk about the final part of my talk which is going to be looking at the output of the engine and try to see what the user can do with it and specifically I will be talking about moro which is the notebooks for scholar thing that we have so part of our goal was to have an IDE that we can do machine learning in and it kind of means very different things from a regular IDE but what we wanted to do in general when you're working with IDE is some interactive visual environment for debugging now for machine learning that means very separate things there are actually three kinds of questions that come up when you're talking about debugging machine learning the first one is you have to know what you're dealing with what data input is it what it output is it and so you have to have a way to visualize input and output data which is usually not just to string or something that the second question is how is it happening and if even if something is working or not like what is the decorative definition of how it is happening and so for that you need to surface to the user details of the model what they wrote actually reflects what their intuitions were or is there some mismatch and finally even if they caught the model right things still go wrong all the time and so then there's the question of why did this thing go wrong and you kind of want to give the details of inference and sort of have a provenance to all your predictions when you're doing that so in order to sort of get towards this kind of an idd we sort of started working on morrow which is in some sense a browser-based allow rapid along with some notebook like features it's available as a separate tool kit from wolf but kind of is part of the whole wolves toolkit and you know you have to think about ipod python notebook but for scholar and for people who are not familiar with it it basically allows you to mix markdown lata akima jizz with code and execution but what actually excites me a lot was the visualization of scholar objects so we just have a simple way to extend this where you provide an implicit converter that takes an object of your type and converts it to an HTML string and as long as that is in the scope the visualization just works so since I have some time I think I'm going to quickly walk through the demo that I have I can't quite see the screen while I'm typing so excuse that yeah so this is what I'm doing is kind of writing out the matrix factorization code and so in the top case we just defined the case classes and for both the users and the weight vectors and here we just define an object of the case class but when it renders it out it kind of rents it out slightly in a more structured manner we'll see a little bit more of that then I wanted to define some input to this whole thing we're here the inputs are the input ratings so I have a bunch of numbers here that my pairs of integers two doubles right and we can try to to string them and it basically looks like this which is no more legible than what was in the code we do convert them into case classes and so it's possible to kind of run it and see the case classes which is slightly better by the way this is all in chrome and everything is just running on my laptop but this one looks slightly different but it's basically mapping all of those input ratings into case classes it's still not exactly what you want right and so all I do after that is kind of go through all possible pairs of integers and see what the rating is and then just create this double dimensional sequence and if I run that you see in visualization that's much more closer to what you would want so you have three users here you know a number of users was not easy to find but you have three users here for movies the first one likes the first two doesn't like the second to the second one who doesn't like the first two likes the second to the third one we have a rating for the first movie we just don't know the rest of it and so if we continue we can write our model here this is the same thing that I had on my slides where you just do the dot product i kind of split it in two more functions there's some rigorous a shin here but basically the last term is again the square of the difference between the rating and the dot product and what we can do here and it's not very legible it's just kind of look at what this score function is which does a dot product and in this case all we are doing is calling the function but it returns a wolf object that we can visualize as an expression tree so you can't read this here but there's a dot product and then there's a sequence apply which takes a specific vector and then there are four vectors here corresponding to the for users and five vectors here correspond to five movies so sorry three and four and then there are these constants one and two that basically pick which vector we're kind of working with moving on we can run training just by doing an arc max and the output tensor just kind of looks nasty so I'm just going to get it out this to string and just do a regular case class visualization which kind of lets us see all the user vectors right next to each other and we can see okay you know the first and the last one look quite similar to each other it's minus point eight both of them point eight and minus 0 point for point five and the second one looks kind of nasty then suppose we want to go and predict something we have a prediction function that basically calls the score function and we can call it with individual numbers so we can do 22 we can do to one but all of this again it's not really clear how to visualize this so then we go back to our matrix visualization which goes over all pairs of numbers and calls predict on it and here we can get a prediction so now earlier on we had these three things which we didn't know what they were and now filion and this is just a simple notebook that kind of shows the whole pipeline the rest of the demo here basically adds this additional preference factor that I talked about and a new loss that is some of the old loss and preference and then when you look at the weight vectors now it's come it's trying to make one in two similar so now you can see that one and two are kind of similar they're not exactly the same because the observations are different but you get a similar weight vector and then when you actually visualize the predictions these two are more similar than they were before earlier they were completely different now there you can see some light shades here and dark shades here and they're kind of getting closer towards each of those behavior okay and so this was just a simple demo but we have visualization for factor graphs and all other kinds of graphs and NLP specific tagging stuff plots regular social network kind of graphs and images all kinds of things okay so in conclusion I hope I have convinced you that machine learning is inherently iterative you need to try many algorithms many variations many techniques and it's kind of unreasonable to expect that a single tool kit will actually solve all of them efficiently but we still can want to move towards it and particularly we're interested about using this language as an interface that allows other people to maybe plug in their inference algorithms into the same framework so the words that we introduced both this math like language to express whatever you want and both will try to solve it for you automatically sometimes it will succeed sometimes to fail and then we also talked to briefly about morrow the notebook for scholar the kind of is designed for this iterative programming so a couple of future things that we want to look at we obviously want to continue supporting more models and inference algorithms and we are still researchers rereading papers and implementing them in our free time we definitely want to move towards distributed computing back-end probably using spark or something similar and any any ideas towards this actually would be quite helpful we want to clearly document what is supported and what is not and this is one of the tricky things with language like this is user wants to write something that obviously we haven't imagined yet but then we also want to guarantee to them that they little work and so there's some work to be done there and we want to continue expanding the visualizations or that exists in Morro now so if you have any questions or any comments please reach out to me and get in touch thank you if there are any questions please okay so the question is it's a pretty good question the question is whether once the user writes a model is there a way for user to guide wolf into a better algorithm if the first one is seems to be really bad and that is definitely our goal that's what we want to do which is why we are kind of supporting this visualization of the model independent of the inference as one layer of visualization but then once the inference algorithm comes in how is belief propagation happening how is how well is it doing at what point does it seem to be plateauing we want to kind of surface all these details to the user and that way the user can get an idea of which part of the model seems to be working which part is not working and if he just gets rid of one part will everything just be really fast and so yeah that is definitely the image will go we're there part way yep right alright that's a good question the question is there are a bunch of existing two kids that look a lot like Mauro and I Python like Jupiter and spark people have their own and bunch of others that I'm not familiar with but yeah so when we started morrow it was about maybe two years ago where none of these things were at least not public and Mauro for me is becoming a lot more of a pedagogical tool so I'm using it more for courses and classes and for sort of documenting my own stuff and it's not clear how much you know how much more it will be useful to compare to something from spark where they have a team of engineers working on it what I like about morrow is it's a little bit more focused towards machine learning and NLP and stuff like that and that I haven't seen so much unless you're using ml lib the other one wouldn't be that useful moro can be used without any of the existing stuff as well and it's easier to extend and things like that but it would be good as a community community to sort of start focusing our efforts towards maybe one or two but yeah something good question right so the question is our wolf is still fairly I'm going to be paraphrasing a bit of it but I wolf is still in its fairly early stages where not everybody will find it useful and so the question is what is the development plan and when do you anticipate people to start using it more regularly or more loudly so that's a good question and you know that's something we struggle with a lot of the questions that we are trying to address our open questions in the research community it's not clear how you go from this expression tree and always have something that's efficient even if mathematically there is a reduction there but what we do so what we've tried to do is use wolf for our own research and so already we have a group of researchers doing research in machine learning and NLP trying to use wolf and trying to support a broader range of models and I think it depends on the kind of user you are if you're looking for something that you know you have a specific model in mind you don't want to read up a little bit more about the math or anything like that then wolf right now may not be perfect for you but if you are using tools and you're hitting a block that you know you have some ideas about how to address then wolf becomes a pretty useful to even right now and yeah so the question is how much our expression tree confirmations has in common with compiler optimizations and the stuff that spark people doing I'm not too familiar with what those part people have been doing in terms of optimization but as far as compiler stuff is stuff goes we kind of think of it as orthogonal solutions we still want to use all the compiler optimization so everything we do ends up being Scala code that goes to the compiler so those still exist but what we are doing is a lot more optimization at the level of mathematical approximation and optimization which is a like mathematical optimization not compiler optimization which ends up being very different in flavor there are definitely interesting theoretical or formal definitions of the semantics of transformations that we would like to use in wolves that comes from the compiler community and there are some interesting approximate programming work that could be part of what we are trying to do and there are definitely connections we haven't been looking a lot into it any other questions hi thanks