data.bythebay.io: Scott Clark, Optimizing Machine Learning Models
Recording: data.bythebay.io: Scott Clark, Optimizing Machine Learning Models
Excellent. Everybody, thanks so much. Uh, thanks for bearing with the last minute schedule change. Uh, I know you might be expecting to hear a talk about BYU, but uh, I'm going to be telling you about how you can help tune some of the algorithms that are going to be presented in that talk, which is now scheduled at 4M. Um, as was said, uh, my name is Scott Clark, co-founder and CEO of SIGOP. SIGOP is a little startup in San Francisco, a few blocks from here, that helps, uh, people do Beijian optimization as a service. Um, over the course of this talk, I'm going to tell you about why you should care about Beijian optimization, how it can help you tune your machine learning models, how it can help you get better results faster, um, and dive into a bunch of examples, uh, as well as how it actually works in the underlying architecture. Um, if you have questions throughout, please just jump up and ask them
I can just repeat them to the mic so everybody can hear. Um, but I want to make sure I don't lose uh, people along the way. So I'm going to start off the talk telling you two things you already know. One is machine learning is extremely powerful. You can take all of this big data and you can combine it in interesting ways to come up with really interesting insights and search and recommendations and everything like that. But tuning these systems is often extremely non-intuitive. And by this I mean the stochcastic gradient descent parameters of a neural network like in the previous talk or the learning rates or the topologies of a neural net, the kernels of a support vector machine. All these hyperparameters, these parameters of the model itself that are very orthogonal to the domain expertise of the person coming up with features and building these actual models
So Beijian optimization is designed to help with this problem and it's widely considered to be kind of an unsolved problem or a very difficult problem in machine learning. Recently uh Xavier who's the VP of engineering at Kora and used to be the director of research at uh Netflix gave a a question and answer session in Kora and he classified this as the most unsolved problem in machine learning. understanding how these deep neural nets work and then coming up with a more or less automatic way to tune them. So, what I'm going to show you today is a more or less automatic way to tune them. And this is a real problem. Um, this was released by TensorFlow a few weeks ago and was on the front page of Hacker News for a while. Um, but this is kind of a graphical interface for tuning a simple TensorFlow model. So in this case, you can pull up a couple kind of two-dimensional data sets, build up a model, and you get to play with all these knobs and levers, learning rates, activation functions, uh noise, batch sizes, etc
And these are all the knobs and levers that affect how well a specific machine learning, one of these neural nets will perform. And once again, this is very non-intuitive. This is a perfect example of what a tool like Beijian optimization can help you solve. Here we have about 22 different tunable parameters and it's very hard to decide which one will do well. Um, and each time you want to maybe train one of these models, it can take minutes, hours, days, or even weeks. Um, so you need to be able to get to a good solution as quickly as possible. So there's a couple common approaches that people tend to use when they attack this problem both in academia and in industry. So some a very common approach is what's called grid search
where you take all of your different parameters and you lay out a grid and you say, "Okay, I'm going to sample 10 points along learning rate and 10 points along my momentum decay and things like that." And then I I'll just train a different neural net at every single one of these vertices of this grid. Um this is this grows exponentially with the number of parameters you're tuning. Um sometimes this is like comboed with a random search approach. Uh James Bergster wrote a paper uh somewhat recently saying that random search can actually outperform grid search for most of these deep learning problems, but it's usually one of these kind of naive approaches where you just throw a bunch of brute force at the problem. Maybe you were fine. Um but it's very computationally expensive, very expert intensive. And once again, if it takes you an hour to train your neural net, you don't want to use one of these things because it's going to take you months or years to actually get to that best possible solution. And when accuracy matters, when that recommendation engine getting a little bit better gives you a better uh customer experience, when a fraud detection algorithm saves you millions of dollars, you really want to squeeze the most out of these these methods
So there's a somewhat uh lesserk known field of uh research that attacks this problem head-on. So within the operations research community, it's called optimal learning. A lot of times this is called Beijian optimization, sequential model based optimization. There's lots of different terms, but at the core, you can define it a couple of different ways. Here we have my academic family tree trying to define it. Um, you can think of it as I have a time-consuming, inexpensive process like training a neural net and I want to sample it as efficiently as possible. I want to get to the best result as soon as I can. Um, and try to do that as efficiently as possible
And I spent some time in the advertising community applying these methods. So, I like to think of it as just make as much money as quickly as you can. At the high level, um, it's a relatively simply defined problem. Um I'll I'll talk about it here at the super high level and I'll dive into the guts of the math a little bit later after a few quick examples. Um but first we want to define some objective some overall evaluation criterion that we care about. This can be accuracy likelihood AU some combination of different metrics but this is the number that if it goes up you've made a better model and then we have the model itself. This model has these tunable parameters associated with it. the learning rates and stocastic gradient descent parameters of a neural network, the kernels and coefficients of the support vector machine, whatever it may be
This is what we can tune and it's very expensive to try different configurations and we want that objective to go up. So how do we get there as fast as possible? So to illustrate this, I'm going to take a super simple example here. I'm going to take this two-dimensional problem. It's nonconvex. It's deterministic, which is probably not always the case with these machine learning algorithms, but we want to try to optimize this and we want to see what some of these other approaches kind of look like. So, first we can take grid search. And grid search does exactly what it sounds like it's doing. It's just kind of blindly sampling along those indices uh or those vertices of the grid
One thing worth noting here is that no information is conveyed. Like if I sample at one point and realize that it's a poor performing point, I just blindly sample the next one anyway. I'm not really harnessing any of that very expensive information I've already gained. Random search is similar as well where I I drop a random point maybe uniformly on that grid and then I just drop another one and another one. At the end of the day, you look at all your results and you say, "Okay, what was the best I saw?" But you're never really leveraging this in kind of any intelligent way along the along your path. Now, SIGOP and Beijian optimization methods in general are the complete opposite. So, they're trading off exploration and exploitation. We want to take the historical information that we have and use that as intelligently as possible to make the best next step
So, given what we've observed, what's the smartest thing I can do next? or if you can run your system in parallel, what are the smartest set of things that I can do next with the goal of having you climb this mountain as quickly as possible. So in this case, this could be yeah learning rates and momentum decay and this is accuracy. You want to get to that peak that's money for your fraud algorithm. That's accuracy in your uh recommendation system. One thing about grid search though is that exponential penalty starts to really really hurt you. So in this two-dimensional problem, maybe you're only sampling 10 by 10. That's 100. In a four-dimensional, now all of a sudden that's 10,000, hundreds of thousands
Once again, if this can take even a minute to train one of your algorithms, grid search can become impractical very very quickly. And what we find in practice is that these Bayian methods actually scale linearly with the data. taking a good idea of what to do next based off of these historical points can really short circuit how long it takes to get to the best configurations of your models. So how does a system like this actually fit within a machine learning platform? So what we usually see is that there's already a platform in place. You have data. You have data scientists doing what data scientists are the best in the world at, which is taking that ugly data, cleaning it up, coming up with interesting features. Really understanding the business values and metrics that you care about and what to feed into one of these models. And then you have a machine learning model itself
This could be XG Boost. This could be TensorFlow. This could be just a support vector machine. Whatever it is, but there's some output that you care about. So the way that sigopt or another beijan optimization algorithm would bolt onto this would be just taking in that objective metric that you care about and suggesting optimal configurations to try next. So the goal here is to get to better models and as few of these iterations as possible. So instead of doing that exhaustive grid you maybe only over 10,000 points you hit something like sigop a 100 times. And one thing worth noting is it doesn't bleed into the stack at all
It's this kind of clean interface on top of what you're already doing and it just treats your underlying system like a blackbox to optimize. The underlying system treats this as a blackbox optimizer. So no matter what you've built, this can make it a little bit better. And as you continue to make your underlying systems better, this can continue to make them better yet. So they go hand in hand. So, I'll show you a couple quick examples before I dive into kind of the the nitty-gritty of the math um of where we've applied techniques like this before and some of the results that you can kind of see and some comparisons to standard methods. So, a perfect example of this is kind of a standard classification problem. Let's say we want to take all of the loan applications from Lending Club and we want to classify whether or not somebody's going to default on their loan or not
So we can take in things like income, credit score, loan amount. This is somewhere something where a data scientist can provide a ton of value. Kind of understanding that maybe income isn't just a linear feature like it can tail off after you make a certain amount of money and things like that. Maybe you want to combine credit score with age in a in a in a clever way. whatever it is. But once you dump it over to that machine learning algorithm, in this case we used XGB boost, which is kind of one of the standard gradient boosted uh packages out there um very popular in Kaggle. You want to be able to tune all of the different individual parameters, epochs, learning rate, number of trees, number of examples needed to split a node, all these kind of obfiscated uh parameters of that model. So you bolt sigop on top
You say I want to increase cross validated prediction accuracy. Get me to the best parameters as quickly as you can. And we can look at the trade-offs between these various approaches that people tend to take on this kind of cost versus accuracy plane. So cost down here is the number of iterations you had to try to get to a specific level of accuracy. It's worth noting that the cost graph here is on a log plot. Um, you can think of this as your AWS bill. Accuracy, on the other hand, is your business metric. So, this is how much money you lose by giving out bad loans
So, you want it to be as high as possible so you can classify whether or not you should actually give that loan to somebody, but you also don't want to spend millions of dollars on AWS to get there. So once again in this nine-dimensional problem with these kind of first order hyperparameters that you can tune in xg boost you can take it can take tens of thousands of uh actual iterations even with a super coarse grid to get to what looks like mediocre accuracy. Um it's worth noting that this beats the baseline which is if you just take xg boost and you don't tune any hyperparameters um that falls down at about 6 a. So you definitely don't want to do you definitely want to do some tuning and grid search gets you okay random search is a pretty good trade-off for the money um which is what was shown in that paper by Bergstra that I referenced earlier um after about a thousand iterations you get decent results but then SIGOP in these ensemble of the state-of-the-art of these Beijian optimization algorithms is really able to beat both and the intuition here is by leveraging every piece of data you get at every step you can make much better decisions. So you get to higher results much faster. Typically for 5 to 10 dimensional problems, it's about a 100 times faster than these other methods. Um and better results at the end of the day. But it's not just about tuning uh a gradient boosted machine like this is
It can really bolt onto any system. So a lot of other applications we look at are more like classical time series analysis or regression problems. So let's say you have an algorithmic trading strategy. Instead of tuning hyperparameters like learning rates and stochastic rate and descent parameters, they're tuning things like slow and fast moving averages, thresholds for trades, weights for specific data sources, etc. They can take in a bunch of market data like closing prices and days of week and the data scientists and quants that build these models really understand those features and can build it into an interesting way. Um and then SIGO bolts on top of that. So in this case, you want the highest expected revenue in as few of these timeconuming and expensive back tests as possible. So in this case, uh this is part of a joint blog post we did with Quantopian, which is like Kaggle for hedge funds
They do a bunch of competitions. You come up with a trading strategy. If you win their contest, you get to manage a little bit of their money. They take an ensemble of all the smartest techniques and make a lot of money on that. Um, but what we can see is even with a simple trading strategy like we did in the blog post with them, seven tunable parameters, the grid was so coarse in order to finish on time. Uh, there was three trillion different options. They were able to whittle that down to 3,000. Uh, it still lost money
The expert was able to do kind of a localized hill climbing, but once again, humans aren't really designed to do seven-dimensional non-convex stochastic optimization in their head. um at best you get to a local optima which is what they found here and then by applying these state-of-the-art Beijing techniques you can get better results on the back test here beat it by 200% 10 times faster than both that human and uh grid search approach. So really it's the difference between blindly marching through the space and kind of using smart math to get you there faster. And I'll talk about that smart math next. But any questions uh so far about Yeah, we got one here. It doesn't necessarily generalize how do you guard against that kind of Yeah, great question. So the question was how do you make sure you don't overfit? Um and in general this is a very important problem. One of the best things about Beijian optimization is it's extremely efficient at optimizing any metric you give it
But that's also one of the worst things about Beijian optimization because if you give it a metric that will tend towards overfitting, it'll overfit better than anything else in the world. So how you usually counter that is by doing like taking your data set, breaking it up into train tests, have a hold out, doing cross validation and regularization over that actual tuning step. Um, and I'll go into this a little bit in some of these examples uh later on, but you can basically make sure that you when you're tuning, you're not overfitting to the test set. And then you eventually actually run your full uh validation at the end against some hold out. And then the goal is whatever configuration you had, if it works in the holdout, it should extend completely out of sample. Um, but a lot of this falls onto the expert because once again, um, in time series, this is a little bit more difficult. Um depending on what your objective function actually is, you might need to take into account the fact that your data is coming from different places and things like that. But the goal here is if you do proper cross validation, proper regularization, if you have a metric that generalizes, then a Bayian method will just get you to the best version of that metric very quickly
But that's an excellent question. Anybody else? Yeah. So is this optimization only for this type of model or can be extended to Yeah, so that's a great question. Uh, does this only work with these types of models or can be an extended? At the core, it's completely model agnostic. All it assumes is there's a time consuming and expensive process that has tunable parameters. And those tunable parameters could be continuous, they can be integers, they can be categories. Whatever it may be, there's some parameter space and an objective that you want to increase. And the goal is to efficiently search that parameter space to get you to the best objective
So it can be a machine learning model, can be an algorithmic training strategy, could even be a physical uh experiment. And what are some of the scale we have tested on? Great question. So how does scale play into this? Um so I'll show you the math here in a second. Um but it does uh get complicated rather quickly. Um what we found in production is up to a few dozen parameters um a few hundred individual suggestions gets you to a good optima and beats kind of the thousands of suggestions that a grid search would would get you to. But this is definitely designed for problems that don't have 100,000 tunable parameters or things like that. Uh usually though you can uh parameterize one of your problems a little bit easier than that. Like technically if you wanted to go into the the deep depths of a deep neural net uh you could tune learning rates individually per node etc
But most of the time people can phrase this problem in a six to 12dimensional space. Great questions. Any anyone else before I dive into the math? Yeah, we got one in the back. In the case of neural networks, how do you deal with a certain architecture will accuracy. Great question. So, it's the question if I uh heard it correctly is how do you incorporate time into your objective function um or training time uh in particular. Um this is one of the nice things about these Beijian approaches is that the underlying objective function doesn't matter. All it needs to be is something that you can calculate
So you could take accuracy and multiply it by a time threshold. You could have some sort of penalty for exceeding two hours. You could have a penalty for exceeding your memory requirements. Like it can be as ugly as you want it to be. And this method will kind of learn that boundary and be able to say, "Okay, this is the best method given this these kind of higher level objectives that you care about, which can be a combination of anything." Great question. Anyone else? Sweet. I'll dive into the actual math here. So, at the core, what this is is uh sequential modelbased optimization
Um the sequential is a little bit of a misnomer because you can actually get suggestions in parallel. Um but the idea is we want to build up some surrogate model given the history we've already observed. So if we've trained maybe a halfozen different configurations of a neural network, we want to be able to take that information in and say how do all of these parameters interact with each other over what length scales? How do they combine to uh affect this objective function we care about and how can we use that information to extrapolate to parameter configurations that we haven't tried yet. So we need a good way to model this in surrogate where we can actually um do a lot more computational techniques without having to do like gradient descent on the actual underlying system. So we use a gausian process to do this. I'll give you some intuition on that in a little bit. It's this very generic very powerful framework for kind of doing stochastic regression at the high level. Once we have this surrogate model now we want to make sure we have the best fitting one
We want to say okay given our regression problem does it match the data we've seen so far extremely well once we have a good fitting surrogate function now we can actually throw a bunch of other optimization techniques at it we can say what's the point of highest expected improvement so given the information I've observed so far what new configuration is going to beat the best thing I've seen so far by the most and so this is once again trading off exploration parts of the space knows nothing about there could be good configurations out there and exploitation which is saying I think I'm near a local optima I want to make sure I resolve that and get the best value I can out of it before going back out and exploring but there's different acquisition functions we can use here and the idea is given the surrogate model make the best decision for what to try next give that to the user they report it back and then repeat the process again now you have a little bit more information for your surrogate model do that acquisition function again rinse and repeat. So first up what's a gausian process and what's expected improvement. So here at the top we have a gausian process. There's two ways to think about this. One is that we want to model that underlying function that underlying process that's a function of our hyperparameters as a gausian at every single point. So at a certain learning rate we're expecting a certain return. That's the mean of the uh gausian but there's some variance associated with it some uncertainty. So every point in the space is modeled as a gausian
Another way to think of this is from the functional perspective and that's saying at the beginning when we have no information the underlying process could be represented by any function in the universe. So we have a bag and in that bag is every function in the universe. As we start to sample points we can throw away functions that don't make sense anymore. So if we sample a point with no noise, any function that doesn't pass through that point, throw it out of our bag. And effectively what we're doing is we're building up a distribution of functions. And that's what the gausian process is. The most likely function is that mean of the gausian process, this dashed line here at the top. But then there's uncertainty in areas where we haven't sampled
And so there's other functions nearby that we think could also be possible. So once we have this gausian process as we sample more and more points we get more and more information about what the underlying structure could potentially look like in one dimension. This is relatively straightforward but once again the nice thing about these methods is they extend to these higher dimensional problems. So after a few points we've kind of resolved the space a little bit. We've traded off a little bit of exploration and exploitation and we can say okay we think this is a pretty good optima. One thing worth noting and I'll go back to the the bottom here um is that expected improvement acquisition function is the thing that we care about. This is what's telling us what's the next best thing to try. So if we have this gausian process here where we've only sampled this single point in the middle maybe this is learning rate and this is some quality metric with time assoc combined with it or something like that
We say okay we saw a pretty decent point here in the middle. How much do I think I can beat that point if I sample somewhere else? And that's this graph on the bottom, the expected improvement. And so in one dimension, it's really easy just to analytically plot this, but we basically want to pick optima in that expected improvement space to try. So we want points that have a relatively high mean, a little bit of uncertainty that we might be able to exploit. And we can sample there. In this case, it didn't do so well. So now the expected improvement for sampling near there has dropped way down. Now the algorithm thinks there's no way that if I sample near there again I'm going to beat whatever is the best thing so far
So we sample over towards the left where there's still a lot of uncertainty. We do that see a mediocre result. Um this is where the algorithm in the the inside is kind of making that trade-off between exploration and exploitation. at the beginning, it really likes this point over here that has a lot of uncertainty on the far left. Um, but you can see there's a point in the middle too that is a more exploitative point. And so as soon as we sample and reduce that uncertainty on the outside, now we want to go back in and resolve that local optima. And then we want to go back out and explore again. And it's kind of always making this intelligent trade-off
And you can see this little hump here on the the left is also slowly growing. So eventually we'll exploit this reasonable space over here. And the idea is once again especially as this extends to higher dimensions which makes this really difficult to plot but uh it gets you to that best result as fast as possible. So now I have an extended uh set of examples dealing with deep learning um especially uh one on efficiently building connets. But any questions about Gausian processes expected improvement how Beijian optimization works at super high level. Yeah. Sequential. Great question
Uh does it need to be sequential? Uh, one nice thing about grid search is that you could run technically every point at once because it ignores all previous information. Um, so there is a sequential component of this, but it doesn't need to be purely sequential. So you can get multiple suggestions at once. The intuition behind that is if I knew I could sample three points. If I had three free resources at this point and I have this information, then I can condition on the fact that I have multiple resources. And so this expected improvement graph all of a sudden is in three dimensions where I get to pick I want to find the highest sum of expected improvements. It's not quite the sum. The math works a little differently
But you basically say given all these free resources, what can I do? And this can also work asynchronously where you say conditioning on the fact that I know I have a resource working on this configuration. What's the best next thing I can do with this newly free resource? And you can kind of make this trade-off back and forth. That being said, you couldn't do maybe all a thousand points at once because then that would have no information to learn on. So there is a trade-off to be made. In practice, we usually say don't do more parallelism than the number of parameters that you're tuning. Um, but that gives you a decent trade-off. Any other questions? Yeah. Do all start with the same prior? Great question
So um initially they do all start with the same prior um but in that exploratory phase as soon as we start to get information about that underlying space that starts to change. So underlying all of this is kind of a coariance kernel which is saying how much does one point influence another across every individual dimension. There's a bunch of different coariance functions you can choose. There's warping you can do your data. There's a basically a bunch of stuff to make sure that you're doing the smartest thing at every step. But at the beginning, it's mostly exploratory anyway. So if you're sampling some low discrepancy sequence, you can usually get a good amount of information once you start actually exploiting. Great question
Any other questions? Sweet. So let's dive into kind of a little bit of an extended example. Um the talk I'm replacing was about neural nets. So let's talk about neural nets. Um so let's take a common and popular uh tool like TensorFlow and use it on kind of a classic machine learning problem. Um in this case we have a bunch of house numbers from Google Street View and we want to be able to classify what's that house number. Um this is a really difficult problem. Uh I think it's quite a bit more difficult than like the standard minced uh zip code data set because you can see in some of these examples uh different colors, different fonts
Uh sometimes there's multiple numbers in the same uh the same actual little figure. Uh some are blurriier than others, different angles, etc. Um if we want to attack this problem, uh there's we we took an architecture from like a a classic paper on this. Um a lot of times these architectures themselves have lots of tunable parameters associated with them. uh depths and widths, number of convolutions, size of dropout. Um this the actual uh neural network itself has a stocastic gradient descent parameter in it. The previous talk was talking about different ways to do this. Um RMS prop and adigrad are very popular, but they have these tunable parameters
Uh weight decays, um step sizes, etc. And it's usually not very intuitive how you tune that architecture, let alone tune the stocastic radian syn let alone tune them together. Um because a lot of times it's hard to intuit it. Here's three different parameterizations of RMS prop uh or nine different parameterizations of it and they all kind of do the same thing but at different speeds and um it's difficult to to intuitively say what's the best thing to try. So if we take a problem like this and we throw it at SIGOP or these Beijian methods, what we find is after a small number of evaluations, we're able to increase accuracy quite a bit. So the baseline uh for this problem would be 0.1. You just roll a d10. Um and whatever number comes up, that's what you classify your number as
Uh not tuning TensorFlow at all gets you just a little bit better than that. So if you just kind of take their default example and pipe this data set through it, it does a little bit better than random, but not much better. Um, in fact, just throwing it at a random forest does better than that. Um, which is this purple line here. But if we want to take one of these algorithms and actually fine-tune it because it matters to our business, it's a tra it's a fraud detection algorithm or it's a recommendation system, you need to be able to do a lot better. So here we compared sigop to random search. You can see that SIGOP gets better results faster. Um, and this extends to that hold out data set
So that was a great question earlier. When you're actually doing this tuning phase, you need to make sure that you're doing cross validation, etc. Um, and then also keep a hold out to make sure that you're not just overfitting to some training set. So we took the best versions of all of these algorithms and uh threw it at the the holdout trading set. And you can see that once again, TensorFlow gets you twice as good as random without tuning it, but you can do hundreds of percent better by tuning it. And this really matters at the end of the day how quickly you can get to these uh individual solutions. So for this specific case, we were just taking one of these small data sets um from one of the Stanford libraries. Um and you can see that there's cost savings associated with getting to different thresholds of accuracy even on a small data set
But once you start running things in production and in this case with like 50 GPUs and I know that a lot of uh a lot of individual companies have these big distributed GPU systems. I know Netflix uh has a lot of talks about how you can tune their architecture across this distributed system. you can actually get to these results quite a bit faster and cheaper because at the end of the day, iterations are dollars on your AWS bill. Any questions about that data set? I have two more quick neural net examples in my five minutes left. Excellent. So two more quick examples of taking kind of the state-of-the-art of a specific paper and being able to say what if we go back and look at the architecture look at all these kind of underlying assumptions and see if we can tune it a little bit better. So for this we're going to take a slightly different data set CR10 classifying trains, planes and automobiles. Um 10 classes
Once again 0.1 is random accuracy. Uh this is a a standard problem that a lot of people compare against. There's like a leaderboard um where you can see who has the best results on this. Um this paper was on that leaderboard very briefly and on an all convolutional neural network. Um in this case this was joint work with uh Nirvana which is a deep learning cloud platform um where we wanted to show by using SIGOP and their Neon um uh neural network uh architecture that you're able to kind of push the state-of-the-art a little bit further. In practice, once again, in academia and industry, the standard is brute force, grid search, make your PhD student do it. Um, whatever it is, get to a decent result and then publish just the parameters, not how you got there. But what we're able to see is even in a very relatively simple system, um, there's lots of different parameters that are exposed
These are just kind of the highle hyperparameters exposed within neon. Um, just tuning like stochastic gradient descent parameters effectively. And what we can see is over a relatively small number of actual uh evaluations, SIGOP was able to beat the published result in the paper, reducing the error rate by 1.6%. And the the real takeaway here is you can get just as good as a bunch of experts fine-tuning it without having to waste expert time. So humans and data scientists in particular are incredibly good at coming up with features and creatively understanding their business propositions, but it is a waste of that expert time to make them two knobs in the dark. So you can get there or better without having to do that faster and cheaper. Another example is a deep residual net. Um, this one stayed on the leaderboard a little bit longer
Um, I won't go into the depth of the paper. Um, I'll release these slides right after this and you can look it up. But basically a little bit more complicated problem, but once again they got a decent result uh 93.39% but SIGO was able to reduce that relative error rate by 15% in this case being able to get to kind of a better version of those individual parameters getting better squeezing better results out of these kind of state-of-the-art experts. So with that any questions about sigop bijian optimization how tools like this can fit into a pipeline. Yeah. So what kind of GPUs were used for this uh the production slide what you showed earlier. Yeah. So uh that cost slide was uh used uh built using um just the AWS uh I think it was the 4GPU instances
Um uh some of the work was also done on Nirvana's cloud which uh uses Maxwells instead of Keplers. Um so a little bit faster and they have like custom kernels that make it go a little bit faster. But in general the idea is training on GPUs is super expensive. You want to you want to use as few of those as possible to get to the results you care about. Yeah. So how does this compare to other uh nondeterministic algorithm? Great question. Um so there wasn't enough time in this talk to go over like the deep comparisons that we do. Um on our website.comressearch, we do uh some deep comparisons there
We're presenting some work at ICML next month where we compare not only against yeah grid search and random search but things like particle swarm optimization as well as a lot of the open- source beijian optimization techniques like spearmint, hyper opt, smack. Uh I wrote mo uh while I was at Yelp which is another one of these popular libraries and we find that we compare very favorably in terms of performance. Um but we also get there orders of magnitude faster just in terms of computation time because we have a a big distributed system behind this API um doing a lot of smart premputation etc. And it's also a lot easier. Um when I wrote Mo I tried to make it as easy as possible but it still requires administration and a lot of time and the goal of SIGOP is wrap all the most powerful research in the world behind a twoline API so you can just kind of plug and play anywhere. Yeah. How big uh the data sets you usually dealing with? Yeah. So the underlying data set itself uh it doesn't really matter because that's once again on the customer side
So you're feeding a big data set into a machine learning model and all that affects is how long it takes to to observe uh a result given a set of hyperparameter configurations. At the core, SIGOP is designed and these Beijian optimization algorithms are designed to tackle time consuming and expensive problems. But how long it takes to actually observe uh a configuration's uh performance is completely independent to the underlying set itself. That being said, uh the number of parameters does uh affect how long it takes sigop to or one of these Beijian methods to perform. Um in practice, we usually go up to a few dozen parameters um and get a few hundred suggestions is able to get to people to a good uh result. What we find is usually between 10 and 20 times the dimensionality of the parameter space you're tuning, you've gotten to a good global optima. Um, wise. Say it one more time
Yeah. Yeah. So, I'll go all the way back here for a second. Yeah. So the underlying data here for so for these blog posts sometimes it's a few gigabytes for some of our customer customers it can be terabytes um the pabytes starts to get into a little bit of an issue because if it takes you weeks to evaluate a single thing then um being able to optimize within a few months or whatever matters for you uh sigup will do better than these standard approaches but it can still take a long time but it's worth noting that all SIGOP sees is that objective metric and the parameters you tried and new parameters to come in. So this can be as big as you want. This can be as complicated as you want. This can be the combination of many different metrics, but because it's just this kind of blackbox optimization, it doesn't really affect us at all how long the underlying data set is
Yeah. the questions. Great question. So the pricing model for SIGO opt uh it's a subscriptionbased service based off of the number of models that you tune per month. Um so we find that we're usually able to save people quite a bit of money off their AWS bill regardless of the fact that also saves data science time and uh like it gets you better results at the end of the day. So our kind of team-based plan gets you 50 models for $3,000 a month to cover our compute costs. Um, and we find that teams usually rec recoup that very rapidly in the AWS savings. Okay, let's give a big hand to Scott Clark of SIGO
Thank you.