ML Jenia 5 3 16

so hopefully I can provide you with some more hands-on experience on this and also a little bit more low level not just the the high level ideas but also a little tiny details that oh my gosh what's going on here that actually happened when you start trying to do things with finance data so I am a grad student at Cal Tech and my primary course of studies theoretical physics so you may notice that I'm not very serious about finance during this talk or machine learning for that matter sorry so how to create an algorithm that would actually perform in a competition and have chances to win and in short it may be summarized in those three steps so you first of all actually download the tool box whichever toolbox you're using in this particular case we are talking about one text and get familiar with it and it has its limitations so you need to know about them second step is that for a particular competition you and for a particular data set you need to set up your own optimization and cross validation routines for your algorithms and indeed there are more than one ways of doing cross validation sits a cross-validation given the data that you have access to and it's up to you to decide what is the most appropriate way to do cross validation so of course your results will vary strong will depend on what is that framework that you set up and only after that you go to step 3 over there and you start coming up with those ideas maybe we should look at like momentum maybe we should look version maybe you should think about this as a random process maybe we should look for patterns so and all those ideas you don't just write them down you also back test them using this step 2 so not just writing system ones but using your own optimization of parameters and cross validation protocols and then as long as you believe that your protocols are reasonable for this particular problem then you will be able to choose and tell whether your strategies are any good so in those first two steps actually involve some kind of tricks of trade that are very counterintuitive so hopefully this presentation will allow you to breeze through them without actually like spendin time stressing out about something that is not worth it all right so before we even begin if you go to a economics department then University and talk to economists about and finance majors about what do they think about quantitative training trading they would tell you that stock prices fundamentally cannot be predicted actually their Nobel prize-winning works that claim that well there are probably uh Nobel prize-winning works the claims the opposite but the idea is that as soon as there is a signal in the data something that allows you to make profits then somewhere in the world there should have been a person that notice that would notice the signal and he will make a lot of money using the signal and he will keep making money until the signal disappears so and supposedly this a helpless response happens really fast and any signal that you saw in like 10 years old data is gone by now so this is their point of view but we don't just like think about it in words whether it's logical or not we actually want to take a machine learning approach to answering whether like it's pretty is it predictable or is it not so we want to let the number stylist and yeah I guess the question that we're asking like roughly very simplified manner is that your given this this curve that goes up and down and you want to tell what's next so this is the problem for now so now I would like to have a metaphor so something that you can remember once you walk out of this room what is that that's different between machine learning and finance so the classic machine learning task would be this image recognition so you're given like a little drawing of somebody written by Han hundred a number and you need to tell what number is this from 0 to 29 so the question is what's in the pictures and we'll show you the picture however with price predictions something else happens they still ask you what's in the picture but they hold the picture face down they just look at you and ask you what's in the picture so this is a metaphor of course you shouldn't take it seriously but there are two important interpretations of this metaphor so first of all we are trying to predict what's going to happen tomorrow so we should just wait and see what's going to happen tomorrow this is the most natural way to find out Ivan and predicting this kind of not natural it's like they hold it face down and we're trying to say and more specifically the processes that are market-making tomorrow the process that will actually tell which direction the price goes they have not even started today and they are very highly random so what can we do today so this is the simple idea and the idea is that I guess finance is much more noisy for this particular purpose of making money than the typical machine learning data sets but you can say that okay machine learning data sets also have noise so I would like to point out another more subtle meaning of this metaphor so for the image recognition data set you immediately assume that there is no correlation whatsoever between different different like rose in your table like bit with different pictures that they show you so whichever pictures the referee show showed you before doesn't have any effect on what picture he will show you next in finance however those pictures that the person holds in the competition like showing you before actually is the only thing that you can use to predict so you have no other choice than to assume that maybe maybe they are actually correlated with the picture that he's going to show you next and this was not the case for those digits okay so now we're done with the metaphor we can get two more specific and more concrete mathematical things that we can say so machine learning is a general field and Finance is very special part of it that first of all has the noise that's much bigger than your usual data set so it would be equivalent in the digit that is said to ask it to ask in what digit is that in the picture we and you cannot really tell because it's all smeared and like the person wasn't really sure what he's gonna write and there are also other differences so prediction by itself is kind of not very useful because this is just a number but this number is where the price is going to be but the number that the broker actually wants to know like your investor wants to know is what kind of orders do you want to make knowing that your prediction right so you need to convert your predictions in two orders of a specific instruments on a stock market and the third thing that is kind of subtle but you also want to keep in mind that if prediction digits kind of is something that you don't really care about intrinsically right so if your digit predictor was working yesterday but suddenly stopped working today you won't go and throw yourself out of a skyscraper but with finance it's kind of the other way around yeah yeah well but it's still kind of very emotionally emotional rollercoaster where you submit the algorithm and wait for results of the competition or when you back test it for that matter all right so we would like to proceed yeah when I talked about noise I just like to stress out that we actually understand what it is more or less it's the contributions to this data that are completely inaccessible to us when we look at other columns in our like data set and they cannot really be modeled with information available to us and in the case of stock market there's plenty of those contributions so there are other players they're doing something they're thinking something they are like all those unpredictable events that just affect go and affect the stock market and there's also inside their information that somebody gets and starts using but not as so all of that is kind of not in our data set we and we already know that it's not so what's left it's pretty much clear by the time like we repeated many times it's a hard problem so let's get technical ready good let's go so we are going to show this slide as a useful slide I always want at least one slide in my talk to be useful and at least one slide in my talk to be mathematical proof so it's actually an extension on just usage of the toolbox as from the previous slide presentation so this load data function you should put one into it after s so that it loads the new data this is something that I only learned today and actually I was always loading the old data it was really annoying but okay other than that what this shows you is that after you download the tool box you get those standard functions get settings and load data and with those functions you can easily look at the whatever what is that that we are trying to predict the movement of the price over one day so this data close is the clothes price and the second index of it corresponds to which futures which rate so now we trade futures number two so they'd sets all futures and the first index is the index of a day so we just form a avec a array of change well this particular case a vector of changes of price of these particular futures between every like consecutive two days so then we get rid of nuns and we get rid of the high jumps so solid that to plot a nice histogram and then we stare at this histogram because what we see is that well essentially it just goes up and down and it looks like more or less like a Gaussian ish distribution centered at zero so and after staring at it like we come to a really interesting conclusion that nothing's going to change right so just by kind of staring at this data we see that it's kind of equally likely to go up and down and this prediction that nothing's going to change actually gives the best mean squared deviation of predicted price from the actual price compared to the other algorithms that you may write if they're simple enough so is that it if you just measure like the mean squared deviation from like the of the price movement and actual end the one that you predicted like predicting 0 for the price movement actually gives you the best so what do we do we we cannot be satisfied with this because we want to make money and if you project zero every day like we don't particularly feel like buying and selling something that we don't expect to change and it means that our objective function that we've chosen this mean squared deviation is not really appropriate for the task if we actually care about making profits so we need to choose an objective function somehow favors the predictions that are not 0 even if they will increase the mean squared deviation so this is actually a question so I mean it's not immediately obvious that this is possible we want to make profits with a prediction that is worse at predicting stock market than just this nothing's going to happen prediction so this is like a mathematical problem that I want to demonstrate you the proof as I promised Earl so in my talk there'll be one slide was like useful slide and one slide of proof so this is a problem i want to show you like a slide of or two of proof so the problem is as follows so you have you want to come up with some kind of prediction depending on the past that you make your decision based upon and you base your decision upon it and then you just calculate profits as your decision which is your order times the change of price and you sandwich overall days and this profits you want to be positive however your prediction we are interested in this kind of counterintuitive case where your mean squared deviation of this prediction is actually bigger than the mean squared deviation of a trivial prediction so this is like a full formulation of that mathematical problem so cool in the audience I'd like to make a poll who thinks that this is possible Wow who thinks that this is not possible all right yeah so this is actually yeah I mean like if we take a majority vote vote will get it right this isn't it possible and it is in fact since we have this decision vector right we actually have plenty of ways to make it happen because there is a lot of freedom and how we make our decision and to prove that is possible we just need to give one example of how this can happen and this is somewhat i contrived example but something like that cannon would help and would happen in in your practice so suppose that wherever prediction you have your decision to buy and sell is just sign of that prediction so you always buy and sell just one item and the true change of price that happens every day is a random variable that is minus two minus one plus one plus two is probability one for so one of the four and your prediction for the corresponding event will be as follows kind of deterministically so your algorithm for some reason outputs for this four numbers the other four numbers minus one plus two minus two plus one so it gets it right for big jumps but it gets it kind of terribly wrong for small jumps so for this particular decision and prediction your mean squared deviation is 20 which is like two times bigger than the mean squared deviation of the drivel prediction however we are making profits so we're making point five dollars or whatever every day on on in the expectation value and the average so all right hopefully are convinced that this is not such a strange thing and indeed mean squared deviation is kind of not not so useful by itself however we may notice another property that kind of also says that profits by itself are not so useful because what we have taken is the the decision is just plus minus 1 so we buy only one out or sell one unit but if it just said this decision to 10 then we our expectation value just gross proportionally so we get five dollars every single day on average so that's why I mean there is some problem if you just say set decision equals prediction and we can tweak a prediction in such a way that like it will lead to infinite profits just kind of mathematically we know that something kind of grows and we just put everything into it like every time so like the mathematically this problem of optimizing the profits is kind of ill-defined that's what I'm trying to say okay so we can bound the decision which is the normalization kind of thing which is what point X actually intrinsically doing but another way of thinking about it is we instead of profits as our objective function we may want to normalize our objective function so that there are no infinities of that sort so we construct this objective function where we give something that's our profits in the numerator but in the numerator we have essentially the standard deviation of those profits or like here I did not subtract them in but the meanest epically smaller than the standard deviation so shouldn't be much difference so this is an objective function that would actually work because there's like a square root kind of if we increase the profits by 10 like then the the deviations will also increase by 10 and by definition of what is called Sharpe ratio and what we failed to see the slide with the definition of privilege in the previous talk but actually this is the definition of Sharpe ratio if you throw away all the finance jargon so if you don't throw away all the find the sergeant's kind of impossible to understand what Sharpe ratio is but if you do throw it away then it's kind of actually really simple it's the mean over the standard deviation or given like a specific realization of data it's just some overall days or a square root over some or old days of some kind of square all right so now it's it kind of works it's a valid objective function conveniently it's also exactly the same objective function that context uses in its competition so in optimizing this objective function we may come up with something that has positive profits even if it kind of not even if it doesn't have pretty prices better than a trivial prediction all right but ideally we would want to come up with a prediction that also it's better than the trivial prediction of course so now there is also another thing that is interesting about point x and this is kind of a technical slide I will come back to it let me just explain what I'm doing here so pretty much everybody who was inspired by my result and went into this toolbox and started working with it eventually started reading this open-source ranches function and then there was they were completely confused because there's this variable equity and you look again in investopedia or whatever what equity is and then you look at the code and it just doesn't make sense so I think that it's kind of instructive to explain what that equity actually is a variable that kind of doesn't make sense the variable it does make sense is called fund equity which is the I kind of aggregate so equity is defined for all the futures individually but and X is defined just for your portfolio together so in simple layman terms or in the language we actually speak a fund equity is just our total money and that day so all right now we can actually look at what has this vector of orders that we need to feed into the point X algorithm and in words it would be that it's a fraction of our net worth or our total money that we decide to use to buy or sell at this kind of future number k 11 on a specific day that we feed it in into the system so in as a formula it would be essentially the number of shares of the future that is held at the end of the day after we sent our orders times the price of that future and divided by the total money that we have at the moment so this is the target for whatever the the back end kind of trading that is going on with the broker and something that we don't really need to care about but we only need to know that this is the meaning of PK and our returns are to the first up mation just change in the price in the same formula and this returns will give you the total if you sum them up it will give you the relative change in the total money but those individual red red k are actually those equity variables so it's the change in the value of given futures / your total money so that's why I mean it's not something that like it's easy to say in words what it is because it's only defined really when you have this kind of portfolio the specific constructing the specific way so and this red k is called equity so hopefully you guys if you all ever have trouble reading the strands yes open source code then like possibly it's because this equity is not what you think okay and the notation that we used so far i called p the decision so i will keep using decision hopefully you guys are okay with that now we don't do any kind of tool box specific things I'd like to talk to you probably all have questioned by now like how do we actually make this decision what's that the core of the algorithm so first of all I would like to say somewhat trivial thing but it's still worth saying it out loud that okay so this is a way that any time series can be broken into a table that is more familiar for like machine learning and data science purposes so essentially what you do is that you have like time that's long you take some chunk of it and then you duck duplicated for every day that you want to predict so you drop it down once then shift a little bit drop it down twice so you get this kind of letter and then you just kind of collapse it into a table where the last column is what you're trying to predict and these are the look back amount of days that you use as features for the thing that you're trying to predict so in the end you come up with a form that is very familiar for machine learning you get a table and you're trying to predict a column in that table now probably this is not the best way to come up with features so you want to somehow restructure what's in there all right so I would all right Wow I would use a decision kind of of that gray rectangle to denote the decision function the algorithm that we used to get our decision out of our pee out of out of features and there will be a lot of these great rectangles further down the line in particular the toolbox provides two algorithms already they're called mean reversion and trend following and I denote them as mrn here in fact each one of those algorithms if you look at it has two parameters the long period and the short period which are just number of days for a particular mathematical comparison so our algorithm can be said like it's it's something that returns a decision based on features given those parameters so in more general way it can be written as a decision based on features with some parameters in this particular case our parameters are which algorithm are we going to use and what's the the two periods so there is one more thing here though we actually have multiple futures about 50 of them by now we can use different algorithms for each one of them so it doesn't have to be the same algorithm on every single futures so in our general notation so we we want to introduce this index K so mean reversion and trend-following they kind of so they return re either plus or minus one for every futures so these are the algorithm that work for every future and they return plus minus ones so they essentially split our portfolio equally every day between the futures they actually decided to trade and don't don't have anything in the ones that are zero so it's not probably the best way to do it so we want to have a more general way to split our portfolio so we denoted in this function kind of that i symbolically say called combined so suppose that we have a bunch of decisions for every single futures and we have some kind of method of combining them so our total decision will be a combination of decisions of individual future say with some weights and the way to choose these weights is the method that we use so this is just the notation and now we can write that our decision function depends on parameters which is which are different for every future so this which algorithm and periods are different for every fusions and method is the general parameter so the method that we choose to come up with weights so we have a whole bunch of parameters now every futures can be traded with its own set of parameters after that like the knife thing that you would do is just see which if our parameter space for those two algorithms is not very big actually so we can just try everything and we can check what which one gives the best Sharpe ratio and I thought that the parameter space not to be we probably won't overfeed all right so i did that i found with the parameters and i submitted them and this was the q2 competition so about a year ago I lost this is how the out-of-sample the competition days look like and what happened next so you see it's essentially kind of stays horizontal and of course the way to to protect yourself against this thing happening is to do what's called like actual out-of-sample in this particular case I want to phrase it differently so since you are a machine learning crowd you're probably familiar with parameters and hyper parameters so essentially you can rewrite this algorithm instead of just parameters you can put two things there and I would like to stress out that this is not actually something that people talk about a lot but the distinction between parameters and hyper parameters is actually completely artificial so whichever you put into one or another side is kind of up to you and whichever way you want to set up your kind of machine learning but then you can have a more sophisticated way of coming up with an algorithm that you actually submit to the system first so you split the data into training set and test set which is tested as your kind of out-of-sample that said then you maximize your Sharpe ratio for certain values of hyper parameters fixed you find the parameters that maximize Sharpe ratio on the training set and then using those found parameters you actually go to the test set and find the maximum on the test set Oh over the hyper parameters so which hyper reverence should should you have used on the training set to come up with the parameters right so this desire like the two-step procedure which you need to set up after the end at the end of this procedure you come you end up with a set of hyper parameters that you can use to train your parameters and submit one single algorithm so this is essentially machine learning okay and there's actually very good reference in Wikipedia about that which is my only reference this page and I would recommend anyone to read it carefully this is an example of what happens when you set up all of that come up with an algorithm and actually tune two per m2 hyper parameters so this is the space of hyper parameters and it's represented as I see with a bunch of islands so everywhere like you see a sea of losing algorithms so sometimes it's really deep because algorithm loses a lot and you see a bunch of those yellow islands where the algorithms actually are not losing yet so those are the islands that you want to find and the more you explore it is see the more little islands you're going to find and then you probably want to think about which one's of them you like the most how do you see change out your it actually changes in a very interesting way but it highly depends on the way you set up your cross-validation routine for this for this optimization so the way split and training set and test set and in my particular case actually the algorithms work better on the last few years then than before which is somewhat counterintuitive yes so I would say that the islands stay where they are they just kind of grow with that which is nice and completely not what I would expect maybe when the algorithm gets deployed they will stop growing anyways this that that is what i wanted to tell today there are also things that i do not really have time to talk about much but i still want to comment on that so there is slippage in this particular a model and some kind of sleep which are commissioned in any trading so the answer to those economists from the beginning of the talk who think that prices are not predictable is that technically they are wrong it's actually easy to predict prices suggest that you only predict the movements that bring less profit if you use them then it costs in fees to actually do follow those movements so there are plenty of stuff you can't predict that is useless and finding something that you can predict and it is useful for making money is what actually hard so the another yeah another kind of two things that I would like to mention really briefly is about greed is really incredible thing when i wrote this algorithm and i mentioned it to people it's you just notice how like the eyes start to Sparkle and they start to talk to you and like my attention to you but actually like it's just because when they think it's about the stock market it's about money they somehow immediately make this transition if it's about maintenance about big money it's not always the case if it has money in the name doesn't mean it's big money right another thing personal financial decisions is essentially like okay should I put my own money into it and typically with your own money buying hold of S&P 500 is something that is something that probably makes more sense that's my personal opinion I don't have time to elaborate why and yeah yeah that's definitely true you need to use a lot of third parties to actually get your trades an algorithm out there with your small amount of money of an ordinary person and whether is it time to start a hedge fund once you get a single algorithm well starting a hedge fund itself costs about a hundred thousand dollars in legal fees so need to consider that okay thank wait a second maybe there are questions yeah yeah certainly I'm register take questions yeah if if you guys have questions to us Virginia or London my what Lecter buckle I is there a way to look at things differently over time as a measure is X cease to exist that compression yeah that's kind of hard the markets change constantly I can't believe recommend to come up with a mineral and single method is ideally not able to go five years ago but there's really certain specific effects yeah yeah I also told I can answer this i heard the question as whether you want to use the gold five years ago for your prediction and yeah so that that is probably true however what you can do indeed is you can come up with stable algorithms that work both five years ago and now and these are actually the only algorithms that you can cross validate an any meaningful sense so i guess that's my answer oh yeah please analyze complex idiot um all right well complexity is very much model dependent in general there is this the streakiest step right that I mentioned that you have parameters and then you want to split them in parameters and hyper parameters so typically your parameters is something that you have a formula for so you directly produce parameters from the data using some kind of like a formal or algorithm and the hyper parameters are just the kind of the little tuning knobs in that formula so you cannot really directly a reverse engineer them from the data so this is the typical rule that allows you to control that no complexity will ever get it to hyper parameters we will never make it to hyper parameter so that's about controlling complexity now about throwing away the unimportant features well the short answer is I don't know you just need to try different features and see which ones don't give good algorithms and then you probably don't want to use them yes yeah that's a question that usually people don't answer I i would recommend the following thing first of all their example algorithms second of all there are online a lot of people that share their code for different algorithms and this is probably a place to look in a sense that to look for ideas and also possibly that their algorithm will probably not be machine learning algorithms however you they can give you ideas for future engineering right and also they may give you just what's it called yeah they may give you like a kind of component strategies that then you can fit in into this cross validation scheme as I use this to example strategies them in following and trendy version so and then you just use this cross validation scheme you don't really need to use any kind of machine learning tools if you just use the cross validation skin and the strategies that you found so this is something that can potentially be a good idea however there when when you start wanting to use a tools from like machine learning kind of folklore like whatever the random forests quick and dirty so if you try that well first of all it will be like a more complicated algorithm so it will take a little bit more work to write it and then you may find out that it just doesn't work so I do not know how to recommend like in this huge world of algorithms but I am open to discussing I mean I certainly haven't tried everything but I I would like to I mean what what do know that works for small and noisy data sets probably nothing really works okay Markov models no I have not looked at Markov models in fact this is some a little bit of white spot in my knowledge about machine learning I mean I I can use their results for any kind of text tasks but I don't know what's what's inside them that well to actually write my own I should probably say that actually for these competitions that I want I had to write my own thing it wasn't too complicated but I didn't use any standard library or token well my algorithm actually is positively correlated so when there was a drop then there's also a drop in my algorithm but unlike I said before I founded my algorithm kind of recovers relatively reliably so you don't have to wait for half a year for it to recover you guys mentioned it being like a major sort of pearl the algorithm has to cross use something like let's say bitcoin you you're directly for some intermediate finish big scoops these were these transactions algorithm have a better chance better yeah the way I would say it is that many of the ideas that you would immediately throw away in this scenario would actually work there you guys have any plans at blockchain base currency as future to be applies to this watching or other future markets as long as there's a great history of theta been available yes but the problem with luck chambers on mortgage guy you sadistic do not warm enough around to give us a meaningful back tests on baby exact person so it will take a couple of years before we actually consider any more questions help you find a significant difference in performance when you're basically using play money actually started trading that's probably for you okay how close is the life trading results of the back tests that's the summary to question it just the back tests but like you can watch it yeah okay yeah yes how close are these results anyway okay we have to do a couple of things that change the life result or may be different from the simulated about life result of the operon for once we have to discretize our positions so you give us a double of your desired location in percent 12 to certain asset we have two disparate eyes that to fool contracts so that means depending on the amount of capital regenerate your strategy with this might be closer or not so close to your target application so this is the first thing but the good mathematical property about that is the larger the amount of capital at betrayed the third outwardly the closer it is to the simulated performance simply because the allocations the area between target locations and three locations that we can get gets lower and lower than higher two capsules so that's one of the reasons for it for the difference is the other reason is of course that the real slippage and permission marketing back these kinds of things might be different from our very simplistic model but to sum it up we're happy proud to life results match the life simulation very real and you can look that up on the website as well as of the orange curves off of the trading systems that are returning these are the best results they mesh the sig- life results pretty yes so send orissa mention something dropped suddenly we have seven bridges the authors okay question was what what if some chalk happens to be have circuit breakers that take care of the investor does not moves over tournaments yes of course we should have them in production we also we have a very strict risk management in production with stops and other things that will protect the investor from worst cases but this is you don't need to worry about that response this is something we worry about together with the investor it's also the investor because to select the level of risk that they want to trade your strategy with so let's assume your strategy practices in volatility or risk at five percent then investor can easily drape the same strategy with ten percent twenty percent usually institutional investors want to have a higher volatility other products so we think we can leverage your strategy a factor of four if the investor desires to do that and that the appropriate risks risks thompson place yeah yeah yeah alright well I mentioned random forests which are contained consists of this is they consist of decisions reason they are example methods by definition so and I have tried them i mean i can say whether i ended up using them or not but yeah sure well why not that's perfect rating like any given for what's the distribution of life the number of life is the algorithm least one is that way on projects they or do that change and sure yeah the question was how our location between the algorithm city actually trace changes over time and if it does no it only changes with the performance of the algorithm so we allocate our current allocations our results of our competitions so yeah one competition get an initial investment limited dollars and are entitled to to that investments basically as long as their algorithm on that data yeah so these allocations don't change but of course the amount of assets under management changes with the success or failure of the algorithm overtime so at the moment we have seven algorithms that we trade honey Gallagher like I said pretty happy with me yep overall goes to get together investors that kind of house how it's going to go with institutional investors so how these affect the algorithms or how they money around today I believe I'm a girl yeah okay okay one of this in areas as we open up so we are first part in place that's the division so we want to connect your up early to cattle from institutional investors not all the other words on our platform will have investments obviously but the top five to ten percent will most likely have investments our target is to work with institutional investors that's why we also work with futures and highly scalable strategies and thin data at least of every single one everything is matching our bedroom should at least be able to so genius the strategy for example I think the math recently can handle three me anyway but there's a cap to every strategy how much money it can Neville with a reasonable market impact and yeah we have to compute that on individual basis this is how r much money the strategy puts in liquid pour the liquid assets basically there's a captain we would never take more as it's a romantic for strategy than that camp but other than that institutional investors are free to select from the portfolio we provide them of course with active the place is run based on correlations on their own portfolio what they have going for test with their strategy and we made them a suggestion but it's up to them how often are the competition's of the next one when it's kinda usually quarterly we're keeping that place probably for this year so you can expect the next competition to be 28 to 30 this year we will call it the q 6 or 6 competition and we're going to announce it probably in one or two weeks from now but there will be a next competition is when we're pretty similar to the previous competitions oh my god so basically like for example he said your particular strategy poorly yes at the moment depression was how we select the strategies and how their correlation or how about the work together affects our decision to allocate money to them the answer to the question is at the moment we don't select species other than their performance in our competitions so the only section that we have in place is actually we take the Sharpe ratio of the backtest receive it is your strategy in three months of live data we take the best free spirit choose from four different ones allocate capital to me so it's a very very simple approach and that's not necessarily what you would do as a fund manager I also want to add that but we do it to incentivize people to something to training authorization to keep them results after three months and months force them to wait for half a year or a year to see if they're elderly people so that's not Portugal do as a punk manager still evening by doing so queer absolutely fun any more questions yes Americans for the good training algorithm arch the variance in prediction the very good algorithms perform on nice data equally well as the in the back desk so these are compared to the back distance low very low this is also how our scoring works but it's enough to have a knife a system with a knife shop ratio of more than one that's already great so you can access it fetishok ratio 5 and your life results has a separation of one that's still good at ease it could be okay for the winning strategy was typical range of Sharpe ratios profit gracious the typical range of Charisian I think Ginny I want the q3 with a shellfish shelf above to discover one of two and i wish in stitches like results had a chapter shift of 2 and 0 axis plus five which is pretty agree I a typical range should be between 1 and 24 minutes of the competition so in that three-month period usually operation from 12 mm and allocation performance wise it's hard to tell because performance is just another metric that doesn't per se matter that much to us only in and only how it goes together that's really important so performance alone since you can't leverage future strategies as a genius purchase a car says I think the volatility of only four percent performance there is no water and put it into leverage so you can pretty much adjust the investor was kind of whisk you focus on good stretch yes one last question okay I special for you think yes where are we going to both sides in a group and yeah we posted on our ties