data.bythebay.io: Martin Froehler, Quantitative Trading with Machine Learning
Recording: data.bythebay.io: Martin Froehler, Quantitative Trading with Machine Learning
thank you thanks for the intro and thank you everybody for hanging around at this late hour tonight today yeah I want to tell you something today about quantitative trading with machine learning yeah machine learning becomes increasingly popular in quantitative finance in the industry also on our platform and yeah that's why I want to give you a little in-depth insights to quantitative finance yeah first of all let's maybe define what is quantitative finance so quantitative trading is generally speaking the methodical way of trading so you come up with with an algorithm with a computer program with a rule that automatically buys and sells to markets so a simple example would be if gold gained five percent in the last 10 days then we buy gold if gold lost five percent in the last 10 days we sell gold this is a simple rule it can be tested on all the known data that is available and you can see how well this rule would have done in the past so quantitative trading trading tries to find and exploit recurring patterns in equity or futures prices yeah the big advantage of quantitative trading is compared to discretionary trading that it eliminates much of the human error from trading so even the best trader might have a bad day and might make a different decision on a bad day then on a regular day computers always decide the same way all right it's hard to hear okay yeah thanks okay so what does the quantitative finance industry look like today today the the industry is basically in the hands of a couple of large and very successful quantitative hedge funds most of them are in New York so you might have heard already off of renaissance technologies or de shaw or to sigma or Citadel just to name a few so very successful quantitative hedge funds now it's today a 300 billion dollar industry and on the one side these big hedge funds very successful hedge funds employ scientists to build trading algorithms to build these computer programs for them and on the other side we have high napped with individuals and institutional investors who then invest in the products that are offered by these quantitative hedge funds so trading algorithms developed by scientists and engineers and high net worth individuals as clients yeah our role the role of my company and also why I'm here today at least for part of the reason why I'm here today's we want to democratize this industry so we want to give basically everybody with the right skills access to quantitative finance so today only about 10,000 people work as Kwan's in the industry writing trading algorithms for these big hedge funds all day long but there are many more people out there who might be great at predicting financial time series and that's exactly what we try to do with our company want to democratize the access to that industry and it should not be your job or your title or your academic degree but Radha your skills applied on the data that decide about success or failure in the industry so we want to make algorithmic trading accessible for everyone so we're the first marketplace we connect user-generated trading algorithms with capital from institutional investors so in a nutshell you write a trading algorithm and we connected to capital from additional investors if it performs well our business model is straightforward institutional investors invest in your algorithm and you get two pockets ten percent of the profits of your algorithm the institutional investor gets to pocket eighty percent of the profits because they bear all the risk obviously so no financial risk from your site and quantity X also pockets ten percent so we charge institutional investors the industry typical twenty percent of the profits of any trading algorithm twenty percent and split that amount 5050 between us and the Quan to wrote the strategy so pretty straightforward all right but let's get back to quantitative trading and to machine learning for quantitative trading so quantitative trading is data-driven trading and that basically means you come up we have a very well defined process to write the quantitative trading algorithm so first of all of course you need data machine readable data data that can be processed by a computer then the second step would be you need to come up with a model you need to have an hypothesis of of a pattern so what what happens frequently on equity prices on futures prices is there a pattern that can be harnessed it can be exploited by an algorithm so you come up with a model this is the creative part and the fun part of quantitative trait that model ultimately should give you a prediction what happens next with the markets that's of course the goal we want to predict the future we want to predict what happens tomorrow or in a week from now at the very same market that we have modeled yeah then ultimately we use that prediction to trade simple as that all right the data you can use all kinds of data for quantitative trading the most trivial and most straightforward data is of course the market data itself so the time series of the market that you want to trade usually this time series comes with four fields this would be the open the high to low and the clothes so four data points per session basically so the first price the highest price the lowest price and the final price of a trading session and then usually also volume and informations on the market so this is the first and most common type of data that could be used for financial modeling then a second also very popular block of data would be fundamental data so for example corporate earnings numbers price-earnings ratios stuff like that for for equities fundamental data also available and also of course in a nice format all ready to work with it quantitatively then just recently in the last yeah a couple of years sentiment data became increasingly popular so this is this tries to capture market sentiment as the name suggests so you try to find out basically how traders or analysts feel or you know what opinion they have about a market and this market this data can also be of course used for quantitative modeling and quantitative predicting yeah and then ultimately we have also news data this might be the data that is most difficult to handle because it requires some natural language processing usually so if you I don't know have a Bloomberg or writers terminal and you try to parse and used and you you need to make sense of the news that you get through the terminal yeah and this is opens up a hue huge Universal problems basically yeah but it's a very interesting and then sophisticated way of course to trade the markets yeah and generally speaking regardless of what kind of data you use for quantitative trading unfortunately a financial data usually comes with a lot of noise so this is probably one of the noisiest data that you can encounter for data science yeah hi noise and typically financial data comes of course in time series so you usually have a precise value per point in time yeah some economists argue still today that financial data cannot be predicted yeah I've been in the quality in industries I have kind of a different idea about that then the economist's but at the first view this might be a pretty reasonable assumption because as you can see here this looks pretty much like a Gaussian distribution these would be the one day returns of the future of the S&P 500 so of the largest US stock market index and yeah this looks pretty much like Gaussian distribution so there's on the first view you might get the idea okay there's nothing much to gain here we have a maybe a slight skew or positive mean or something like that but pretty much a caution distribution but then on the other side if you take a look at the chart this would be the S&P 500 since 1990 so 26 years of data and I daresay this is clearly not a random walk yeah you can see that that it has a clear shift over all the 26 years upwards fortunately because companies build stuff they are successful they create revenue so eventually it says at least an up drift and then we can also see simply from looking at the chart that the face is where the market losses or usually ya have a different shape when you put it this way then the faith in faces when the market goes up so usually losses happen much faster than then gains here in the markets so this is I daresay clearly not a random walk alright so the task is to predict these markets how can we predict a time series in our case of financial time series so usually yeah we have time and we want to predict one particular index usually tomorrow or the next week or whatever what we want to predict is basically what's the price going to be tomorrow or in the next period whatever period we're looking for so a very naive approach to that kind of problem would be we simply take the time series as it is and slice it into chunks of let's say one day returns for example now this would be the blue square here and we take the history up to that point as featured as variables to predict tomorrow symp like that so this would be the other the light blue bar here so we have a series of price changes maybe let's say up until today and we want to use that to predict tomorrow's time change price change this dark blue curve here yeah if you do that for a time series so we simply split the time series into I don't know even sized chunks were always the target variable the variable we want to explain would be next periods return then we have a pretty standard problem that can be addressed with a lot of different machine learning methods so we have here basically a matrix with a lot of features variables that can be used to predict one particular variable one column here yeah the output of tomorrow so the price changed all tomorrow so this is pretty straightforward and didn't require a lot of thinking all right yeah now let's get maybe to the applied part of this very naive approach let's try to build such a modeling Madeline with machine learning tools let's take a neural network a very naive neural network and try to model the price change for tomorrow so what we're doing here basically I used hear some of the embedded functions embedded in our toolbox we offer two different tool boxes to help you to model predictive or quantitative models for trading we have one toolbox in matlab and one in python i'd personally it since I mathematician I come from a maths background so i prefer matlab but i dare say most of the audience here prefers python so the toolboxes have the same functionality in both languages we support both languages simply pick whatever language suits you better so today i'm working here with matlab because it's more convenient for me at least and what i did here is I simply load two markets I only need the first one for today's demo I load to markets of free market data from context this would be the the sp500 future the e-mini of the sp500 and the 10-year note future so a US government bond I defined your arguments that I want to have so I wanted to get the date vector open high low close and volume for every point in time in our cases and update data server every day I would get those six values basically and then i define yeah from when on and to what point I want to get the data so at this case I want to get all the data that's available after 1999 and then I simply use the load data function that's embedded in our toolbox you can simply use that to to load market data in a matrix and to do whatever you want with it machine learning or any any other thing you want to do with the market data so and then the only computation I do here is I take the clothes values so the settlements of every session the last values in every session and subtract yesterday's clothes from it so I take a look at what happened in the last 24 hours and i divide that by the close of yesterday so this would simply give me the percent return in the last 24 hours very straightforward i define that as a return this is the return in percent on that market in 24 hours okay and then i define two vectors the 1i poll in sample which is simply a subset of the returns that I've selected or computed here I take the first 75 percent of the data to build my model and I reserve the last twenty-five percent of the data to test my model after i'm done building it this is generally speaking good practice for financial modeling alright so this was all the code that is necessary up until this point and then let's take a look at matlab and what we can do with this data okay yeah this was basically here the code executed so all i have here now in my workspace is i do have the data the raw data and here are two variables the one would be in sample the other one is out of sample these are simply two vectors with the returns the last 26 years so i split that up into chunks of theta okay and now I'm going to use the neural network toolbox that MATLAB offers here so very convenient what we want to do in our first naive approach we want to predict the time series so let's hears like time series app okay we have a couple of things that we could now try with our time series we want to do in this very simple demo a nonlinear auto regressive meds so we simply take the time series and we say assume that it has this structure so the the variable we want to predict is simply just a function of the last percent changes so the percent change of the model is simply a function of the last d % changes of the same margin there are no external no other variables that influence the market we assume we can completely explain perfectly explained the price change for tomorrow with the last d values of the same time series all right so let's do this we pick this okay our targets we want to take the in-sample vector and this is a matrix column this is simply the vector of our price changes on the S&P 500 in the last 26 years so what do we want to use for training for validation for testing we just go with the defaults here so training on seventy percent of the data appears to be reasonable why not I'm testing it on fifteen percent also why not let's go for this so as you can see very convenient here in matter and this is the place where I would define the network architecture so I have here two parameters basically the first parameter is the number of hidden neurons think of it as complexity of your network so how much degrees of freedom do you want to ground your model in this case and then the number of delays so in this first first step were my first chart here on the data is that I simply take two days the last two days and try to predict with it tomorrow let's see what happens with the complexity of 10 hidden neurons okay then here we can select the training algorithm so we have a couple of choices here let's go with the second one this takes usually more time but can be a good result and a good generalization especially for noisy data sets which is good and let's train these guys this should not take too long since the my list is very straightforward okay let's give it a little time good practice generally speaking for building quantitative models regardless of if you use machine learning methods or if you do something else with the data it's always good practice to reserve some of the data for an out-of-sample test in the end you will see and this is also one of the takeaways of today's session I can tell you right right now already that most of the models that you come up with fail on an out-of-sample test and this does not necessarily mean that you did something wrong this is part of the game of quantitative finance you can have the best idea the best model even some Nobel laureates had great models long-term capital for example model everything right and then eventually fail out-of-sample this is a very good and test that you should always do because this is the closest you get to real training so if you reserve a chunk of the data at the end that you do not look at you don't use it for for modeling then you can test it on our sample data and this is the closest you'll get to your performance of your model on life data okay so now let's take a look what our algo found here in this case mm we have here an R which would denotes the correlation of our outputs and the targets so we have an R here of yeah at least almost thirty percent so within a training set we were able to explain thirty percent or we have a correlation of thirty percent of our targets and the outputs of our model this is already okayish to some extent but then here we see already where it fails so at the testing set this correlation drops to only one percent almost nothing so this already fails on the test set of the very same data chunks so this is not even an out-of-sample test yet that we have performed so as you can see this model will probably have no predictive quality at all on real financial data okay let's take a look at the responses here let's maybe find out why that is ok again MATLAB here offers very convenient plots but you can do the same thing in Python or in our so this is really there's no reason to prefer MATLAB unless you're more comfortable with it okay as you can see here so this would be a graphical representation of what has happened so the yellow lines these are the errors the small blue dots on top of here are the training targets or think of it as what the market really did within 24 hours and the Blue Cross would be the training output so what our model predicted but the market would do and as you can see the errors or the targets are far away from whatever our model predicted here and then another maybe interesting take away at this point already is that our model degenerates pretty much to a trivial estimation so as you can see here if the model usually predicts a change for tomorrow that is very close to 0 much closer to 0 then we have seen in the first histogram of the of the results so it degenerates a little here to a trivial estimation okay now as ruki data scientists that we pretend to be here today what's the next best thing we could do we could of course increase the complexity of our model we say okay it's impossible to explain tomorrow's price change with simply what happened yesterday and the day before yesterday so let's give our model a little more data let's say we want to give our model 30 days of for example instead of only two and that we don't need to wait for that again I did that already train again i want to show you the screen here okay this is 10 too it's just what we just tab let's close this guy all right here's the other guy so we do the same thing again this time with more complexity here we are okay so we have selected here okay now 30 about 20 we increase the complexity of our model which is basically the same thing yeah we just said okay it's impossible with two days let's give it 20 days or we could of course give it two hundred and whatever and then at the same time we increase the number of hidden neurons so we also give the model more degrees of freedom to find a good fit for the next day all right let's do the same thing again as you can see the structure of the lattice is still the same so there are no other exogenous variables all right what can we see here so this is the result we take a look at the responses as well okay this now looks a little more convincing even though than if the number is less so we have here a correlation of our model interestingly to only twelve percent present with the returns with the targets of our optimization and the test set is slightly better than what we had in the last but it's still down to four point five percent so this is still not satisfying or satisfactory let's take a look at the response plot here and then if we can see some some change that happened with this new model here yeah so what we can see here already at least a bit is that our model now is a little more away from from zero but still very close to zero so even with with a look back of 20 days this still degenerates to a very trivial solution even though in some faces our net already makes a little more or Boulder predictions than hit it before okay so the important thing is what can we learn from that results that's what is the takeaway of this yeah we did something that is very simple very trivial so our first takeaway might be we need a little more than that yeah we actually need to build a model we need to come up with a hypothesis so the first takeaway you might be don't think of machine learning as a black box does that does the modeling work for you think of it as an optimizer so you cannot simply take the data and throw it into a machine learning method of your choice and expect it to come up with the cutting edge model for quantitative training the data is much too noisy for that so there's much much too less information in the data that the machine would find it then the second takeaway is yeah machine learning will always find patterns even if there are not so they will tell you the you will the more parameters you use the more degrees of freedom you use the easier it is to fit a known data set so the more complexity you're allowed for the better your fit will be on historical data so this is a hearing example this is just a visualization if I got 11 data points I can fit them perfectly with the polynomial of degree 10 this is trivial I can retell the past perfectly but the predictive quality of this model is inexistent basically so a much better approach in this case to estimate these dots here or these observations would be a regression a linear regression so keeping it simple is actually good for billing among all right the second takeaway preprocess the data again financial data is very noisy that means if you throw it into a neural net you usually don't get a good result if you throw it into a neural net as it is so what you want to do is you want to filter it to smooth it and to extract features from it meaningful features features that are maybe more meaningful than the return of yesterday for predicting today's return so how could we do that we could for example simply take the data and run an SMA over it as standard moving average moving average we could use some other technical indicators or filtering methods to try to smooth away some of the noise of this chart of this financial data yeah so pre process it and feed this as a feature in your neural network then from the very same time series you can extract different kinds of information so not necessarily only a direction like we did here so we looked at the percent changes and that is a directional indicator basically and we only use directional indicators to predict the direction of tomorrow but you can also extract some information like volatility how volatile is the market and this is somewhat even though if it comes from from the same time series this can be interpreted sometimes as somewhat orthogonal to to the other information soft to the directional information so it's always good to feed your net with or your support vector machine or whatever i'm not saying here the neural nets are the best tool to predict financial time series so feed your model with real features derive those features from the market data so this is this is actually the modeling work that needs to be done yeah and then also of course normalization is an issue so in financial data let's just think of two different stocks they're both priced at one hundred dollars at the moment one of them has a volatility or is a movement per day an average movement per day of 5% the other one is an average movement per day of one percent so clearly the first one is more risky than the other this might also be something that you want to consider in your model especially if you if volatility is one of the indicators of the target indicators that you train your netbook for then it often happens that your solution converges to the least volatile market depending on the loss function that you have so this is something that you might want to consider so pre process the data normalize it and then extract features from it so the features can be much better than the features that we have used now in our first simple demo here the third take away and this is also a very important take away what we tried here was we tried to simply predict the price change off tomorrow this might not be enough even if our model is very good simply the price change of tomorrow tomorrow having a week estimator for that might not be enough because in reality whenever your trade you've got you encounter costs trading comes with costs we summarize that on context is slippage in reality it slippage and Commission so usually your broker wants to be paid usually you if you buy a market you have to bite at the wrong side of the spread for you so if you think of the order book in the middle there is nothing then this is exactly the best bit and better best ask and usually you have to cross the spread so you have to usually you get I don't know depending on the stock or what our future at least a five cent punishment so to say or a five cent worst price than you actually has to make you would get and then of course the larger your lot size is the more market impact your order has and so on so we summarize this effect in the colloids to pitch and slippage is obviously a function of the volatility of the market so the more volatile market is volatility would be price change per ton per unit of time the more volatile market is the higher is usually the slippage that you encounter our model here is a very simple model I'm not saying that this is the best ever so there are much more so many more sophisticated models than this but we take simply five percent of the daily range and the daily arrange is for us simply high- low so this is where the volatility of the session now would come into into the play so to get back on our define a YT define a variable that you want to estimate even if you have a model that predicts you tomorrow's price change that might not be enough let's assume a situation you predict today that the markets that you want to be long in the market there are your predict rising prices for today so for tomorrow session so you buy the stock all right the new model tomorrow tells you to sell the stock again so you have to basically do a two hundred percent flip of your precision from one hundred percent long to one hundred percent short this means that you encounter a lot of a lot of slippage every day and your model might simply not be good enough to overcompensate for that slippage even if you're right so even it's not necessary that the percent change in 24 hours between clothes and clothes is actually higher than a slippage and the slippage always works against you even if your prediction is right so even if tomorrow's close is higher than today's close that Delta might be lower than search so define a target function that is a little more robust I can only advise you to maybe not look one day in the try to predict one a one day return but maybe a 10 day return or a 30 day return so thinking longer term and the last and yeah maybe most important takeaway here is that a different objective function from the standard or classical objective functions that are used in machine learning can give you much better results than the normal loss function so typically a machine learning method tries to minimize the loss so basically the error between prediction and observation and this is something that we have seen here this degenerates in some cases to the trivial estimation simply because it is very noisy and with 0 you're always a doubt right and yeah basically the last function this is often actually a minimum of the loss function yeah so for financial time series you need to dig a little deeper and in order to avoid that the generation to to the trivial estimation you might want to introduce another another objective function and I can only recommend one of the objective functions that are out there is the Sharpe ratio and the Sharpe ratio is not only an objective function but it is also a performance number that's very important in finance and this is the only mathematical slide to a this is the definition of the Sharpe ratio so this is basically in finance the performance or a very central performance number that she use to compare the performance of different funds or assets or whatever so the Sharpe ratio is a very very commonly known and very commonly used number in in quant finance and in finance in general so how it how how is it derived we take again the percent returns here so this would be e would you know the equity curve or the return curve of your portfolio so we built the first memento of this curve and divide it by yesterday's values we again get the percent returns and then we simply annualize that and define a yearly volatility this is a simple simply a factor the square root of 252 we only use that to annualize the daily returns so we have here returns on a daily scales of the elite market data and on average we have 252 trading days per year so that's why we multiply the number of the square root of 252 and take the standard deviation of the returns and define that as volatility and then we have we compute of the same returns and index and compute the daily return think of this as the average daily performance of your trading system and then the Sharpe ratio is nothing else and then of course we analyze it again here's that is what what we do here return daily would be 1.0 hopefully 1.0 if it's positive 00 something and we take that too powered 252 to analyze it again and we get the and substract one this is the return of your trading strategy or trading strategy per year and the Sharpe ratio is then simply the quotient between the yearly return and the yearly volatility and this is a very again a very central performance number in finance you're not so much interested in the return that you make per year of course you're interested in return as well but you're interested in the return with respect to the risk it took you to achieve set return so simply saying I gained one hundred percent in the markets that's a strong and bold statement of course yeah but if I had to risk my basically a thousand percent of my portfolio value then this is not a good trade this is a sharpe ratio of 0.1 if I gained one hundred percent and I only had to risk ten percent of my or I had ten percent volatility that is a very good shop ratio the Sharpe ratio 10 something that's not achievable I dare say we'll see alright so this is the Sharpe ratio can be used to measure the performance for the success of a trading strategy and is also a pretty decent and pretty good objective function for machine learning all right now you might wonder can it even work at all so our first two attempts failed miserably and I only told you what we did wrong so now the question is can it work yeah and the answer to that question is yes it can this is not my trading system but we're very proud and happy to have this trading strategy obviously this is was written by jenya he's a theoretical physicist from Caltech and one of our most successful bonds he won our q3 competition with his system here over cars based on a machine learning method just FYI and we traded since a couple of months now with a million dollars in futures and we're pretty happy to train it so it has a live Sharpe ratio of of two at the moment so yes it can work by the way this is not an investment advice here yeah this is just FYI this is just what's possible with the data okay nobody can invest in this all right yes alright so let's close this session or almost close the session with good practice and pitfalls for quantitative trading the natural enemy of quantitative trading is overfitting it's easy to retell the known past if you allow for sufficient degrees of freedom so you want to keep it simple you want to feed your model with a with a few yeah I'm pretty good features actually keep your model simple don't allow for too many degrees of freedom you will fail on live data usually if you over fit then stability is the second issue you might want to check how your model react if you change some of its parameters by say ten percent does it give you still a same or similar results or does it feel completely this would be not a very good indicator if it feels completely or if it does something completely different then the third thing is performed cross validation of your model so what we do here did was it was very simple we just took the what matlab suggested so we have training sets and we have tested it can be much more scientific or a much more yeah cross validate your your your results you can use one set to Train test it on the second you can do it the other way around and so on you can do this mod with only two sets but maybe with three four or five and so on so cross validation is very important save some of the data for an out-of-sample tests this is also very important this is the closest you can get to life training out-of-sample testing and then ultimately a avoid market selection bias this is also a very common mistake especially for stock trading so usually people think they are creative or good if they pick the right stock where everybody knows that this stock one in retrospect and of course I would have picked Apple 1990 who wouldn't have so avoid market selection bias usually if there is no quantitative reason to pick a particular market don't pick it's much easier with futures then we start all right to sum it up we provide you with free market data off the end of data at the moment 49 of the most liquid US and European futures we prefer to work with futures so do our investors because futures are offering a huge universe of diverse and uncorrelated markets that are highly liquid markets yeah so the trading strategies that are the work with futures are highly scalable we can manage with hovercars 300 million that's a nice number that's usually not possible with with with the equity trading strategies so futures are offer a very diverse instrument that's why we focus on that if you prefer to work with Fox however we also have the sp500 stocks I think back until two thousand or so so our market data is free so is our software you can simply visit our website and download it download our toolbox to perform back tests if you want to come up with your own quantitative trading strategy yeah here's a brief overview of our futures data as you can see a lot of agricultural futures bonds currencies in Jesus and precious metals and other stuff yeah how can you get the market data you can simply use the functions from our toolbox toolbox s to be precise so either MATLAB or Python the load data function would give you the market data on your computer or you can simply use that link down here to download the market data we update daily you can simply visit our website side and download the market data with your own sprints and use it for whatever you want yeah we also maybe that's interesting for you we frequently host competitions just like the one that jenya has one with this algorithm where the first prices of these competitions are guaranteed investments in the performing algorithms usually of a million 750k and 500k in futures the art q6 will be announced our next competition will be announced in the next couple of days and you have time to submit a trading system until june 30 2016 and the winners as usual get to pocket ten percent of their algorithms profits so we run the algorithms we validate them three months on life data and after these three months we allocate investments to the algorithms and you get the pocket ten percent of your algorithms profits as usual on our platform this was pretty much it I say thank you for your attention and if you want to dig a little deeper in algorithmic trading with machine learning or oh good morning then I can recommend the resources on our website we have a reading list with a lot of good papers also about machine learning and then of course also video tutorials I can also recommend one video tutorial of Kenya the guy who wrote that system gave a talk for the machine learning meet up here in the Bay Area and yeah talks a little about his tricks for a quantitative trading all right this was it thanks a lot and shoot if you have questions any questions yes okay who owns the ID if you submit integrin the quant owns the IP and grants us the right to market otherwise our marketplace would be useless so we are here to connect your algorithm to capital from institutional investors that means you upload it to our machinery you upload the source code you've got to live with that so the same but you get to keep the intellectual property of your algorithm so we're not allowed to look at it to redistribute it to publish it or whatever so it remains your intellectual property yes okay does algorithmic trading influence the markets a lot and the subdue the models take that into account already is that correct okay to be honest yes it does so high frequency trading for example is also usually considered to be algorithmic trading and this influences two markets a lot but not necessarily in the negative way so high frequency traders provide liquidity and liquidity is a very nice property of the market but of course every participant in the market influence system could be a large discretionary fund that influences the market so if Bridgewater does something you better do the same thing you want to do this evening so yes algorithmic trading influences to markets like any other trading style as well and depending on the style of algorithmic trading so high frequency trading maybe even more than other size yes that's correct okay one one more question if there is one yes people submit the trading strategy and do we have a consistent API that evaluates hinges okay yes absolutely okay if we do have a consistent ap a position monitor or buy and sell API that basically tracks to the performance of the algorithm it can be used to training yes we do have that but that's you don't need to worry about that as a qantas oh the you you're left alone with the hard part coming up with the model once you did that you can upload it to our website and we take care of the live trading operation we also build up the life track record of your algorithm because this is of course what investors ultimately want to see how well your model performs on live data so they're not not that interested in the back this loan but also in how well your model perform live data and this is what we show them so as soon as we get a system on our platform we start building up its track record so every day as new market data arrives we evaluated and then if it if it gets an investment we also traded on behalf of the investor so also the investor of course never gets to see the source code it's a black box to them too all right I yeah one last question okay what kind of algorithm work better or what kind of machine learning method works better for algorithmic trading frankly if you use it wisely any of them can work that's really it I cannot recommend you support vector machines work and neural networks down so that that's simply not true it depends on how you use them the same is true for for all the filters that I've mentioned so the SMAS and then the r is high and maybe the standard deviation of the returns and stuff like that it always depends on how you use them for your specific model so from from what we hear from our advanced neural nets are pretty popular and support vector machines are pretty popular but I would not recommend you the one over the other alright thanks so much if you have questions I'll be hanging out hang out for quite a while and just