Any way to backtest an automated strategy?

[Euler]

My issue is to back test a tennis bot, using the historical Betfair data about odds AND the corresponding score.
I think I know the "well known software" you mention, but it does not allow to use tennis scores within the bot.

This more or less highlights the issue I was speaking about. If you backtest against scores you will find the market is pretty much perfect in terms of pricing and scores. It's not what has happened that's going to determine your profitability, it's predicting what will happen.

Tennis Trader perfectly models Tennis scores and odds, so playing with that will allow you to plot all possible paths and your goal as a trader is to anticipate price action rather than follow it.

[Euler]

If you take price action at the centre of the book for example, you need to know the fill rate. No amount of backtesting will tell you that, so you need to actually put money in the market to test it under different scenarios.

The only reason I'm saying this guys, is that I've wasted soooooooooo much time coming up with really neat strategies and ideas only to find they fail at the implementation stage. Some of my best strategies were formed by putting something into the market and realising I'd spotted the reason for failure and going on to exploit the reasons for that in a positive way.

You shouldn't underestimate the amount of time you will save by moving to implementation quickly and testing it live. You may even find that your presence in the market may influence the outcome.

[xitian]

Yeah, you’re asking in the wrong place I’m afraid.

This is indeed a forum for BetAngel (or completely unrelated stuff like investing). So any methodologies different to those that BetAngel supports or recommends probably won’t be received well.

Unfortunately there aren’t many other well documented and active communities which are truly impartial. With reference to backtesting and systematic strategies, all I can say is that it’s a perfectly valid route. You can earn as much as 7 figures a year if you’re good, and not have to click a button on a day if you don’t feel like it. Because Peter hasn’t succeeded in this area it doesn’t mean that you can’t or shouldn’t look into it. However to be really flexible I would recommend learning to program, and for many people that’s probably a big barrier to entry (but also a benefit).

There might be other software out there that does backtesting, but not sure how flexible they are. In the end it’s best to have full control and write stuff yourself. Betfair provide good documentation and sample applications, and I seem to remember LinusP has released some Python code for accessing streaming data before.

Entry costs if you’re going to program yourself are going to be higher though. You’d need to stump up for an api key (£300) then either spend ages collecting data, or paying for it and spending a few hundred pounds. You’d also want to hire a server to run stuff on, but that’s further down the line.

If you want to discuss anything more specific, let me know. Perhaps I can open a reddit topic where people can talk about platform independent stuff.

[Euler]

So any methodologies different to those that BetAngel supports or recommends probably won’t be received well.

That's not true or fair, that's more or less saying that I'm not allowed to have an opinion. The only reason I have opinions such as one of this thread is because of my experience in the market. It's nothing to do with promoting a viewpoint. I actually stay out of a lot of active discussions to avoid tainting them.

We are happy for people to discuss a wide range of topics but have suffered from the forum being abused in the past, so try to toe that line between active discussion and not letting people directly or indirectly use the forum for the wrong reasons. Given the growth in the forum and the type of people on here, I'd say we have that about right.

We don't have any objections to people discussing the pro and cons of backtesting here, I can't see why we wouldn't. It's good to have differing views and to encourage debate. Removing a backlink to somebody that previously abused the forum doesn't change that.

I've used back testing in the past, but prefer to model a market and apply ideas directly to the market. I'm just not a believer that backtesting on it's own is good enough to get confidence in a process.

[pdenoeud]

Yeah, you’re asking in the wrong place I’m afraid.

This is indeed a forum for BetAngel (or completely unrelated stuff like investing). So any methodologies different to those that BetAngel supports or recommends probably won’t be received well.

Unfortunately there aren’t many other well documented and active communities which are truly impartial. With reference to backtesting and systematic strategies, all I can say is that it’s a perfectly valid route. You can earn as much as 7 figures a year if you’re good, and not have to click a button on a day if you don’t feel like it. Because Peter hasn’t succeeded in this area it doesn’t mean that you can’t or shouldn’t look into it. However to be really flexible I would recommend learning to program, and for many people that’s probably a big barrier to entry (but also a benefit).

There might be other software out there that does backtesting, but not sure how flexible they are. In the end it’s best to have full control and write stuff yourself. Betfair provide good documentation and sample applications, and I seem to remember LinusP has released some Python code for accessing streaming data before.

Entry costs if you’re going to program yourself are going to be higher though. You’d need to stump up for an api key (£300) then either spend ages collecting data, or paying for it and spending a few hundred pounds. You’d also want to hire a server to run stuff on, but that’s further down the line.

If you want to discuss anything more specific, let me know. Perhaps I can open a reddit topic where people can talk about platform independent stuff.

I am an experienced IT engineer, I already have a BetFait API key, I have the framework to write bots (in Java language). I could write bots in other languages actually, learning new langages would not tak eme very long.
BUT I have no way to back test my bots. Writing a back testing framework, using the Betfair data (which is costly), would take very long, it is not an easy task.
And Betfair data does not include scores. I can't see how I can easily link the evolution of odds in time with the evolution of score. Football is quite easy, but tennis is not, and I am more interested in tennis. Unless I deduce the tennis score from the odds...

That is a lot of work on a path which might lead nowhere (according to Euler), so I am hesitating:-)

[pdenoeud]

You may even find that your presence in the market may influence the outcome.

Euler, this is true only if your stakes are big compared to available/traded volume in the market, right?

[Euler]

The way to look at Tennis is that a player can win a point on serve. So say that's a 50% chance, then to win 40-0 you need to calculate 0.5^4

If you look at all available test data and find that the average chance of winning on serve is 50% then backtest that to all available data and odds then you will find it matches the model perfectly. So the defining characteristic actually becomes the chance of winning a point on serve in that particular match and how it is changing and nothing to do with the historic chance of winning a point on serve. The odds will move to reflect the number of paths left to finish the match. If you model it, you can plot all available paths and work to anticipate them.

I suggest the best way to test this is to model just one game of tennis then backtest to check. Tennis is specifically related to the underlying score and any change in momentum or physical condition of the player.

[Euler]

You may even find that your presence in the market may influence the outcome.

Euler, this is true only if your stakes are big compared to available/traded volume in the market, right?

It would be proportional, yes.

[xitian]

So any methodologies different to those that BetAngel supports or recommends probably won’t be received well.

That's not true or fair, that's more or less saying that I'm not allowed to have an opinion. The only reason I have opinions such as one of this thread is because of my experience in the market. It's nothing to do with promoting a viewpoint. I actually stay out of a lot of active discussions to avoid tainting them.

It may not be to do with promoting a viewpoint but you invariably do because of who you are as the BetAngel owner, feature designer, and “most profitable horse trader ever”. Perhaps you should create an anonymous profile and post from that if you truly don’t want to influence people.

That is a lot of work on a path which might lead nowhere (according to Euler), so I am hesitating:-)

Exactly my point. Don’t let Peter’s weight sway you.

Peter’s wrong here, backtesting can be a hugely powerful tool if used correctly.

You already have the right skills, and that’s really the biggest time sink (learning the skills). Writing a backtesting simulation system might take a couple months depending how much time you spend on it? Imagine how many years you could use it in future though, and how many ideas you can trial. Just make sure you keep some out of sample data, and make sure you know what assumptions you’re making when you backtest/simulate.

[xitian]

The way to look at Tennis is that a player can win a point on serve. So say that's a 50% chance, then to win 40-0 you need to calculate 0.5^4

If you look at all available test data and find that the average chance of winning on serve is 50% then backtest that to all available data and odds then you will find it matches the model perfectly. So the defining characteristic actually becomes the chance of winning a point on serve in that particular match and how it is changing and nothing to do with the historic chance of winning a point on serve. The odds will move to reflect the number of paths left to finish the match. If you model it, you can plot all available paths and work to anticipate them.

I suggest the best way to test this is to model just one game of tennis then backtest to check. Tennis is specifically related to the underlying score and any change in momentum or physical condition of the player.

So wouldn’t it make sense to have a backtesting system where you can test your model of calculating or predicting a server’s chance of winning a point?

Then I could see how accurate I had been over 1000 matches. Or tweak my model to fit 500 matches and test 500 out of sample matches?

[pdenoeud]

If you backtest against scores you will find the market is pretty much perfect in terms of pricing and scores. It's not what has happened that's going to determine your profitability, it's predicting what will happen.

Tennis Trader perfectly models Tennis scores and odds, so playing with that will allow you to plot all possible paths and your goal as a trader is to anticipate price action rather than follow it.

BUt for instance, tennis matrix (game and set) show estimated odds in the future. I guess you refer to that in your post. Those estimates can be used to start or finish a trade at "good" entry/exit points. But those estimates are themselves calculated accoring to historical data I guess. So the past helps us modeling the future and take good decisions.

Also, if "he market is pretty much perfect in terms of pricing and scores", then those "good" entry/exit points wouldn't really exist : the odds would always reflect the estimated ratio reward/risk. And so the only way to be profitable would be to consider other parameters (than score and odds) in order to take the right decisions (like how tired/motivated/emotionaly balanced/... a player seems to be for instance). Thus you would need to follow the match live in order to take the right decisions...

[Euler]

It may not be to do with promoting a viewpoint but you invariably do because of who you are as the BetAngel owner, feature designer, and “most profitable horse trader ever”. Perhaps you should create an anonymous profile and post from that if you truly don’t want to influence people.

TBH, If I used an anon profile then people would claim I was being underhanded. Ultimately I've learnt that If you have a high profile you can't win really even if you put your money where your mouth is, I don't really have an issue with that, it's just the way it is. All I've ever wanted is for the markets to gain broader legitimacy against the prevailing negative background they attract.

Anybody that does anything clever or innovative is to be applauded.

My views on backtesting are not that it's useless, it's just that if you just use that on its own, it's not much use. If you use it to model or simulate the market then it will be obviously useful.

[jimibt]

It may not be to do with promoting a viewpoint but you invariably do because of who you are as the BetAngel owner, feature designer, and “most profitable horse trader ever”. Perhaps you should create an anonymous profile and post from that if you truly don’t want to influence people.

TBH, If I used an anon profile then people would claim I was being underhanded. Ultimately I've learnt that If you have a high profile you can't win really even if you put your money where your mouth is, I don't really have an issue with that, it's just the way it is. All I've ever wanted is for the markets to gain broader legitimacy against the prevailing negative background they attract.

Anybody that does anything clever or innovative is to be applauded.

My views on backtesting are not that it's useless, it's just that if you just use that on its own, it's not much use. If you use it to model or simulate the market then it will be obviously useful.

i think all i've ever suggested (both here and in my BAMI post) is that BA develop the core engine further to allow historical markets to be downloaded and backtested inside the current BA UI/guardian.

Apart from generating additional revenue (for BA), it would allow users to decide from the get go whether a strategy was viable or not, rather than spending weeks/months finessing something that is ultimately fragile and suceptible to minor changes in the market(s). the ability to (more) *easily* identify issues in the context of a set of historical markets, runnning against an automation rule, (whislt using all the standard BA features) is invaluable - and obviously as more features are added, all the more valuable.

[edit] - i'd go as far to say that such a piece of functionality would make the testing/validation of new features (at BA HQ) far easier to boot...

[ShaunWhite]

I think there's several things being discussed here, strategy hunting, proof of concept, refining and tuning etc. All are subtly different and all do benefit to some extent or another from looking at historical data.

'Back testing' is rather more complex than just looking at snapshots of historic data, to be effective I beleive it has to take the form of a full simulation to take into account the things Peter talks about such as fill rates. Doing so is extremely complex as you need to remember what money in the markets wouldn't be there if you'd already taken it, and what money should be in the market if you'd been putting it there previously. Even small amounts have an impact as the sports markets are tiny, often just 2 digits at any given price.

I've just come back from a weekend away at my bro-in-law's (so what you say)....he happens to be head of dev at Morgan Stanley FX which is a strange coincidence as I was in a similar position at OMAM, yet how I met my wife was nothing to do with the city...I digress. We talked bout strategy development and testing quite a lot although details were scant as security there is extreme to say the least. The message is very similar to the way we'd ideally operate; establish a concept through fundamentals, prove that concept in static data, and then use a highly sophisicated simulator (aka backtest). When that's done, forward testing usings live data and simulated orders. At no point do they consider throwing a few million at the market to see if an idea floats, that's only done when the simulation is shown to be profitable beyond the margin for error expected during implementation.

As for the tools you need then that depends what you want to achieve, you're not going to be able to do large scale HFT with excel, and if you have a system capable of that then you wouldn't use it to trade one or two specific setups a day which would be best done manually. A lot is said about how long it takes to become a profitable trader (for arguments sake call it a year), depending on the route you choose it might be the case that 10 months out of that year is learning and building a system that suits your needs. If that's not appealing, then it's perfectly possible to use a product like BA but you may have to take a more suck-it-and-see approach, testing ideas in the live markets.

It's all a balance of what type of trader you want to be, what skills you bring to the table, and what unpaid work you're prepared to put in to give you a fighting chance.... and how much money you have to invest in ideas. One talented trader I know says you should consider the costs as the price of the ticket to enter a prize draw; how likely you feel you are to win and the size of the prize you want dictates the amount you're prepared to pay for that ticket.

[ShaunWhite]

As for a product that will unlock the treasure chest, the potential user base is smaller than the potential gains to be had keeping it to yourself. And what would you charge for it, what's it worth? £10k per licence? Then data fees on top of that.
I mentioned security at MSFX, just fyi, nobody has external email, phones handed in under lock and key, no printing documentation or code fragments, instant dismissal and probably legal action if those rules are broken. It's like GCHQ.