Any way to backtest an automated strategy?

[PeterLe]

Ive never used back testing either (although I know people who do to give you an 'indication')
Even if you find a pattern, lets say laying at 1.01 on certain courses..it won't necessarily work (I read this link by Smarkets with interest this week.. link posted by smarkets https://en.wikipedia.org/wiki/Adverse_selection). In the instances you are wrong, your bet will get matched and the times you are right, your bet will get matched "sometimes"
(Which incidentally, That is why you simply cant take a losing strategy and do the opposite to develop a winning strategy)
Regards
Peter

[Atho55]

So how many commenting on this topic are actual BA users as in using the software or just forum members using other software they have developed or had made by others.... BA user

[ruthlessimon]

backtesting is simply testing an idea against a data set i.e. I think laying every 1.01 will reap rewards so I test it against the data then start booking my holidays as this time next year we'll be millionaires with such a ROI.

I've just backtested it, a £2 stake @ 1.01, 1.02, 1.03

6mths (6000+ markets)

1.01: +£51
1.02: -£68
1.03: -£187

Backtesting has allowed me to see that there's 0 leeway - 1 tick matters hugely - you won't catch me trading this £mil dollar strategy

I prefer to see something working at a range of prices - just so I know if I miss the fill it's ok.

Given that same token, Peter doesn't like back-testing, & Peter is a far far better trader than me, hence why I'm fascinated in this alternate view to idea generation & testing

[spreadbetting]

At the end of the day you need to be playing to your own strengths not Peter's. There's always more than one way to skin a cat.

[foxwood]

I've just backtested it, a £2 stake @ 1.01, 1.02, 1.03

And there's the difference !

Your "backtesting" process shows you could make a profit of about £100 pa by laying the field at 1.01 for £2 stakes (£240 liability) but you dismissed the strategy.

[comment: if you laid for £100 stake you could make £10k - would that be attractive based on the backtest ?]

Anyway for various reasons you probably wouldn't make the profit level as per your backtest so you are probably right to let it go. There are however people on this forum who do profit nicely from that classic strategy - think on

So what is the difference between what you did and modelling the 1.01 lay ?

To me, I would model it by thinking what would happen to the market if the certain winner suddenly looks like losing. Follow that line of thinking and also consider how to manage the delay and the risk of only one runner trading at 1.01 and you end up with a strategy that could be nicely profitable with very little downside.

Backtesting is looking at consequences - the data values and moves etc shown in the data are the result of something that happened and just because there is correlation doesn't mean common causation.

Modelling is looking at causation and using the data to confirm your view of what the consequences would be.

Simples ? Probably not - may be coat time

[Euler]

I'd say that was a good summary.

[ruthlessimon]

To me, I would model it by thinking what would happen to the market if the certain winner suddenly looks like losing. Follow that line of thinking and also consider how to manage the delay and the risk of only one runner trading at 1.01 and you end up with a strategy that could be nicely profitable with very little downside.

Backtesting is looking at consequences - the data values and moves etc shown in the data are the result of something that happened and just because there is correlation doesn't mean common causation.

Modelling is looking at causation and using the data to confirm your view of what the consequences would be.

Simples ? Probably not - may be coat time

Blimey, that's gonna take some time to get my head around (getting my head away from causation having to be linked to something fundamental)

Really really good insight there though Fox, appreciate it!

No, it defo isn't simple - no-one else was able to explain that view so concisely

[ruthlessimon]

To me, I would model it by thinking what would happen to the market if the certain winner suddenly looks like losing. Follow that line of thinking and also consider how to manage the delay and the risk of only one runner trading at 1.01 and you end up with a strategy that could be nicely profitable with very little downside.

Here's the way the way I think - if you (or anyone) can explain the error I'd be so so grateful.

We know laying 1.01 generically, back-tested is (technically) an edge - although done IRL, probably ins't (hence the dangers).

Let's say, I watch a race live, & notice 1.01 is hit extremely early, before the race has even reached half-way - & the fav gets caught.

I subsequently make a hypothesis that 1.01s hit early in a market tend to fail - & backtest it.

What's the error, basically why isn't that a model?

Cos, that's how I think, & those are the types of ideas I generate daily (albeit using technicals, rather than fundamentals)

[LinusP]

I don’t think I have read a thread on this forum with so many conflicting opinions, which if you ask me is a good thing.

I backtest but looking back it has been very good at telling me what definelty doesn’t work, helpful? Yes, very. I follow Peter W’s advice In getting money through the market ASAP, however the advice being given is very much dependant on the market, sport and preplay/inplay. Without knowing what different forum members specialise in you are in the dark. The sensitivity of backtesting can vary a huge amount and you want to limit this to as close to zero as you can.

[sa7med]

But to model the market and understand how it works with a view to exploiting that understanding of the market.

Would you explain in a little more detail what that process is? Would love to see a video on modeling, are there any existing ones? Is the result something like soccer mystic or tennis trader? How would it work for racing? Are you modeling likely odds or momentum of trends or everything and anything?

[ruthlessimon]

Interesting read:

https://blog.quandl.com/interview-with-a-quant-part-one

My pick:

"I use Excel for back-testing. This is a deliberate choice: Excel is not as powerful as Python, and this means there is an upper bound on how complex I can make my trading rules. This is a good thing: a strategy that requires complexity to be profitable is probably not a good strategy in the first place.

Excel also allows me to see my assumptions made explicit; it’s easy to lose track of such things when you’re working in code. It allows me to visualize performance statistics (risk, return, drawdowns, capital efficiency, Sharpe ratio and so on) quickly and clearly. Even if my model “works”, there’s no guarantee that a trading strategy built around the model will be economically viable, so these statistics matter."

[ruthlessimon]

Btw Peter; enjoyed the video

[ShaunWhite]

Interesting read:
"I use Excel for back-testing. This is a deliberate choice: Excel is not as powerful as Python, and this means there is an upper bound on how complex I can make my trading rules.

This makes no sense, just because Python executes faster (it's not inherently more 'powerful') there's no need to overload it with rules. Simple rules in Excel are no different to simple rules in Python.

Excel also allows me to see my assumptions made explicit; it’s easy to lose track of such things when you’re working in code.

If your Python output goes to Excel then there's no difference, visualise to your heart's content. And aren't you working in code in both?
Use whatever suits you, both are fine, no need to justify Excel to any tech-snobs.

-------------------------
I do find the continual bluring between analysis and backtesting strange for a community who live and breathe this stuff. I'm sure others here who've worked in tech esp financial tech, must feel the same way.

Simon's analysis on 1.01 above isn't backtesting, it's data analysis or modelling.

The whole purpose of a backtest is to see if a solution found by analysis would have suceeded in a live environment. Otherwise, what are you testing? And if you call that backtesting, then where's your analysis? Backtesting requires the best available simulation of the live environment. Not necessarily amazing, but the best you can piece together (Something easier to do in Python than Excel btw).

The analysis would show there might be an edge. The backtesting would show it's not obtainable.

That's presumably why in the video, Peter says he looks at lot of data (analysis) but doesn't do much backtesting. Two different things. I suspect that's because when Peter started trading, backtesting would have been really difficult, so he 'grew up' on analysis and then going straight to live. But if you have the capacity to do backtesting of your analysis, then why not? Peter started in the 20th century, times move on, tech improves and the subject matures. Someone looking at this afresh in the 21st century certainly has more options available to them. He'd probably be the first to admit he's not a 'techy' and doesn't do any/much hardcore coding. But if you can............I wouldn't be so keen to dismiss backtesting.

I think Jimi gets what I mean, Jimi can Excel the pants off anyone, but lobbys for the ability to replay data. He does his analysis, now he wants to backtest, I guess? The forum has got lots of threads on analysis, I've never seen one about backtesting and it would be interesting if there were.

I think I've laboured that point enough now

[jimibt]

I think Jimi gets what I mean, ...... I've never seen one about backtesting and it would be interesting if there were.

this is exactly my point Shaun, you arrive at your conclusion(s) via the analysis and even if the automated rule is super simple, it's still a pain to have to backport it into another coding language (which in turn could introduce small differences that behave in a different way to the BA rule).

having a replay facility literally would assert whether your analysis and more importantly, your BA rule, was functioning as intended. this replay facility would also highlight areas that needed defensive coding etc. the aim (from my perspective) is not to hack our way thro' random code changes, but rather to test that the assumptions are correct in the environment where it will all play out.

[Bet Angel]

having a replay facility literally would assert whether your analysis and more importantly, your BA rule, was functioning as intended.

The issue here is what would you replay? We would not be allowed to sell a product that allowed you to replay your own self-captured betfair data, it would have to be the official data sold by betfair itself. The only data that they sell which is of the quality and detail to reflect the actual market is £199 per month per customer per sport. I don't think we'd get many users willing to give betfair that amount of money for data.
We've looked at the cheaper data that is available that would be allowed to be resold, but it's pretty useless and would give far too many false signals. Imagine the differences between current Practice and Live mode but magnified ten-fold. It's such a fine line between any profitable and loss-making strategy, that such differences are too significant.

Considering the amount of development effort required, any replay service would need to be actively used by 100+ customers to justify spending the time on the feature. We just can't see that many people willing to purchase the £199 data on a regular basis. Perhaps if betfair get more realistic with their data pricing, we can revisit this.