Any way to backtest an automated strategy?

Post Reply
User avatar
jimibt
Posts: 3641
Joined: Mon Nov 30, 2015 6:42 pm
Location: Narnia

BA - i get you. The cost, per seat, does seem prohibative. oh well, make do and muddle thro ;)
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

Forum debates are tricky enough without us all working from the same glossary. (There's an idea for a sticky if ever there was one. The recent difference of opinion about a simple term like 'weak market' is another example.)

Have we all now agreed that there's a difference between data analysis (usually in Excel) and what's more traditionally known as 'backtesting'....and that although the word backtesting is frequently used here, almost nobody does any?

Where I'll agree with BA Dev, is that adding that functionality to BA wouldn't just be prohibitably expensive, it's also bloody difficult. It's in the category of sounds simple but doing it in such a way as to make it meaningful is a real headscratcher. From a purely selfish perspective I wouldn't be keen to see that in BA, afterall those of us who arrive here not especially interested in sport, but interested in tech need some way to level the playing field a little :ugeek: That said, anything that encourages market participation is a good thing, so as with almost everything else, both positions have their pros and cons that could be debated long into the night, or long into a dull winter's afternoon.
foxwood
Posts: 390
Joined: Mon Jul 23, 2012 2:54 pm

Still not on the same sheet - imho data analysis is not modelling as was suggested in an earlier post - my definitions and sequence for developing a strategy ...

1) Modelling - predicting how the market will behave in response to a real world event or unusual market event - then looking at data to check and develop strategy - all about causation of changes in the market - not just chasing a statistical edge.

2) Data analysis - if not an extension of modelling then is effectively the same as data mining and/or artificial intelligence ie looking for profitable statistical combinations in the data - often with lots of filtering to improve confirmation bias ;)

3) Validation (backtesting) - applying the rules of a data based strategy to a totally different dataset to that used while developing the strategy - proves /eliminates whether the strategy has been backfitted to the original data. Vital step often omitted.

4) Testing - feeding data to a strategy implementation to ensure the strategy performs according to specification (ie what jimb asks for - I think). In practice, a bit of thought and code walkthoughs should fix most things.

5) Run live with small stakes.

6) Scale up gradually if works as intended else rinse and repeat from appropriate step.
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

From a quick scan I think I'd be happy to sign-off on that one foxwood.

Sounds like a strong framework and could be used as is, or by skipping the odd step if it's not appropriate for the idea or your facilities. I think everyone from a pure quant to a purely manual trader could perm a few of those and wouldn't be far off. I mention those just as examples of extremes of trading styles, there's no intention to suggest that either are superior or inferior to each other.

...that said, I think 4 is more than a code shakedown, it's what I call the backtest/simulation against an historic data feed. Eg Would I have actually got the prices I need? am I fast enough? how did the outages and api blips affect it? It's the step that happens in the big firms that your average one-man-band doesn't have the facilities to do and gets replaced with your step 5.

yep I see your step 4 as being the unit test phase of the (seperate) development cycle, ie does the code perform to spec.

jeez this feels like being at work again. :shock: I started trading to leave all that paperwork behind :)

4 Code/rule testing
5 Simulation (aka my backtest)
6 Start small in live
7 Rinse and repeat.
Last edited by ShaunWhite on Thu Dec 20, 2018 5:00 pm, edited 1 time in total.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

ShaunWhite wrote:
Thu Dec 20, 2018 5:06 am
This makes no sense

If your Python output goes to Excel then there's no difference, visualise to your heart's content. And aren't you working in code in both?
Use whatever suits you, both are fine, no need to justify Excel to any tech-snobs.

I think I've laboured that point enough now :)
I haven't got a clue lol; I was just quoting the woman :)

But I do like seeing "metrics" - i.e. that excessive winrate thingy
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

foxwood wrote:
Thu Dec 20, 2018 4:04 pm
1) Modelling - predicting how the market will behave in response to a real world event or unusual market event - then looking at data to check and develop strategy - all about causation of changes in the market - not just chasing a statistical edge.
Where does this fit in?

Image

Because to me, that's a "model"
User avatar
firlandsfarm
Posts: 2688
Joined: Sat May 03, 2014 8:20 am

For backtesting to be of value there has to be a pattern that is expected to be repeatable for the future. Random data can show profits when backtested that trick you into thinking you have a system. For example, if you had placed a £1 bet to win at BSP on every horse running at Southwell in 2018 who's name began with the letter "A" you would have made a profit of £402.46 after 5% commission and that wouldn't have been a one off, in 2017 you would have made a profit of £178.01. But I don't think that is a creditable system (unless somebody can explain a pattern! :) ).
ThomasAJ
Posts: 4
Joined: Mon Jul 09, 2018 11:40 am

pdenoeud wrote:
Mon Nov 19, 2018 6:19 pm
I would like to back test automated strategies (mainly for football and tennis) but I can't find any tool to do that.
Does anybody know a tool allowing to:
- create my own bot
- back test it on historical betfair data, using scores of the events tested


That feature is THE key feature to help elaborating profitable strategies. The market is quite mature now, I would be surprised that there is not such a tool somewhere.
I have read through the many posts and have not seen any mention of 'Machine Learning' (https://en.wikipedia.org/wiki/Machine_learning) nor 'Data Science'.

Broadly speaking Machine Learning (ML) is the science of determining the 'what' of 'what causes data to look the way that it looks'.

The 'what' needs to be determined before a bot (a ML model) can be created and tested against historical data.

To be able to use ML effectively one needs to have years of data science education and practical experience.

Given the above plus the fact that meaningful granular historical data takes up 100s if not 1000s of Gigabytes of storage which necessitates massively fast computers to process it which in turn means BIG $s per hour for many many hours and I think it's very clear that Bet Angel will never add a back testing feature.

In one/some of the posts Euler mentioned that he wasted years on back testing and I can fullly understand why.

Without ML you have no chance. ML is big business used very successfully by big business with big $s.
User avatar
GaryCook
Posts: 128
Joined: Sat Jan 27, 2018 6:35 pm

I wanted to make a backtesting system that simulated races based on real historical data but not using the original data itself.
However, the challenge would be how to simulate the market. Would take some serious resources.

I think it was Peter or somebody that inferred above that probably the ideal thing is just to understand the market and how it generally reacts and be the first, or contrary. Depending on your style.

Although, saying that. If anybody could achieve making such a sim it's us. :)
Post Reply

Return to “Betfair Data”