Do you Collect Data on Betfair Markets?
- northbound
- Posts: 737
- Joined: Mon Mar 20, 2017 11:22 pm
I’m not a hoarder but wrote software that takes a snapshot of metrics (ltp, best back, best lay, wap, wom, traded ladder, etc) at different points in time for a number of sports.
For example on horses I take snapshots at 10min, 3min, 1min, 0min before the off. I also integrate this data manually with other stuff, for example Racing Post tips for every race, etc.
For example on horses I take snapshots at 10min, 3min, 1min, 0min before the off. I also integrate this data manually with other stuff, for example Racing Post tips for every race, etc.
One question I have is how do people who collect data here analyse the data?
I am assuming there would be data in some sort of a database, so are there specific tools ye use to get what ye want out of that data to make sense of it so to speak ?
I am assuming there would be data in some sort of a database, so are there specific tools ye use to get what ye want out of that data to make sense of it so to speak ?
-
- Posts: 575
- Joined: Wed Apr 19, 2017 5:12 pm
- Location: Wolverhampton
Got about a weeks worth of greyhounds just to model some strategies and then replay them, I've built my own way of running simulations which is just recently working as I hoped it would months ago. Got about a month's worth of horses and no football yet but my hope is to deploy this full-time soon on those three sports. Not sure about how much in gb/mb I have, will have to have a look tomorrow if I get chance.
- northbound
- Posts: 737
- Joined: Mon Mar 20, 2017 11:22 pm
Personally it all starts with a real world hypothesis. Stupid example: horses with low draw at Chester 5f races getting backed before the off.
Then build some software script that extracts the relevant metrics from the database and exports them into Excel / CSV format.
Play with Excel filters to see if the original hypothesis has legs. For example simulate backing said horses 10min before the off and laying them at post time.
Is it profitable? What about if I filter in only stalls 1-2? What about 1-2-3? What about only horses in those stalls and a certain price range? What happens in-play? Could I take 25p or 50p profit fir each £1 on most of these horses?
And so on...
worth looking on udemy.com - under data science
Sounds simple to extract data, analyse and then create a strategy, but there is a danger of over fitting and many other ways you could go astray
You would need two sets of data, one out of sample, so you can test your theories etc
Anyway, Im no expert, Im only two steps in front of you
(By the way; if it seems too complicated...I watched a ted talk yesterday about learning. Yes it has been stated that you can become an expert of a skill in 10,000 hours, but this guy stated that with the right application and focus, yo can make massive headway with just 20 hours)...heres the link https://www.youtube.com/watch?v=5MgBikgcWnY
You may never get to the lofty heights that some achieve on here, but even if you get above average, then that should be enough
Sounds simple to extract data, analyse and then create a strategy, but there is a danger of over fitting and many other ways you could go astray
You would need two sets of data, one out of sample, so you can test your theories etc
Anyway, Im no expert, Im only two steps in front of you
(By the way; if it seems too complicated...I watched a ted talk yesterday about learning. Yes it has been stated that you can become an expert of a skill in 10,000 hours, but this guy stated that with the right application and focus, yo can make massive headway with just 20 hours)...heres the link https://www.youtube.com/watch?v=5MgBikgcWnY
You may never get to the lofty heights that some achieve on here, but even if you get above average, then that should be enough
- firlandsfarm
- Posts: 2724
- Joined: Sat May 03, 2014 8:20 am
I basically do the same using Access as well as Excel though I'm needing to gradually upgrade to SQL Server but keeping Access as my front end. The skill is not so much the extracting and manipulating of the data, I see it that the skill is spotting the difference between a statistical freak and a trend and not over fitting. For example and building on Northbound's example, if the profit was found in stalls 1 and 3 but not in stall 2 I would say that is a statistical freak but then you have to judge which is the freak … is it the stalls' 1 and 3 return or the stall 2 return!northbound wrote: ↑Mon Jun 24, 2019 12:19 amPersonally it all starts with a real world hypothesis … software script that extracts the relevant metrics from the database and exports them into Excel / CSV format … Play with Excel filters … Is it profitable? What about …
3 MySQL databases, 100gb for order / market data (anything that comes from the API), 200gb backtesting database that contains processed streaming data and another which is more of a ‘data lake’ containing aggregation of everything (including logs) which I think is around 200gb.
I then use S3 for all logs (last 120 days) and raw streaming data:
- UK/IE racing since streaming began
- AUS/US racing past 12 months
- Greyhound racing past 12 months
- Tennis, major tournaments
- Other random sports markets
I then use S3 for all logs (last 120 days) and raw streaming data:
- UK/IE racing since streaming began
- AUS/US racing past 12 months
- Greyhound racing past 12 months
- Tennis, major tournaments
- Other random sports markets
- ruthlessimon
- Posts: 2096
- Joined: Wed Mar 23, 2016 3:54 pm
I like thatfirlandsfarm wrote: ↑Tue Jun 25, 2019 4:56 amThe skill is not so much the extracting and manipulating of the data, I see it that the skill is spotting the difference between a statistical freak and a trend and not over fitting.
Does it look "beautiful"? Is it "elegant"? Does it have smooth, defined, consistent curves - LOL
I'm pretty certain someone like Peter will get joy at the "aesthetics" of his strategies
-actually here's a checklist:
A proof that uses a minimum of additional assumptions or previous results.
A proof that is unusually succinct.
A proof that derives a result in a surprising way
A proof that is based on new and original insights.
A method of proof that can be easily generalized to solve a family of similar problems.