Number Crunching After Capturing and Sorting.

A place to discuss anything.
Post Reply
spreadbetting
Posts: 3140
Joined: Sun Jan 31, 2010 8:06 pm

ruthlessimon wrote:
Sat Sep 15, 2018 6:48 pm
The benefit is excel isn't biased - & looks to be finding crazy cool looking stuff.
Thanks, do you have any live bots or are you still data gathering? Playing live usually highlights any flaws very quickly but also throws up lots of quirks you might want to exploit that probably won't show up in raw data.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

spreadbetting wrote:
Sat Sep 15, 2018 7:04 pm
Thanks, do you have any live bots or are you still data gathering? Playing live usually highlights any flaws very quickly but also throws up lots of quirks you might want to exploit that probably won't show up in raw data.
Not at the moment, & the reason for this (as I've been looking through the data more closely in recent days), I can see the strategies excel are pumping out aren't optimised correctly (i.e. stupidly refined i.e. only trade between 3mins & 2mins remaining to post)) - & perhaps this is a bias of mine - but why should I trade something, that I know can definitely be improved (even if it is already an edge) - for the risk of variance/low returns.

Although that's a rotten bias! One which we've talked about before :)
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

Plausability ∝ confidence in existence. It's stats, not religion.

Si, the saddest thing I saw recently was you saying you "need" Peter. Any self respecting addict needs more than one dealer in their phone to get themselves through times of drought, and when all else fails, grow your own.

I get it though really, what you need is confirmation you're on the right track and some assurance that you're not being mugged off. It's like those situations when you realise you always call your girlfriend but she never calls you.

I like the idea of weighting recent data, it's logical. 1915 is irrelevant so 2015 is on that relevenace slope somewhere.

Sb and foxwood speak a lot of sense.
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

If edges in data sometimes don't turn into a £ profit then it's perfectly possible that £ profits don't always show themselves much in data.

A change is as good as a rest. Invest £20 in trying an idea that's technically doable even if it looks marginal.

That seems to be what pw has said, and once you're working with live data and matching you'll be in a much better place to tune it.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

ShaunWhite wrote:
Sat Sep 15, 2018 7:32 pm
Si, the saddest thing I saw recently was you saying you "need" Peter.
... Says the guy who applied for the "Day with Peter" competition ;)

But yah like you say, it's about knowing if we're on the right track/advice for taking it to next level understanding
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

ruthlessimon wrote:
Sat Sep 15, 2018 8:34 pm
ShaunWhite wrote:
Sat Sep 15, 2018 7:32 pm
Si, the saddest thing I saw recently was you saying you "need" Peter.
... Says the guy who applied for the "Day with Peter" competition ;)
How else would I be able to install my keylogger? ;)
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

foxwood wrote:
Sat Sep 15, 2018 6:59 pm
Trying to get some idea of the timescale these pictures are representing. When you say 1500 "trades" do you mean markets or are these individual price movements you're latching on to ? What's the overall timescale the picture represents, is it a year, is it a week ?

The answer to that helps to answer if there may be an edge or not imho.
Individual price movements

Let's say I enter @ 2.5, & exit @ 2.0, that equals a trade. If I re-enter @ 1.95, & exit @ 1.90, that equals a trade. In that market, I've taken two trades. Therefore, those 2 trades become 2 data points on the x-axis.

The timescale I'm currently using is 3mths. Although what's good is that Sept isn't included atm (uncleaned) - which means it can be tested on unseen data - which I plan to do come tomorrow.
LinusP
Posts: 1871
Joined: Mon Jul 02, 2012 10:45 pm

ruthlessimon wrote:
Sat Sep 15, 2018 8:34 pm
ShaunWhite wrote:
Sat Sep 15, 2018 7:32 pm
Si, the saddest thing I saw recently was you saying you "need" Peter.
... Says the guy who applied for the "Day with Peter" competition ;)

But yah like you say, it's about knowing if we're on the right track/advice for taking it to next level understanding
Start putting some money through the markets, that will teach you more than any number of backtests. There is such a fine line in being profitable that a profitable strategy on paper can fall to pieces when put live, pre-race and inplay.

https://www.betangel.com/blog/why-does- ... ou-use-it/

You have put up a few graphs but to me these are meaningless without knowing at high level what the strategy is doing, it looks like you are just overfitting.

Something I like to do is pick a variable / trigger which I believe to indicate predicted movement or the chance of wining and then graph it against the profit (variable on x axis). This quickly shows if there is a relationship and if my hypothesis has legs, if not rinse and repeat.

For example check out the graph below, I believed that a positive value should indicate that the horse is a value lay. Quickly graphing it confirmed this and that 4 as a trigger was a good place to start, I then confirmed it was valid with some out of sample data before putting it into action. This was only last week so I haven’t reviewed the results yet (normally leave it at least a month) but any bets placed will be far more valuable than any of the backtesting.
figure_1.png
You do not have the required permissions to view the files attached to this post.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

LinusP wrote:
Sat Sep 15, 2018 9:05 pm
You have put up a few graphs but to me these are meaningless without knowing at high level what the strategy is doing, it looks like you are just overfitting.

Something I like to do is pick a variable / trigger which I believe to indicate predicted movement or the chance of wining and then graph it against the profit (variable on x axis). This quickly shows if there is a relationship and if my hypothesis has legs, if not rinse and repeat.
Oh yes, I'm absolutely aware they're overfit - hence why I'm extremely cautious, & also hope to be a good example for some people that a solid looking equity curve cannot be called an edge.

But I'm coming at the problem from the perspective of an early Peter.

For example, if I don't know what a Maiden is, I can't build a hypothesis around how "I think" a Maiden should behave. Excel doesn't know what a novice stks race is - but if novice races generally contain "non-random" characteristics, Excel should find it. i.e. Momemtum strategies outperform on weak Novice races

This is what I believe my graphs show (but in a different (unknown) context) - which I'm trying to find out etc.

It's difficult to explain ;) but I'd liken it to the following real physics problem.

Our current technology detected our galaxy isn't moving like it should. Something, is causing this anomaly - but we don't know what. Discover the what, we have real knowledge (https://en.wikipedia.org/wiki/Great_Attractor)

In the context of pre-race, Excel has detected an equity curve that isn't behaving like a random edge. Something is causing this anomaly - but we don't know what. Discover the what, we have real knowledge, & potentially a real edge.

The difference between the Great Attractor, & the pre-race example - Peter actually cracked it

Certainly a ramble that, apologies lads ;)
CallumPerry
Posts: 575
Joined: Wed Apr 19, 2017 5:12 pm
Location: Wolverhampton

I think this thread, if nothing else, just shows how everyone approaches their research in different ways. I feel confident with what I'm doing behind the scenes and I think that in itself is the beginning of finding an edge. I've done plenty of reading (and still do each day) but I'm starting to take that information and do something with it that hopefully nobody else is. It's not revolutionary and some others may be doing (have been the last 10+ years) exactly the same but honestly I'm enjoying myself so what's the problem. I come home from work and do a bit of researching, tweaking spreadsheets, coming up with ideas etc and I compare it to how others sit down and play sudoku for a few hours. It's not about the money, it's become a Maths based hobby that hopefully one day will change my life and allow me to do wonderful things.

I feel like I'm at the stage where I'm close and that's exciting but also, I've covered all the basics. There's not much intermediate/advanced betting exchange information that's relevant for what I'm trying to do that is easily accessible on the internet (hopefully that means I'm not developing an off the shelf type of system). So it's hard to escape the beginner band but reading what some of the successful traders on this forum divulge really helps! Please keep sharing any useful tips for number crunching, interpreting, developing and deploying. It's helping!
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

CallumPerry wrote:
Sun Sep 16, 2018 6:56 pm
I'm starting to take that information and do something with it that hopefully nobody else is.
I've gotta try & find the vid, but I did note down the trader's exact words:

"The edge we have, & the reason I've been trading 30 yrs, is because everybody else thinks they know what they're doing"

It's a bold statement - but I think certainly can't be ruled out for explaining why some edges simply don't erode (although it's a bit of a paradox!!)
foxwood
Posts: 390
Joined: Mon Jul 23, 2012 2:54 pm

Linus gave you a practical example of how he identifies and explores a strategy based on analysis of the data.

Have you tried doing something similar and put money in the market to understand how theory and reality sometimes blend and other times conflict ?

Do that, then analyse the results and think about why some worked and others didn't.

Either then adjust and try again with a modified version or abandon that idea and move on to the next one.

Simples - perpetual rinse and repeat of ideas.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

Absolutely yus, I'll keep ya posted as the days go by - hopefully, it'll serve as a great case study for all :)
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

Anyone remember doing transformations in maths??

90degree rotation?

Image
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

I think incorporating volume into any strategy is absolutely worth doing - the problem is - when we do - it becomes almost a multidimensional problem.

For example, how do we define "low volume". Here are a couple of factors I can think of:

1. Low volume in winter, is different to low volume in summer
2. Low volume @ 5mins, is different to low volume @ 1mins
3. Low volume @ 1.50, is different to low volume @ 15
4. Low volume @ Lingers, is different to low volume @ Chelt
5. Does low lay volume, behave differently to low back volume

What an effin nightmare :D

A traditional hypothesis would be low volume suggests lack of conviction - suggesting a propensity for mean-reversion - problem is my gut says it's not quite that straightforward when it comes to pre-race (i.e. that would bias mean-reversion to occurring before the liveshow - considering volume is far lower hrs prior)

Perhaps the best way to gauge the importance of each variable - is to simply build a strategy around each concept (i.e. backfitted to time, backfitted to course, backfitted to price), then see how they compare over a week/month of forward trading
Post Reply

Return to “General discussion”