How do you know you have a profitable system?

[PeterLe]

I think the following should do the trick: Open excel or libreoffice and put your profits and losses in column A. On B1 put 0. This is the amount of money you want to prove you can (on average) at least make. Now go to data->statistics->t-test. For 'variable 1 range' enter column A. Set 'variable 2 range' to a single cell: B1. On 'results to' enter C1 and hit OK. Around C1 you will see amongst others see 'alpha', 't-stat' and 'P(T<=t) one-tail'. If alpha is 0.05, and t-stat is lower than P(T<=t), it means with 95% certainty that you will in the long run make more than 0. If not, either you don't have an edge or you need more data to prove that you do.

You can play around with alpha and B1 to answer questions like 'am I 99% sure my average PnL will be more than 10 pounds'?

(if there are any statistics gurus that want to comment, please do - this is not my area of expertise but I'd love to learn more)

I read the above post by MarksMeets and thought this was worthy of a thread on its own (Hope you dont mind M?)

How do you go about testing to see if you have a profitable system and what could you expect on a month basis and an annual basis?

This is how I test it, and would be grateful for the statisticians amongst us to confirm its about right or way off the mark...

This is a REAL senario that I will use for an example its not been running long, but its looking good...

This has been running for 189 Races

Ive worked out the strike rate and what the average win is and average loss and then extrapolated it across 9000 races (ie approx 1 year of racing)

So by doing this I can see that if things were to continue it would produce circa £16,648

I then created a monte carlo simulator using a random number generator in excel (=((RANDBETWEEN(0,1000)/1000)) for 9000 individual entries
if the random number is less than 0.3545 then its a win and if not its a loss and then broke it down by month (Tabulated in the bottom table)

It can be seen that just by pure randomness (ie no changes to the system), Feb/Mar were low whereas Oct looked high

Overall, Im fairly confident to continue

(Ill ask another question later about how to increase stakes (Some form of Kelly perhaps?), but just wondered how you test yours or any comments
Thanks
regards
Peter

Ps I just run the monte carlo ten more times, these are the figures (in £)

1 16483
2 17211
3 17107
4 16483
5 14769
6 13418
7 18224
8 16249
9 17782
10 15730

[marksmeets302]

Great to give this question it's own thread! The thing with statistics is, if you look at it a slightly different way you get massively different answers. And this drives me nuts.

Right now I'm looking at the following question: I have a system that works, and I can make it even better by tweaking one of the variables. By testing it on historical data I can pick the best value. I know that this can lead to overfitting, so if I do a test for significance again I have to compensate for the fact that I already picked the best variation. From the internet I've already found 3 approaches on how to do that and they all give different results...

In despair I picked up the phone and called the local university. Very happy to report they're willing to help me! Once I get the answers I'll provide them here as well.

[marksmeets302]

PeterLe, wouldn't it be better to work out the standard deviation of the results of the 189 races you had, and use that to give a prediction of your year-end result?

For instance, if your average pnl is 2, and the standard deviation 10, then after 9000 races with 95% certainty you would have made between
(2 pounds)*9000 - 2*sqrt(9000)*(10 pounds)
and
(2 pounds)*9000 + 2*sqrt(9000)*(10 pounds).

This works out to be 16102 and 19897.

[xitian]

Good topic! I'm pretty amazed that I've got this far in my trading/betting life without learning proper statistics. I think it's because I originally started with trading and basically anything which wasn't profitable in one whole day's trading I didn't consider safe enough. Now that pure trading has got harder, plus has limited scalability, I've begun to delve more into more risky betting systems, plus taken on more volatile sports like in-play football.

Peter, I agree with Mark in that I think the standard deviations of your wins/losses is quite important in this case. Basically, how scattered are the data points you're trying to predict? You can imagine that if you have wins and losses ranging from +£500 to -£400, that your total profits of £349 could be wiped out by one loss. Alternatively if the wins are always exactly £18.62 and loss always exactly -£7.36 that a few more losses in your sample wouldn't have a great impact on the overall profit of £349. Mark's t-test should reflect this.

As a simple eyeball test, your average profit is £1.85 per race, on a turnover of around around £11.35. That gives you a margin of around 16% which seems unrealistically high. Of course I've assumed each win/loss per race is just a single bet and not a set of trades, which would increase the turnover and potentially make it more realistic. From experience, trading systems achieve around 0.25% to 1% profit on turnover, and betting systems might have an edge of something like 1% to 5% on turnover. (I've found that in-play racing can create very high margins though, but you already know that )

One other thing to consider other than "Does my system show enough promise to invest real money into?" is when do I STOP the system if it starts to lose money? Is it just an unlucky period, or was my initial testing period just lucky? What I do is decide on the maximum running loss (max drawdown) I'm happy to sustain before I absolutely must shut it down. If you don't decide this in advance then it's like entering a trade without knowing where you'll exit if it goes to pot. In the worst case, it acts as a starting bank (or stoploss) you're prepared to lose in exchange for the potential upside of the strategy, and on the upside it acts as a sort of trailing stoploss on profits.

Staking is other thing you mentioned, and I'll generally base my stake on what I predict the maximum drawdown to be. So for example if I have a bank of £1000 (max I'm willing to lose before I call it quits), but the stake I use could potentially cause a drawdown of -£2000 then there's a chance I'm going to go bankrupt. Estimating what the potential drawdown is is the tricky part! I think Kelly staking is fine, but you need to know your edge quite accurately otherwise you could be in trouble. Plus Kelly won't take into account diminishing returns and market impact of bigger and bigger stakes.

And of course another consideration is just how much you can stomach psycologically. A system might indeed be profitable, but if it means potential drawdowns of £1,000s with overall profits of £1000s, do you know you can handle it without losing your bottle?

[xitian]

Right now I'm looking at the following question: I have a system that works, and I can make it even better by tweaking one of the variables. By testing it on historical data I can pick the best value. I know that this can lead to overfitting, so if I do a test for significance again I have to compensate for the fact that I already picked the best variation. From the internet I've already found 3 approaches on how to do that and they all give different results...

In despair I picked up the phone and called the local university. Very happy to report they're willing to help me! Once I get the answers I'll provide them here as well.

I'd definitely be interested to hear what you find out!

I'd plot a trendline for profitability against value of the parameter. If it's very volatile or even loss making for some values of the parameter then it's not robust you should be careful! You did say it's optimisation rather than being dependent on it though.

I've always just backtested over as much data as possible to optimise, but agree with the overfitting argument. Ideally you can remove the parameter altogether, or else somehow trade a range of values for that parameter. Not always possible though.

[PeterLe]

Good topic! I'm pretty amazed that I've got this far in my trading/betting life without learning proper statistics....

James/Mark,
Thanks for your comments.
James - I just dropped the latest p&l on this strategy into a spreadhseet..

Total Races = 185*
Total Profit (after Comm) = £472
Avr Profit/Race = £2.55
Total Stakes = £5577
Avr Stakes/Race = £30.14
Margin (profit/Turnover)= 8.46%
Biggest Loss = -£11
Biggest Win = £104.14

Im only using small stakes whilst I establish if this is a viable system and is very early days I hasten to add

(*- My opening Post said 189 races. The exact start time of this new strategy was a bit sketchy, so I took it back to a date when I know for def it was running, hence theslight change in fact/figs)

The average /race can comprise of a single bet of a series of bets and yes margins are better in play

I know exactly what you mean by "at what point should I stop". Last summer I had what I thought was a cracking system on the greyhounds. The profits were just north of £3k I seem to remember. Then suddenly the profits started to drop.
I remember posting on here at the time and asked if there was a seasonality factor (think Gazuty confirmed there was).
After reviewing the data I came to the conclusion that it was probably down to luck. So I decided to scrap it (it was still a couple of thousand up), for the winter and return to it again this spring (I never got around to it and focussed more on other systems - the lower hanging fruit!)

I have spent a lot of time studying Kelly and its more more difficult to apply in play than pre off. I have my own version of kelly which is calculated on the fly, its simple but it works a treat. (thats why from the above you can see my biggest loss is only £11 and my biggest win is £104)

What I was thinking of doing is t increase the stake by a small amount (Say £1) each time it made another £250. It doesnt sound much but this tiny compounding effect really takes off and soon it hits a level of stake that would be ridiculous in play!

Ill def take a look at the stn dev suggestion* (See footnote)
Thanks gents
Regards
Peter
PS I think this is why I really love this game..(even if Betfair banned the UK frm using real money, Id probably still trade the markets each day just to see "what happens"

Edit, Mark I revisited my early college days and got my head around this using the P&L to the end of day..

* - Standard Dev 16.55426
So I make this £19,845 to £26,954 with a 95.4% chance on current stakes?
Enjoyed working through that; thanks for the suggestion!

[marksmeets302]

That sounds right. Congratulations on finding this. Looks like your grandchildren have more ice cream coming their way

Just got back from the university. They were actually quite happy with the question as they were looking for a project they could present to their students. Only drawback is it will be scheduled for November. So it will take a while before I get an answer.

[BobHigt]

i dont think i have a system yet, i'm still a newbie

[JohnTitor]

How many bets/trades do you guys think a system needs to have done before you can be fairly sure it is profitable long-term? I have a few automated systems that I've tried out with small stakes in Guardian the last few months, and some of them looks quite promising, but it might of course just be a lucky streak as it is still early days. But I am unsure of when I should be confident enough to up my stakes? After 100, 500 or 1000 trades? What do you think?

[marksmeets302]

There's not a particular number as an answer to this. It depends on how your profits and losses are distributed. It takes far less samples if you have a strategy that produces nothing but the same kind of positive numbers (ie, if you plot your net.liq., it's a straight line pointing up).

But, if half your profits are negative, but the other half has bigger positive profits it will take a lot more samples. If you would plot that, you'd get a very wiggly line that seems to show an upward trend.

Then there's the confidence you want to reach. Are you happy with a 90% confidence your strategy has a positive expectancy? Or would you wait for it to reach 99.9% before turning it on full speed?

Plug your numbers into excel the way it was described a few posts earlier; that should get you going.

[xitian]

Mark, did you ever get/find an answer to your optimisation problem? Or is it still a case of "there's no (known) optimal way to optimise"?

Thanks.

[Euler]

I recently did a video on assessing the profitability of a system in a simple way: -

https://www.youtube.com/watch?v=S-i3fSkT9LI

Returns should regress to mean the longer you run a system.

[marksmeets302]

Mark, did you ever get/find an answer to your optimisation problem? Or is it still a case of "there's no (known) optimal way to optimise"?

I have a deal with the university where they will present my exact case to students. This will happen in november.

[xitian]

I have a deal with the university where they will present my exact case to students. This will happen in november.

Oh, yeah, sorry. I got my dates mixed up and thought that was from last year November!

I'm so used to time flying nowadays that I got confused. Just ignore me!

[marksmeets302]

Returns should regress to mean the longer you run a system.

Very nice video; it captures what the "excel method" tries to do. As you gain more samples, the average return divided by turnover should stabilize. That is, it converges to a number, and the wiggles in the line become smaller: the standard deviation gets lower. When you get to a point where the average minus 2 standard deviations is still positive, then you have shown that with 97.5% confidence you can state that your strategy works.

You can also reverse this, and this is maybe a nice solution to a problem addressed in the video: you're in a drawdown and your gut tells you to call it quits. If you do the same calculation and the average plus 2 standard deviations is negative, then yes, with 97.5% certainty this is a bad system. If not, it might just go through one if its natural down cycles and it might be better to hold on.