How do you know you have a profitable system?

[xitian]

I can't find it now but I posted a link to a job advert a while ago from Betfair which was basically advertising the fact they trade there own markets with zero commission.

I do vaguely remember you posting that, LinusP. I don't think I found it that suspicious at the time. I just assumed they were hiring traders for their sports book. Betfair have been open in the past about using the exchange for hedging their sports book though, so I guess that could be qualified as "trading their own markets". I don't see why they would hire traders to trade specifically on the exchange though. They have an army of people (PC payers) doing that already who get paid profit share effectively. If you're good at trading, why would you want a job there when you'd probably get paid less based on how much you could win them. Alternatively if their salary would be more than you can earn by trading yourself independently then surely Betfair wouldn't want you as a trader because you wouldn't be good enough. Doesn't entirely make sense to me. The Premium Charge basically takes care of it for them.

Anyway, my point was that these kind of conversations are just a bit of a distraction. It's still currently possible to make money even if Betfair are inspecting our accounts. As I think I've mentioned elsewhere, after 6 years, I'm still slowly scaling up.

I propose we get this thread back to the original topic as it's an interesting and relevant one. Here's where we left off:

I think Euler's video on testing is pretty good. (The guy is making a living off this, so we are not surprised...)

I also agree with the comment that profits of a strategy will often decline over time. So it is best to have multiple **profitable** strategies running. If a strategy loses enough times, you should have a mechanism to stop you trading it further. Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.

I saw someone mention "optimizing" strategy parameters to maximize profit. This is the wrong approach. You should really be measuring the parameter sensitivity of your profitability. If profits change a lot when a parameter value changes, you are in trouble! (i.e you will be overfitting by optimizing the parameter)...

Good luck

Thanks lonestar. Is there a way to quantify this?

[steven1976]

Maybe they could trade their markets without inplay delays and mop up before the suspend in some sports when they see bets arriving which can't be cancelled. These type of things PC payers probably can't do and it would be free money for bf.

[ShaunWhite]

Maybe they could trade their markets without inplay delays and mop up before the suspend in some sports when they see bets arriving which can't be cancelled. These type of things PC payers probably can't do and it would be free money for bf.

I can never understand why people always think BF have got their hand in the till. Last year approx £18bn in matched bets, £9bn in wins, even at 5% that's £450m in commission. Why would they risk their licence for an inside operation that's only going to earn a miserable 4 or 5 million at best, less than 1% on turnover!? Would you take 1.01 on them not being caught? Do you think they would?

It's the same with banking, people just don't understand the scale. Fund managers are criticised for £10m bonuses but as a percentage, that's less commission than a guy gets for selling double glazing.

[Bluesky]

I can never understand why people always think BF have got their hand in the till. Last year approx £18bn in matched bets, £9bn in wins, even at 5% that's £450m in commission. Why would they risk their licence for an inside operation that's only going to earn a miserable 4 or 5 million at best, less than 1% on turnover!? Would you take 1.01 on them not being caught? Do you think they would?

I sort of agree with you on this but then sometimes greed takes over from logic. There have been numerous examples of when people have been making a fortune but then do something a bit dodgy just to make a little bit more and get caught.

A recent but perhaps not a very good example would be Sam Alladyce (although I don't think he did anything illegal) but it still cost him his England job.

Also it is often the case that the people at the very top of an organisation don't actually understand exactly how all the profits are made. Or at least they ensure there is no paper trail showing that they actually knew what was going. A good example of this would be Nick Leeson and Bearings Bank.

The directors of that Bank were very happy with the profits Leeson was producing at one point, but did not really understand the derivatives market, but as long as the money kept rolling in they weren't too bothered.

I am with you in that I doubt BF is doing something that if found out would cost them their gold mine, but if it did happen I would not be totally shocked.

The directors of BF would I am sure do what most directors do in this situation, explain that they were completely unaware that shady practises were being undertaken by people lower down in the organisation.

[spreadbetting]

Why would they risk their licence for an inside operation that's only going to earn a miserable 4 or 5 million at best, less than 1% on turnover!? Would you take 1.01 on them not being caught? Do you think they would?

Why would they lose their licence? They've continually said they've been active in markets even as far as saying they seeded markets in the early days and x matching is no more than seeding their markets with pretty much no risk.

I'd imagine with all the info available to them they don't need to do anything particularily dodgy to make money from trading their markets. Plus there are players out there already making a miserable £4-5M on Betfair so you'd have to be very incompetent not to match that with all the advantages and data Betfair would have available to their traders.

[ShaunWhite]

There's a lot of truth in what you say Bluesky....I'm just trying not to be a cynical old sod, but it's a hard habit to break.

Don't talk to me about dodgy Sam, I'm a Palace supporter. I wouldn't have hired him to sweep the car park.. I don't care how great he used to be.

[Sovereign]

I think the following should do the trick: Open excel or libreoffice and put your profits and losses in column A. On B1 put 0. This is the amount of money you want to prove you can (on average) at least make. Now go to data->statistics->t-test. For 'variable 1 range' enter column A. Set 'variable 2 range' to a single cell: B1. On 'results to' enter C1 and hit OK. Around C1 you will see amongst others see 'alpha', 't-stat' and 'P(T<=t) one-tail'. If alpha is 0.05, and t-stat is lower than P(T<=t), it means with 95% certainty that you will in the long run make more than 0. If not, either you don't have an edge or you need more data to prove that you do.

You can play around with alpha and B1 to answer questions like 'am I 99% sure my average PnL will be more than 10 pounds'?

(if there are any statistics gurus that want to comment, please do - this is not my area of expertise but I'd love to learn more)

PeterLe, wouldn't it be better to work out the standard deviation of the results of the 189 races you had, and use that to give a prediction of your year-end result?

For instance, if your average pnl is 2, and the standard deviation 10, then after 9000 races with 95% certainty you would have made between
(2 pounds)*9000 - 2*sqrt(9000)*(10 pounds)
and
(2 pounds)*9000 + 2*sqrt(9000)*(10 pounds).

This works out to be 16102 and 19897.

I apologise for bumping this thread!

I've found the information here very fascinating, and I've been trying to apply it to my own set of results. I have an automation strategy that I'm testing that so far has been profitable. I am by no means a statistician (and my maths is a little rusty, so I apologise if my questioning seems rather obvious) but I've ran my results so far through a t-test and then tried to replicate through a monte carlo simulation of so many thousand results, and I either seem to definitively prove it's a success or show it's a failure, as well as somewhere inbetween!

The strategy has been running for 210 races currently, where my average pnl across the set is 0.28, and profit after commission of £59.44. My 't-stat' number is lower than 'P(T<=t) one-tail' (0.0617 vs 0.4753) (which, going by what marksmeets302 stated, is good? ) Additionally, applying the formula that marksmeets302 mentions in the second post, with standard deviation of 4.5941 (which I derived from my list of profits and losses), I get between £2397 and £2642 applied over 9000 races (for reference, =0.28*9000-0.28*SQRT(9000)*(4.594) and the same with a plus rather than a minus - I'm hoping I've applied those formulae correctly). As I was curious to see how it compared, I also applied my average profit and loss amounts to historical betfair data from the last couple of years, and the yearly figures I get seem to be similar to the figures of approx 2300-2600 mentioned here (perhaps a little less, but I believe my average is a little higher currently than the historical data). Statistically, it seems I would have a losing percentage of around 15-16% over a year (currently, based on this set of results, I am on 14.75%). Additionally, my profit over turnover figure is 0.1375.

As you can see, lots of 'it seems' and 'I think', I am very unsure of myself here and entering some muddy waters!

Making it worse, I try and use my average loss, average profit and my winning percentage on a monte carlo simulation over the same amount of races (9000) and get wildly different figures, from +1215 to -18, which seems lower than estimated and doesn't seem to tally to where I would be going by historical data or from my own (albeit much smaller) data set, but I'm not sure why?

Can anyone offer any advice on how I should proceed? I'm going to run my strategy over the next month to get a fuller set of data, but I'd like to know how I can best analyse what I see.

Thanks for any advice!

[Sovereign]

I think the following should do the trick: Open excel or libreoffice and put your profits and losses in column A. On B1 put 0. This is the amount of money you want to prove you can (on average) at least make. Now go to data->statistics->t-test. For 'variable 1 range' enter column A. Set 'variable 2 range' to a single cell: B1. On 'results to' enter C1 and hit OK. Around C1 you will see amongst others see 'alpha', 't-stat' and 'P(T<=t) one-tail'. If alpha is 0.05, and t-stat is lower than P(T<=t), it means with 95% certainty that you will in the long run make more than 0. If not, either you don't have an edge or you need more data to prove that you do.

You can play around with alpha and B1 to answer questions like 'am I 99% sure my average PnL will be more than 10 pounds'?

(if there are any statistics gurus that want to comment, please do - this is not my area of expertise but I'd love to learn more)

PeterLe, wouldn't it be better to work out the standard deviation of the results of the 189 races you had, and use that to give a prediction of your year-end result?

For instance, if your average pnl is 2, and the standard deviation 10, then after 9000 races with 95% certainty you would have made between
(2 pounds)*9000 - 2*sqrt(9000)*(10 pounds)
and
(2 pounds)*9000 + 2*sqrt(9000)*(10 pounds).

This works out to be 16102 and 19897.

I apologise for bumping this thread!

I've found the information here very fascinating, and I've been trying to apply it to my own set of results. I have an automation strategy that I'm testing that so far has been profitable. I am by no means a statistician (and my maths is a little rusty, so I apologise if my questioning seems rather obvious) but I've ran my results so far through a t-test and then tried to replicate through a monte carlo simulation of so many thousand results, and I either seem to definitively prove it's a success or show it's a failure, as well as somewhere inbetween!

The strategy has been running for 210 races currently, where my average pnl across the set is 0.28, and profit after commission of £59.44. My 't-stat' number is lower than 'P(T<=t) one-tail' (0.0617 vs 0.4753) (which, going by what marksmeets302 stated, is good? ) Additionally, applying the formula that marksmeets302 mentions in the second post, with standard deviation of 4.5941 (which I derived from my list of profits and losses), I get between £2397 and £2642 applied over 9000 races (for reference, =0.28*9000-0.28*SQRT(9000)*(4.594) and the same with a plus rather than a minus - I'm hoping I've applied those formulae correctly). As I was curious to see how it compared, I also applied my average profit and loss amounts to historical betfair data from the last couple of years, and the yearly figures I get seem to be similar to the figures of approx 2300-2600 mentioned here (perhaps a little less, but I believe my average is a little higher currently than the historical data). Statistically, it seems I would have a losing percentage of around 15-16% over a year (currently, based on this set of results, I am on 14.75%). Additionally, my profit over turnover figure is 0.1375.

As you can see, lots of 'it seems' and 'I think', I am very unsure of myself here and entering some muddy waters!

Making it worse, I try and use my average loss, average profit and my winning percentage on a monte carlo simulation over the same amount of races (9000) and get wildly different figures, from +1215 to -18, which seems lower than estimated and doesn't seem to tally to where I would be going by historical data or from my own (albeit much smaller) data set, but I'm not sure why?

Can anyone offer any advice on how I should proceed? I'm going to run my strategy over the next month to get a fuller set of data, but I'd like to know how I can best analyse what I see.

Thanks for any advice!

I thought I'd just add to this after a further day of automated trading.

So I had a pretty good day today (overall just over £30 profit) and the variation in my figures from today compared to yesterday seems pretty big (so my losing percentage has gone down to 13.59%, average pnl up to 0.41 etc) so I've decided that I just don't have enough data to confirm whether what I'm seeing is significant just yet, and really, I need to come back to this at the end of the month or so, when I have 1000+ results behind me and I can analyse the data then. And indeed, using the figures as they stand after today, a monte carlo simulation ran several times all generated reasonable profits, whereas yesterday it was much wilder (strong profits to the odd strong overall loss, etc).

So, optimistic and pleased so far, but still early days!

[marksmeets302]

I find it a bit strange that your average pnl can go up from 0.28 to 0.41 after just one extra day of racing, since you started with 210 race results. Are you sure of this?

[xitian]

I think the sample is just waaay to small for now. I mean the standard deviation is over 4, with average less than 0.5 if I'm understanding correctly. That's presumably how one day of races has shifted the average so much? So you have 58.8 total profit, and yet there's a 5% chance of a race producing +£9 or -£9. So yesterday you had a lucky one with a few big wins and ended with +£30 odd?

What percentage of races are matching your trading criteria?

[Sovereign]

marksmeets302 - I double checked my average pnl for that original set, and it was actually 0.31 - I dunno how I made that error, apologies (EDIT: I actually determined it was 0.28 up to that set of 210 races, but the full pnl up until the end of that day was 0.31), It certainly seemed quite a jump, so my reckoning is that I just don't have enough data to confirm either way, the fluctuations on a day to day basis are too large. It's now at 0.35 as today progresses.

xitian - I think what you're saying is right - I definitely don't have enough data, so I'm going to wait out the month and see where I end up, taking snapshots of the average pnl figures, my stdev etc as the month progresses to see if the fluctuations start to diminish. A picture, perhaps, paints a thousand words:




As for percentage of races meeting the criteria, roughly 75%.

[marksmeets302]

It's a bit surprising the t-test says you have a profitable strategy; from the graph alone I would also say you need more data. But hey, congrats! I'd be a bit conservative at the moment and not take out a second mortgage to leverage your system, but things are looking good. Get some more data, get acquainted with the system and try to improve - you could be on to something!

[foxwood]

For instance, if your average pnl is 2, and the standard deviation 10, then after 9000 races with 95% certainty you would have made between
(2 pounds)*9000 - 2*sqrt(9000)*(10 pounds)
and
(2 pounds)*9000 + 2*sqrt(9000)*(10 pounds).

Been looking closely at this to evaluate a promising but volatile strategy and it struck me that this would only work for a "normal distribution" of the pnl.

For a profitable strategy however, I think the pnl distribution will always (?) be skewed positively. This would apply to trading and even more so for level stakes betting ie there would be a large number of small losses with smaller numbers of varying profitable amounts.

So, I wonder if it is valid to use the standard deviation / average eg this article talks about using logs or regression to eliminate skewness - kinda fries my brain trying to work that lot out ! https://www.ma.utexas.edu/users/mks/sta ... tions.html

PeterLe's monte carlo analysis looks interesting - might try and replicate that and see if it gives anything like a similar answer which might help.

[marksmeets302]

foxwood, I usually remove the outliers which are a result of problems with betfair, internet connection etc. After that, I'm often left with something that is normally distributed. Of course it will have a positive or negative mean depending on the profitability of the strategy, but a skew is rare in my opinion. Some years ago I spend some time on how to check if a distribution is normal. It seems you can use something called the Jarque-Bera test for this. If you google for it you'll find tools that perform this test for you.

[gazuty]

Doh, this excellent thread is racing along while I'm on holidays.