How do you know you have a profitable system?

[xitian]

Thanks for sharing the students' report, Mark. I finally got round to reading it.

It wasn't quite the question I was expecting to be answered either (and not the method as you already mentioned). It was quite an interesting approach though.

But I was more expecting you to ask them how to safely select the best parameters to a trading model. And/or compensate a significant test based on the fact you had backfitted the best input parameters. (Surely the answer isn't that you don't have to?!)

Right now I'm looking at the following question: I have a system that works, and I can make it even better by tweaking one of the variables. By testing it on historical data I can pick the best value. I know that this can lead to overfitting, so if I do a test for significance again I have to compensate for the fact that I already picked the best variation. From the internet I've already found 3 approaches on how to do that and they all give different results...

Maybe that question is just impossible to know the answer to since you can never know how much you have overfitted. Perhaps the age old method of using out-of-sample data for cross validation is the way to go. But even that is fitting to data to an extent.

I'm actually warming to method the students used now actually. I can see why they did it. It's just very dependent on your timeseries model and relying on what's happened in the past (which is obviously a big problem). And not very helpful for our sports scenarios as you already mentioned.

[marksmeets302]

After describing the problem to the students they called me the next day, pointing out that I was asking a weird question (although they were very polite, that's what I thought at that moment ). Basically I was asking for confidence intervals on something that wasn't a stochastic process - I had just one fixed set of data. So, they said, we can tell you that with 100% probability you had a sharpe ratio of x in the past. I know, I said, but what about the future? That's how we got from the original problem to the one they solved.

I'm still a bit uneasy about how they generated the new time series, because now the problem shifts to "how confident can we be that those new time series describe new possible futures". In a sense the whole process left me more confused than I was before.

I guess the difference with testing whether a cure is effective is that the way people react to the cure shouldn't be dependent on time. The way a strategy reacts to the market is something that changes, because the market changes over time. That's why you can do a statistical test on drugs, but it gets a little trickier in "our" case. Not sure if this is a good analogy, but it makes sense to me.

[PeterLe]

Gents,
I was just thinking..
A new member joined recently (lonestar) who has a Phd in maths/Stats. Maybe he would provide you with a view?
Regards
Peter

[marksmeets302]

I send him a PM, hope he takes the bait

[lonestar]

I think Euler's video on testing is pretty good. (The guy is making a living off this, so we are not surprised...)

I also agree with the comment that profits of a strategy will often decline over time. So it is best to have multiple **profitable** strategies running. If a strategy loses enough times, you should have a mechanism to stop you trading it further. Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.

I saw someone mention "optimizing" strategy parameters to maximize profit. This is the wrong approach. You should really be measuring the parameter sensitivity of your profitability. If profits change a lot when a parameter value changes, you are in trouble! (i.e you will be overfitting by optimizing the parameter)...

Good luck

[marksmeets302]

Thanks lonestar. Is there a way to quantify this?

[Aqverius]

Is this really true:
Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.
Every effort to make a profitabel bot seems to bee a joke then,when you know that Betfair i Pissing on you on The other side:-)Every bot Dallas show seems to bee a ongoing joke to hold your hope up:-)

Regards LH

[xitian]

Is this really true:
Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.
Every effort to make a profitable bot seems to bee a joke then,when you know that Betfair i Pissing on you on The other side:-)Every bot Dallas show seems to bee a ongoing joke to hold your hope up:-)

There's really no evidence that Betfair are doing this, nor evidence that they are trading their own markets, nor evidence that they are manipulating it in any way (other than cross-matching of course). That's not to say that they couldn't if they wanted to. But as yet, there's no clear evidence that they are. We're better off putting those conclusions aside and working hard on improving our skills than succumb to paranoia and decide that not trying anymore is a better solution.

What people claim to be Betfair "manipulating things against you" are probably just (new) market factors that people don't yet understand and want to find an obvious scapegoat for. What's more likely is that someone else (potentially/probably with automation) is just competing against you.

Now of course none of the above says anything about Betfair's attitude to other market participants manipulating the market. Nor the potential for Betfair to make technical changes which might affect the exchange ecosystem in serious ways (e.g. API changes or fee changes, etc..). Those changes may also be based on market participants' successes (Premium Charges and cross-matching being the obvious examples).

[ruthlessimon]

Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.

i think this does go on & is a reason for traders to build dynamic strategies based on market principles, rather than fixed signals. if it doesn't go on betfair have missed a trick!

some guy became a millionaire by fading consistently losing traders - the logic works both ways!

[LinusP]

Is this really true:
Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.
Every effort to make a profitable bot seems to bee a joke then,when you know that Betfair i Pissing on you on The other side:-)Every bot Dallas show seems to bee a ongoing joke to hold your hope up:-)

There's really no evidence that Betfair are doing this, nor evidence that they are trading their own markets, nor evidence that they are manipulating it in any way (other than cross-matching of course). That's not to say that they couldn't if they wanted to. But as yet, there's no clear evidence that they are. We're better off putting those conclusions aside and working hard on improving our skills than succumb to paranoia and decide that not trying anymore is a better solution.

What people claim to be Betfair "manipulating things against you" are probably just (new) market factors that people don't yet understand and want to find an obvious scapegoat for. What's more likely is that someone else (potentially/probably with automation) is just competing against you.

Now of course none of the above says anything about Betfair's attitude to other market participants manipulating the market. Nor the potential for Betfair to make technical changes which might affect the exchange ecosystem in serious ways (e.g. API changes or fee changes, etc..). Those changes may also be based on market participants' successes (Premium Charges and cross-matching being the obvious examples).

I can't find it now but I posted a link to a job advert a while ago from Betfair which was basically advertising the fact they trade there own markets with zero commission.

[xitian]

I can't find it now but I posted a link to a job advert a while ago from Betfair which was basically advertising the fact they trade there own markets with zero commission.

I do vaguely remember you posting that, LinusP. I don't think I found it that suspicious at the time. I just assumed they were hiring traders for their sports book. Betfair have been open in the past about using the exchange for hedging their sports book though, so I guess that could be qualified as "trading their own markets". I don't see why they would hire traders to trade specifically on the exchange though. They have an army of people (PC payers) doing that already who get paid profit share effectively. If you're good at trading, why would you want a job there when you'd probably get paid less based on how much you could win them. Alternatively if their salary would be more than you can earn by trading yourself independently then surely Betfair wouldn't want you as a trader because you wouldn't be good enough. Doesn't entirely make sense to me. The Premium Charge basically takes care of it for them.

Anyway, my point was that these kind of conversations are just a bit of a distraction. It's still currently possible to make money even if Betfair are inspecting our accounts. As I think I've mentioned elsewhere, after 6 years, I'm still slowly scaling up.

I propose we get this thread back to the original topic as it's an interesting and relevant one. Here's where we left off:

I think Euler's video on testing is pretty good. (The guy is making a living off this, so we are not surprised...)

I also agree with the comment that profits of a strategy will often decline over time. So it is best to have multiple **profitable** strategies running. If a strategy loses enough times, you should have a mechanism to stop you trading it further. Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.

I saw someone mention "optimizing" strategy parameters to maximize profit. This is the wrong approach. You should really be measuring the parameter sensitivity of your profitability. If profits change a lot when a parameter value changes, you are in trouble! (i.e you will be overfitting by optimizing the parameter)...

Good luck

Thanks lonestar. Is there a way to quantify this?

[steven1976]

Maybe they could trade their markets without inplay delays and mop up before the suspend in some sports when they see bets arriving which can't be cancelled. These type of things PC payers probably can't do and it would be free money for bf.

[ShaunWhite]

Maybe they could trade their markets without inplay delays and mop up before the suspend in some sports when they see bets arriving which can't be cancelled. These type of things PC payers probably can't do and it would be free money for bf.

I can never understand why people always think BF have got their hand in the till. Last year approx £18bn in matched bets, £9bn in wins, even at 5% that's £450m in commission. Why would they risk their licence for an inside operation that's only going to earn a miserable 4 or 5 million at best, less than 1% on turnover!? Would you take 1.01 on them not being caught? Do you think they would?

It's the same with banking, people just don't understand the scale. Fund managers are criticised for £10m bonuses but as a percentage, that's less commission than a guy gets for selling double glazing.

[Bluesky]

I can never understand why people always think BF have got their hand in the till. Last year approx £18bn in matched bets, £9bn in wins, even at 5% that's £450m in commission. Why would they risk their licence for an inside operation that's only going to earn a miserable 4 or 5 million at best, less than 1% on turnover!? Would you take 1.01 on them not being caught? Do you think they would?

I sort of agree with you on this but then sometimes greed takes over from logic. There have been numerous examples of when people have been making a fortune but then do something a bit dodgy just to make a little bit more and get caught.

A recent but perhaps not a very good example would be Sam Alladyce (although I don't think he did anything illegal) but it still cost him his England job.

Also it is often the case that the people at the very top of an organisation don't actually understand exactly how all the profits are made. Or at least they ensure there is no paper trail showing that they actually knew what was going. A good example of this would be Nick Leeson and Bearings Bank.

The directors of that Bank were very happy with the profits Leeson was producing at one point, but did not really understand the derivatives market, but as long as the money kept rolling in they weren't too bothered.

I am with you in that I doubt BF is doing something that if found out would cost them their gold mine, but if it did happen I would not be totally shocked.

The directors of BF would I am sure do what most directors do in this situation, explain that they were completely unaware that shady practises were being undertaken by people lower down in the organisation.

[spreadbetting]

Why would they risk their licence for an inside operation that's only going to earn a miserable 4 or 5 million at best, less than 1% on turnover!? Would you take 1.01 on them not being caught? Do you think they would?

Why would they lose their licence? They've continually said they've been active in markets even as far as saying they seeded markets in the early days and x matching is no more than seeding their markets with pretty much no risk.

I'd imagine with all the info available to them they don't need to do anything particularily dodgy to make money from trading their markets. Plus there are players out there already making a miserable £4-5M on Betfair so you'd have to be very incompetent not to match that with all the advantages and data Betfair would have available to their traders.