psheridan050
52 posts
msg #64795
- Ignore psheridan050 |
7/9/2008 12:02:11 AM
I just wanted to open up the debate on the usefulness of backtesting. I’m curious how many people put their faith in strategy backtest results, and how close actual trading compares to the results found within a backtest. Another question for backtesters is what criteria do you use to measure whether a strategy is worth pursuing or dumping; profit factor, max drawdown, etc. Another good question for people who believe in backtesting; to what lengths do you go to prove your strategy is valid? By this I’m asking about the methods you use. Examples: Do you take a 6 month backtest as confirmation your strategy is valid? Does your strategy have to have a PF of 3 or greater in SF’s backtester since 2002 in order to keep using a strategy? Do you run your backtest results through another program such as Excel to verify results? Do you use a battery of statistics tests to ensure the distribution of profits has a fat right tail with an otherwise small standard deviation?
For those who don’t believe backtesting is useful, how and when do you decide to use a particular strategy when trading? Along these same lines, in what way do you determine a strategy is working for you and how long does it take to make such a determination? I assume the only way to find this out is with live trades which can get expensive if it’s a bad strategy.
I guess a debate such as this can be narrowed down to two main ideas:
* Do you believe that the past performance of stocks is indicative of future performance?
* Are you a mostly a mechanical trader or a discretionary trader?
|
cunparis
71 posts
msg #64797
- Ignore cunparis |
7/9/2008 6:30:18 AM
Let's say we backtest with a certain condition (crossover, candlestick pattern, volume, whatever) and we go long for 1 day hold. Quite simple. Let's say backtest says this works 65% of the time with a 1.5 reward/risk ratio.
Does that mean that tomorrow when this condition occurs, there is a 65% chance of the trade working? No. because the conditions for tomorrow are different. There could be an announcement, oil could hit a new high, etc.
But over time, if this pattern didn't work 65% of the time, then the backtest statistics would drop down.
let's say 2002-2004 65% 1.5 ratio.
in 2005 either we get more than 65%, 65%, or less than 65%. Either way, for 2005 the results will show what we observed.
So my opinion is that what happened in the past will continue to happen in the future. This principle is called "yesterday's weather". But one must acknowledge that at any given moment the results could change. so if using a system based on backtest, it must be monitored to see if it has changed.
The more complex the system (indicators!!) the less reliable in my opinion. The more basic, the more it relies on human nature which doesn't change.
For the SF backtests, I find the stop loss very limiting and I can't get any strategies profitable with stop losses. For example I'd like to put my stop below the low of the last 5 days. I can't do that with SF. That kind of ruins it for me. It's fun to test out ideas but it can only go so far.
|