In-Sample vs Out-of-Sample Testing

If you optimise an EA on a stretch of history and then judge it on that same stretch, you have learned almost nothing — of course the settings fit, you chose them to fit. Out-of-sample testing fixes this by hiding some of the data from the optimiser and keeping it back as an exam. In-sample is where the strategy studies; out-of-sample is where it sits the test blind. The gap between how it does on each is one of the most telling numbers in the whole process.

What they are

Split your history into two parts before you touch the optimiser:

In-sample — the larger chunk you optimise on. The tester is free to find the best settings here.
Out-of-sample — a slice you set aside and never optimise on. Once the settings are locked, you run them on this data once and see what happens.

The in-sample result tells you what the strategy can fit. The out-of-sample result tells you what it can do when it cannot cheat. Only the second one resembles live trading.

Why in-sample always looks good

An optimiser is a machine for fitting the past. Give it enough settings to try and it will find a combination that would have printed money over the in-sample data — including settings that fit the random noise of that exact period, not any real edge. This is curve-fitting, and in-sample results cannot tell it apart from genuine skill. That is the whole reason for holding data back: out-of-sample is the only place where noise-fitting is exposed, because the noise there is different. The more settings an EA hands to the optimiser, the easier this is to do by accident, and the more out-of-sample data you need to catch it.

Reading the gap

Compare the two results side by side. A strategy with a real edge holds up reasonably well out-of-sample; a curve-fitted one falls off a cliff.

Small gap — in-sample profit factor 1.6, out-of-sample 1.4. Believable, and worth continuing.
Large gap — in-sample 2.4, out-of-sample 0.9. The optimiser fitted the past and the edge was never real. Bin it.

The absolute numbers matter less than the drop. A modest strategy that survives the exam beats a spectacular one that collapses the moment it meets data it has not seen.

How much to hold back

A common split is 70–80% in-sample and 20–30% out-of-sample, with the out-of-sample slice taken from the most recent data — the period closest to what you are about to trade. There is no perfect ratio. The point is to leave a genuine, untouched sample, large enough to contain a mix of conditions and positioned at the end of your data so the test sits as close as possible to the future you actually face. Run that test exactly once. If a poor result tempts you back to the optimiser and you re-run it, the slice has quietly become in-sample too, and the honesty of the whole exercise is gone.

Where it leads

A single in-sample/out-of-sample split is the simplest version of this idea. Walk-forward analysis is the same principle repeated — optimise, test on unseen data, roll forward — so the honesty check happens many times across your history rather than once. If a strategy passes a clean split, walk-forward is the natural next step.

Common mistakes

Peeking — testing out-of-sample, adjusting the EA, re-testing. Now nothing is held back and you have quietly cheated.
Too small a slice — a fortnight cannot cover enough conditions to mean anything.
Ignoring a bad result — deciding the out-of-sample run was "unlucky" and trading anyway. It was the honest number.

The Karnek note

Out-of-sample testing is your best guard against fooling yourself before going live — but the truest out-of-sample data of all is the market you have not traded yet. Karnek watches that for you, reading the EA's real trades read-only from the terminal. It can never place a trade or alter the account, so it simply shows whether the live results match the exam.

See it in Karnek: monitor grid and martingale EAs, with drawdown alerts before they run away.