Portfolio & tool shed

Selected project / Personal research

Trading Research Lab

A simulated experiment about strategy selection, holdout testing, and the quality of evidence behind a backtest.

Cole Hover · AI-assisted development & review · Public showcase updated November 8, 2026

The question

How much of a good backtest is the search itself?

Trying many strategies creates many chances to find an impressive result. This project makes that selection process visible: generate 1,000 simulated rules, choose one using training data only, and examine it on a separate holdout.

What was built

A reproducible experiment, an interactive report, and a review workflow covering data quality, trial counts, costs, and evaluation boundaries. The report includes charts and the complete sortable trial table, so the selected result can be inspected in context.

What this demonstrates

Experiment design, separation of selection and evaluation, and the discipline to keep assumptions beside results. The selected rule retains a positive holdout result in this demonstration; the run is preserved rather than rerun to create a more dramatic story.

What it does not establish

These are simulated returns, not account performance. A positive holdout and a statistical diagnostic do not establish a profitable live strategy. The diagnostic depends on assumptions about the trials and returns, including an independent-trial assumption.

A reusable research pattern

Freeze. Test. Inspect. Reconsider.

  1. Define the experimentWrite the question and trial budget before searching.
  2. Separate the dataChoose on training observations; reserve the holdout.
  3. Keep every trialShow the broader search, not only the winner.
  4. Report the limitsExplain assumptions, uncertainty, and what would change the conclusion.

A separate validation review also exposed a data-resolution bug: higher-timeframe context was overwriting execution data. The corrective lesson was to separate the two data streams before interpreting performance.

Take the process with you.

The reusable brief and review checklist live in the same library as the models and trackers.

Educational research only. No investment advice, betting recommendations, or promised outcomes. Disclosures.