PRD: ES TradingView-to-Nautilus strategy evolution system

[sybapp]

Problem Statement

The project needs a reproducible way to turn TradingView-derived market data, indicator values, and indicator graphics into candidate ES intraday strategies that are finally validated by NautilusTrader. Today, TradingView can provide bars, indicator plots, graphics, drawings, and strategy reports, but those outputs are not yet shaped into versioned datasets, structural features, Strategy Specs, Nautilus validation runs, or a repeatable optimization loop.

The user wants to use this repository as the TradingView Collector while adding a Python-side Nautilus Evaluator in the same repo. The final goal is to discover ES RTH intraday strategies with strong out-of-sample, cost-adjusted Sharpe in NautilusTrader, without treating TradingView strategy reports as final performance truth.

Solution

Build the first version of a Strategy Evolution system around a Versioned Dataset Contract. The JavaScript TradingView Collector exports immutable dataset directories containing ES RTH 5-minute bars, indicator-derived Structural Features, and dataset metadata. The Python Nautilus Evaluator consumes those datasets, converts them into NautilusTrader inputs, compiles constrained Strategy Specs into Nautilus strategies, runs walk-forward validation, applies next-bar execution with costs and slippage, and records every backtest or optimization run in a local Run Registry.

The first implementation should prioritize a narrow, reproducible vertical slice: define the dataset schema, validate fixture data, export ES RTH 5-minute datasets, read them from Python, run a minimal Nautilus backtest from a hand-written Strategy Spec, and only then add template search and Optuna-style optimization.

User Stories

1. As a strategy researcher, I want TradingView-derived ES data exported into immutable dataset directories, so that every experiment can be reproduced.
2. As a strategy researcher, I want each dataset to include a manifest, so that symbol, session, interval, contract type, roll policy, indicator allowlist, collection time, and schema version are explicit.
3. As a strategy researcher, I want ES RTH 5-minute bars exported from TradingView, so that the first validation loop has a concrete intraday market scope.
4. As a strategy researcher, I want 15-minute bars supported as a secondary comparison interval, so that strategy behavior can be compared at a less noisy resolution.
5. As a strategy researcher, I want TradingView continuous futures datasets to record roll metadata, so that convenient research data is not mistaken for directly tradable single-contract data.
6. As a strategy researcher, I want TradingView indicator plots normalized into features, so that they can be used by Strategy Specs and Nautilus validation.
7. As a strategy researcher, I want TradingView graphics normalized into Structural Features, so that labels, lines, boxes, zones, and profiles can be consumed without screenshot-based modeling.
8. As a strategy researcher, I want every feature to include an event time and availability time, so that strategies cannot use information before it was visible.
9. As a strategy researcher, I want repainting-risk features marked explicitly, so that future-confirmed indicators do not leak lookahead information into validation.
10. As a strategy researcher, I want the first indicator universe constrained by an allowlist, so that feature extraction and overfitting risk stay manageable.
11. As a strategy researcher, I want TradingView strategy reports available only as exploratory context, so that final performance is always determined by Nautilus Validation.
12. As a strategy researcher, I want a Python Nautilus Evaluator in this repo, so that Nautilus integration can evolve alongside the dataset contract.
13. As a strategy researcher, I want the evaluator to read datasets without calling TradingView live, so that backtests are not affected by login state, limits, or changed historical data.
14. As a strategy researcher, I want Strategy Specs to be structured and schema-validated, so that strategy search stays reproducible and bounded.
15. As a strategy researcher, I want Strategy Specs compiled into NautilusTrader strategies, so that final validation happens in the target engine.
16. As a strategy researcher, I want first-version strategies to be RTH intraday and flat before close, so that overnight risk, gap risk, margin, and ETH effects are out of scope.
17. As a strategy researcher, I want bar-close signals with next-bar execution, so that bar-level backtests avoid same-bar lookahead fills.
18. As a strategy researcher, I want fixed fees and slippage applied, so that high-Sharpe candidates survive basic trading cost assumptions.
19. As a strategy researcher, I want walk-forward validation, so that candidates are scored on future windows not used for parameter search.
20. As a strategy researcher, I want fitness to be multi-constraint, so that raw Sharpe is not optimized at the expense of trade count, drawdown, costs, or robustness.
21. As a strategy researcher, I want a local Run Registry, so that every high-performing run can be traced back to dataset version, Strategy Spec, optimizer settings, and validation windows.
22. As a strategy researcher, I want deterministic or random template search before advanced evolution, so that a simple baseline exists before optimization becomes complex.
23. As a strategy researcher, I want Optuna-style optimization after the baseline, so that parameter and template search can improve while staying bounded.
24. As a strategy researcher, I want the LLM limited to candidate proposal and analysis, so that it cannot bypass Strategy Spec validation or Nautilus scoring.
25. As a maintainer, I want the dataset contract tested with fixtures, so that JS collector and Python evaluator can evolve independently.
26. As a maintainer, I want schema validation for dataset files, so that invalid bars, missing metadata, or unsafe feature timestamps fail early.
27. As a maintainer, I want tests focused at the dataset contract seam, so that implementation internals can change without breaking behavior-oriented tests.
28. As a maintainer, I want the first implementation split into independent milestones, so that agents can work on dataset schema, collection, feature extraction, evaluator, Strategy Specs, and optimization separately.

Implementation Decisions

• TradingView is the TradingView Data Source and candidate feature source; NautilusTrader is the final validation authority.
• The first loop is a Bar-Level Loop rather than tick or quote validation.
• The initial market scope is ES equity index futures, not crypto or all TradingView asset classes.
• The primary validation interval is 5-minute bars, with 15-minute bars as a secondary comparison interval.
• The primary validation session is ES RTH.
• First-version strategies are Intraday Strategies and must be flat before the RTH close.
• TradingView continuous futures such as ES1! may be used first, but datasets must carry explicit continuous futures and roll policy metadata.
• TradingView graphics are modeled as Structural Features, not screenshots or raw chart images.
• Every feature must include availability time; repainting-risk features must be marked and delayed until confirmed.
• Strategy search operates on Strategy Specs, not arbitrary generated Python code.
• The Strategy Spec includes entry rules, filters, exit rules, sizing, risk controls, and tunable parameter ranges.
• The JavaScript-side collector owns TradingView authentication, chart sessions, indicator sessions, data normalization, and dataset export.
• The Python-side Nautilus Evaluator lives in this repository in an independent package or directory.
• The dataset contract is the seam between collector and evaluator.
• Dataset directories include a Dataset Manifest, bar file, and Feature File.
• The evaluator converts datasets into NautilusTrader bar and custom data inputs.
• Bar-level validation uses bar-close signals and Next-Bar Execution.
• Validation includes fees and slippage from the first version.
• Fitness Score is multi-constraint: out-of-sample Sharpe ranks only after survival checks for trade count, drawdown, costs, slippage, and regime robustness.
• Walk-Forward Validation is required from the first version.
• The optimizer starts with deterministic/random Template Search, then adds Optuna-style optimization.
• The LLM role is Candidate Proposer and analyst only; LLM ideas must become validated Strategy Specs before scoring.
• Every validation or optimization run writes a local Run Registry record.

Testing Decisions

• The highest test seam is the Versioned Dataset Contract between TradingView Collector and Nautilus Evaluator.
• Tests should validate externally visible behavior: dataset shape, schema validity, feature availability, repainting-risk marking, and evaluator readability.
• Tests should not assert TradingView websocket internals or Nautilus implementation internals when a contract-level test can cover the behavior.
• The first test fixture should be a tiny ES RTH 5-minute dataset with a manifest, several bars, and representative features.
• The schema validator should reject missing manifest fields, malformed timestamps, features without availability time, features available before they should be, and invalid repainting-risk metadata.
• JavaScript tests can follow the existing Vitest style already used in the repo.
• Python evaluator tests should be introduced with the evaluator package and should verify dataset reading, basic Nautilus conversion, and a smoke-test backtest over fixture data.
• A good end-to-end smoke test runs a hand-written Strategy Spec against fixture data and writes a Run Registry record.
• Walk-forward tests should verify that training/search windows and scoring windows do not overlap incorrectly.
• Fitness tests should verify that a candidate can be rejected by survival constraints even if its raw Sharpe is high.

Out of Scope

• Live trading.
• Tick-level or quote-level validation in the first version.
• General support for all TradingView markets or all asset classes.
• Crypto-first scope.
• Overnight ES strategies.
• ETH as the primary validation session.
• Treating TradingView strategy reports as final performance truth.
• Screenshot or computer-vision modeling of chart images.
• Unrestricted Python strategy code generation.
• Complex genetic programming in the first version.
• Database-backed experiment tracking in the first version.
• Production deployment or distributed optimization.
• Full single-contract futures roll implementation in the first version, beyond explicit continuous futures roll metadata.

Further Notes

The current roadmap is documented in docs/strategy-evolution-roadmap.md. The domain glossary is documented in CONTEXT.md. Key architectural decisions are captured in ADRs under docs/adr/, especially the decisions to use NautilusTrader for final validation, start with a bar-level loop, model graphics as structural features, evolve Strategy Specs instead of arbitrary code, require walk-forward validation, use versioned datasets, split TradingView collection from Nautilus evaluation, and record every run in a local Run Registry.

The recommended first implementation milestone is the Dataset Contract: define manifest, bars, and features schemas; create fixture data; and write validators/tests before building the full collector or evaluator.