Add a daily cache-warming batch job (GCP) — refresh all cached data + build options chain history

[JohnFunkCode]

What

Stand up a scheduled batch job in GCP that refreshes all of the project's cached data once per trading day — at market open or close. It both improves runtime performance (interactive MCP/REST/UI reads hit warm cache instead of live yfinance/Polygon calls) and serves as the mechanism that accumulates options-chain history over time.

Why

• Performance: today many MCP tools and REST endpoints fetch from yfinance/Polygon on-demand. A pre-warmed cache makes the deployed UI and AI tools fast and resilient to flaky upstream responses (cf. the get_trade_recommendation timeout, issue get_trade_recommendation MCP tool times out (cumulative latency, not a single broken dependency) #65).
• History accumulation: several datasets only gain historical depth if something snapshots them daily — options chains, gamma wall, P/C ratio, microstructure signals. A daily job is the natural builder for that history.

Deployment pattern (reuse what exists)

The main.py report already runs as a daily Cloud Run Job on Cloud Scheduler in both projects. Model this new cache-warming job the same way: a containerized entry point (new Dockerfile.* or a subcommand), deployed as a Cloud Run Job, triggered by Cloud Scheduler, wired through the same prod/test CI/CD (deploy.yml / prod-rollout.yml). It should run in-process against the services layer (get_services()), never over HTTP.

Data to refresh (per watchlist + portfolio symbol)

• OHLCV bars (ohlcv table) — daily (and any intraday intervals in use)
• Options snapshots + full chain (options_snapshots/options_expirations/options_contracts) — this is the primary new-history source
• Gamma wall history (gamma_wall_history) — daily MM hedge-bias snapshot
• P/C ratio history — daily aggregate (already has a Polygon backfill endpoint to reconcile)
• News + FinBERT sentiment (news_articles/sentiment_snapshots) via collect_news
• Fundamentals cache (fundamentals_history) — TTL-based refresh
• (Stretch) Microstructure signals (short interest / dark pool / bid-ask) — these are currently computed live and never persisted; a daily snapshot table would give them historical trend data (see capabilities-matrix "Short Interest, Dark Pool, Bid/Ask Data Is Never Persisted")

Scheduling considerations

• Open vs. close: decide per dataset. End-of-day (after close) gives final/settled values and is the natural choice for building clean daily history; a pre-open run can warm cache for the trading session. Could be two schedules.
• Trigger only on US market trading days (skip weekends/holidays) — Scheduler cron + a trading-calendar guard in the job.
• Idempotent / upsert semantics (the stores already use ON CONFLICT upserts) so a re-run is safe.

Open questions

• One combined job vs. per-domain jobs (price / options / news / fundamentals)?
• Symbol universe = watchlist ∪ portfolio (all owners)? Or a broader tracked-symbol set?
• Run in prod (system of record), test, or both? Likely prod for real history, test for CI smoke.
• Failure handling / alerting (Discord notification on job failure?).

Acceptance

• A Cloud Run Job + Cloud Scheduler trigger that refreshes the datasets above on trading days
• Runs in-process via the services layer; idempotent upserts
• Verifiable growth of options-chain / gamma-wall / P-C history day over day
• Deployed through existing CI/CD to test, then prod
• Update docs/capabilities-matrix.md batch-job classification (see issue Docs: refresh capabilities-matrix.md for post-refactor architecture (UI gaps + batch-job classification) #67)

🤖 Generated with Claude Code