Every pipeline — what it produces, where the data comes from, when it last ran, and how to refresh it. Refresh buttons only work when running locally.
| Trigger | Refreshes | Sydney / AEST * | New York / EST * |
|---|---|---|---|
| Premarket (US bot input) | Full refresh (all AI + scores)(L1 → L2 → L3 → L4) | 6:40 PM | 4:40 AM🔒 |
| Market open (US bot input) | Yahoo-only prices(L2 + L4) | 11:30 PM | 9:30 AM🔒 |
| Midday (US bot input) | Full refresh (all AI + scores)(L1 → L2 → L3 → L4) | 2:40 AM | 12:40 PM🔒 |
| Sydney morning review | Yahoo-only prices(L2 + L4) | 10:30 AM🔒 | 8:30 PM |
| Sydney afternoon review | Yahoo-only prices(L2 + L4) | 4:15 PM🔒 | 2:15 AM |
Weekdays only · no weekend runs · manual trigger via GitHub Actions workflow_dispatch
🔒 marks the timezone each job is anchored to — that column stays fixed year-round; the other column's time shifts by an hour whenever DST kicks in on either side.
* Times adjust automatically for daylight saving. View current timezones here
Triggers are dispatched externally by cron-job.org, which calls the GitHub workflow_dispatch API. Worst-case delay is typically under 1 minute.
Refreshes prices, returns, momentum/dip scores, RSI, and valuation metrics from Yahoo Finance for both GICS ETFs and personal holdings. Skips all AI costs — no Finnhub, no Google News, no Claude calls. Existing AI movement summaries are preserved.
.venv/bin/python -m pipeline.refresh_yahooScores all 25 GICS industry groups on their 6–12 month fundamental outlook (0–100). C1 Analyst Consensus (40%): yfinance analyst recommendations + EPS revisions for the top 5 constituents — also stores buy% and EPS revision%. C2 Structural Tailwinds (35%): one Claude Sonnet call scores all groups simultaneously using 8 universal FRED macro series + Finnhub macro news — also stores tailwinds, headwinds, and cyclical/structural flag. C3 Expert Sentiment (25%): Claude Haiku reads Finnhub news headlines for each group over the last 14 days.
.venv/bin/python -m pipeline.layer1.refreshmarket_contextindustry_group_scores — 25 rowsmarket_context — 1 rowCross-asset daily snapshot. CNN Fear & Greed score drives the speedometer. Six expandable tiles cover US equity, global equity, rates, commodities, FX, and crypto — 24 instruments total via yfinance (batch download). Claude Haiku generates a 2–3 sentence AI narrative. All data upserted to the market_pulse table as a single row (id=1).
.venv/bin/python -m pipeline.layer2.market_pulseDownloads 1 year of daily close prices for the primary ETF of each of the 25 GICS industry groups via yfinance. Computes four return windows (1D, 5D, 1M, 3M), 52-week high/low, % from 52w high, 21-day annualised volatility, RSI (14-day), forward P/E, and PEG ratio. Blends returns into a weighted momentum score — 1D×15% + 5D×20% + 1M×40% + 3M×25% — scaled 0–100. Also generates a 1–2 sentence AI movement summary for each group: Finnhub headlines for the primary ETF plus top 3 GICS constituents are fetched first; if no headlines are found, Google News RSS is queried as a fallback.
.venv/bin/python -m pipeline.layer2.momentummarket_pulse — 1 rowmomentum_scores — 25 rowsopportunity_scores — Dip Score column — refreshed by ③ Opportunity Signal belowCombines Layer 1 macro scores with Layer 2 price returns to compute an Opportunity Score for each of the 25 GICS industry groups. The Dip Signal inverts recent price momentum (bigger pullback = higher signal). Final score = Macro×50% + Dip×50%. A Panic Bonus (+15 pts) fires when a group drops ≥5% in a single day and its macro score is ≥60. Claude Sonnet generates a 3-sentence rationale for the top 5 and bottom 5 groups. Also feeds the Dip Score column on ② What's Moving.
.venv/bin/python -m pipeline.layer3.opportunityAI-generated sector verdicts, chained across four Layer-3 jobs. (1) broad_polymarket_events — Haiku generates macro search terms per run and discovers ~15 high-volume Polymarket events. (2) themes_scenarios — Sonnet identifies 2-3 dominant themes + scenario maps, now with Polymarket probabilities, enriched sector data, and cross-asset context. (3) prediction_signals — per-theme dimension search on Polymarket. (4) verdicts — Sonnet emits buy/caution/avoid buckets citing Polymarket probabilities, opportunity_score, and movement_summary context, and may patch priced_in assessments.
.venv/bin/python -m pipeline.layer3.{broad_polymarket_events,themes_scenarios,prediction_signals,verdicts}opportunity_scores — 25 rowsmacro_scenarios — 1 rowDownloads up to 6 months of daily close prices for every ticker in the watchlist via yfinance. Computes four return windows (1D, 5D, 1M, 3M) and scores each ticker using the same weighted momentum formula as Layer 2 (1D×15% + 5D×20% + 1M×40% + 3M×25%) and the same dip signal formula as Layer 3 (1D×20% + 5D×50% + 1M×30%). Both formulas are sourced from the shared pipeline.utils.scoring module. US and ASX tickers are supported. Also generates a 1–2 sentence AI movement summary per ticker: US tickers use Finnhub headlines, ASX tickers use yfinance news — if either returns nothing, Google News RSS is queried as a fallback using the company name (ASX queries append 'ASX' for disambiguation). All headlines are passed to Claude Haiku in a single batched call.
.venv/bin/python -m pipeline.layer4.ticker_momentumRe-generates movement_summary for all watchlist tickers without re-downloading price history. Reads existing scores from the DB, fetches fresh headlines per ticker (Finnhub → yfinance → Google News RSS fallback), calls Claude Haiku, and updates only the movement_summary column. Run this when you want to refresh AI summaries without running the full price/score pipeline (which takes much longer due to yfinance downloads).
.venv/bin/python -m pipeline.layer4.refresh_summarieswatchlist_tickers — 137 rowsticker_scores — 137 rows