Add an append-only forecast track record#90
Open
vahid-ahmadi wants to merge 1 commit into
Open
Conversation
The site could show a current forecast but not a record of past ones: current_forecast.json is overwritten on every data-edge refresh, so a superseded forecast survived only in git history. That is right for the hero chart and wrong for evidence — a forecast only counts if it provably existed before the outturn did. forecasts/ archives each round immutably and scores it once outturns land: - archive.py snapshots the moving artifact into rounds/<date>/<model>.json, refusing to overwrite an existing round. - score.py scores rounds against versioned outturns on four measures — MAE, signed bias, 68/90% band coverage, and naive baselines — and generates both scorecard.json and the marked blocks of the page, with --check for staleness. - forecast-archive.yml fails any PR that modifies, renames or deletes an archived round. The page states the empty record plainly (1 round, 0 scored, first scorable period 2026Q2) rather than deriving a headline accuracy figure from nothing. It also distinguishes this from the existing pseudo-out-of-sample evaluation, which tests the method but was run knowing the answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The site could show a current forecast but not a record of past ones.
papers/boe-svar/figures/current_forecast.jsonis overwritten on every data-edge refresh (commitb66cadfdid exactly that), so a superseded forecast survived only in git history. That is right for the hero chart and wrong for evidence: a forecast counts only if it provably existed before the outturn did.This is distinct from the pseudo-out-of-sample evaluation already in
papers/boe-svar/figures/rolling_evaluation.json— 49 expanding-window origins against random-walk, drift and AR(1) baselines. That tests the method, but it was run knowing the answer. Nothing tested the forecasts in real time.What
forecasts/archive.py— snapshots the moving artifact intorounds/<date>/<model>.json; refuses to overwrite an existing round. Round2026-07-21(boe-svar, 2026Q1 edge, 13 periods) seeded.forecasts/score.py— scores rounds against outturns on four measures: MAE, signed bias, 68/90% band coverage, and naive baselines. Generatesscorecard.jsonand the marked page blocks, with--check, matching themake_charts.pycontract.forecasts/outturns.json— vintage-versioned. ONS revisions append rather than replace, so a published score stays reproducible instead of rewriting itself on every revision..github/workflows/forecast-archive.yml— fails any PR that modifies, renames or deletes an archived round. Load-bearing: a silent rewrite is the one failure that destroys the claim, and in review it looks like a routine refresh.forecasts/index.html— new page; nav tab added across all 13 section-tab pages; sitemap entry.Honest state
The page reports 1 round archived, 0 periods scored, first scorable period 2026Q2. No headline accuracy figure — one derived from zero observations would be worse than none. The page also states the pseudo-OOS/real-time distinction explicitly rather than letting the existing evaluation be read as a track record.
Verification
archive.py --check,score.py --checkpass; page parses clean; sitemap valid.Note, unrelated to this PR
validation/figures/make_charts.py --checkfails onmain(expects 7.vchartSVGs, page has 10) — confirmed against a clean worktree ata0fd90e, introduced by #89. That provenance gate is currently inoperative. Fixed separately.🤖 Generated with Claude Code