fix(validate,coverage): close the silent-empty-load window (#808) - #818
Conversation
📐 Rivet artifact delta
Graphgraph LR
REQ_294["REQ-294"]:::modified
classDef added fill:#d4edda,stroke:#28a745,color:#155724
classDef removed fill:#f8d7da,stroke:#dc3545,color:#721c24
classDef modified fill:#fff3cd,stroke:#ffc107,color:#856404
classDef overflow fill:#e2e3e5,stroke:#6c757d,color:#495057,stroke-dasharray: 3 3
Modified
Posted by |
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 1455538 | Previous: f510645 | Ratio |
|---|---|---|---|
link_graph_build/10000 |
42810511 ns/iter (± 2119683) |
26307444 ns/iter (± 2307946) |
1.63 |
This comment was automatically generated by workflow using github-action-benchmark.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Review: verified, but blocking on Fix 3Verified all four acceptance points empirically against the original #808 repro — they genuinely hold:
Blocking — Fix 3 breaks sigil
sigil's With this change sigil's config fails to load at all — rivet becomes unusable in that repo. sigil is our most active downstream (1242 artifacts), and I told them today to upgrade for the #778 fix. Same failure direction as #808, pointed the other way: silently accepts nothing → loudly rejects what worked yesterday. Fix — make Fix 3 consistent with Fixes 1 and 2Fixes 1 and 2 chose warn by default, error under Hard-fail is a fine end state, but it must be sequenced: warn now → fix sigil's config → escalate later with a BREAKING CHANGELOG note. Second finding — no CHANGELOG entryNo Taking both on myself now. |
|
Acknowledged on both — the sigil For reference in case it's useful, the mechanical change from hard-fail to warn-with-diagnostic looks like:
Happy to push it if you'd rather I do — just say the word. Otherwise I'll leave this to you and the CHANGELOG together. Generated by Claude Code |
Correction to my own suggestion — a naive downgrade reopens #808 case 2I proposed simply swapping Fix 3's hard-fail for a warning. I tested that before recommending it further, and it does not work. Probe: removed Silently green again — even under WhyFix 1's The gap is precise: "a source that loaded zero artifacts" and "no sources configured at all" are different conditions, and only the first is covered. Revised fix — three parts, not one
(3) is the one the current PR is missing, and it is the reason (1) is safe. Also still needed, from my previous comment: a CHANGELOG entry, and artifact trailers — the diff touches |
…e case-2 gap (#808) Maintainer review on #818 caught two issues in the initial revision: 1. **`deny_unknown_fields` on `ProjectConfig` broke sigil** — sigil's `rivet.yaml` legitimately carries a top-level `schemas-path:` that rivet has always silently ignored. Hard-failing on it made rivet unusable in the org's most active downstream (1242 artifacts). Same silent-load failure direction as #808, pointed the other way: silently accepts nothing → loudly rejects what worked yesterday. 2. **The naive downgrade of Fix 3 reopens case 2.** Simply dropping `deny_unknown_fields` and relying on the existing `empty-source` diagnostic doesn't close the reproducer: when `sources:` is mis-keyed as `typo_sources:`, the config has ZERO configured sources — so the loop that emits `empty-source` never runs, and `validate --strict` still exits 0 over silently green output. ## Revised fix — three parts, matching the maintainer's spec 1. **Drop `#[serde(deny_unknown_fields)]` from `ProjectConfig`**. sigil and every other downstream with legitimate extra top-level keys loads cleanly again. 2. **New `unknown-config-key` diagnostic**. `load_project_config_with_report` in rivet-core parses the file once as `ProjectConfig` and once as a plain `Value`, then reports any top-level keys not in the exported `KNOWN_RIVET_YAML_TOP_LEVEL_KEYS` slice (kept alphabetized so a drift between the struct and the slice is obvious in a diff). `cmd_validate` consumes the unknown-keys list from `ProjectContext` and emits one `unknown-config-key` Warning per unknown key, naming the offending key AND listing every valid alternative — same helpful shape the original hard-fail parse error had, minus the breaking behavior. Warning by default, Error under `--strict`. 3. **New `no-sources` diagnostic**. Fires when `config.sources.is_empty()` — independent of the per-source loop, so it catches the exact case-2 gap the maintainer's probe demonstrated (`typo_sources:` leaves `sources:` defaulted to `[]`, per-source loop iterates nothing, nothing else fires). Also closes the plain `sources: []` case that nothing covered before. Warning by default, Error under `--strict`. ## What was preserved from the original PR Every other #808 defense from the first commit is unchanged: `artifact-root-key-near-miss`, `empty-source`, `coverage` rendering `n/a` instead of `100.0%`, the `--strict-empty` flag, `--fail-under` failing on empty load, the `empty_scope: bool` JSON field, and the `is_empty_scope()` helpers on `CoverageEntry`/`ClosureEntry`/`CoverageReport`. ## Regression tests - `rivet_yaml_unknown_top_level_key_hard_fails` **renamed** to `rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict`. Uses a sigil-style reproducer (`schemas-path: schemas` alongside a real `sources:` block) and asserts default validate PASSes with the diagnostic visible, while `--strict` exits 1. Locks the sigil compatibility guarantee. - **New**: `typoed_sources_key_fires_no_sources_diagnostic`. The exact case-2 reproducer from the issue (`typo_sources:` where `sources:` was meant). Asserts default validate surfaces `no-sources` and `--strict` exits 1. This is the test that would have caught the naive-downgrade regression the maintainer flagged; it now runs on every CI push. ## CHANGELOG Added the `## [Unreleased]` entry — Fixed section describing all four diagnostics, Changed section calling out the potentially-breaking JSON shape (`percentage: null` for empty scope, new `empty_scope` field) and text shape (`n/a%` where `100.0%` used to appear) for machine consumers. ## Test plan - `cargo build -p rivet-cli` — clean. - `cargo test -p rivet-cli --test cli_commands` — **167 pass** (162 pre-existing + 5 #808 regressions). - `cargo test -p rivet-core --lib` — **1186 pass**. - `cargo fmt --all -- --check` — clean. - `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` — clean (only pre-existing MSRV note). - `rivet validate` on this repo — PASS (638 warnings; the 3-warning bump vs previous is REQ-294 having no incoming `verifies` links yet, as expected for a just-authored requirement). - `rivet get REQ-294` — resolves; artifact loads cleanly. - Manual smoke on the sigil-style reproducer (`schemas-path:` + real sources): default `validate` exit=0 with `unknown-config-key` warning; `--strict` exit=1. Rivet is usable in sigil again. - Manual smoke on the case-2 reproducer (`typo_sources:`): default `validate` surfaces `no-sources` warning; `--strict` exit=1. Case 2 no longer silently green. Fixes: REQ-294 Verifies: REQ-294
|
Pushed
Verified against your probe: Sigil compatibility guarantee locked by CHANGELOG entry added in Artifact trailer: REQ-294 added to Local: Generated by Claude Code |
…adiness Release planning pass. Readiness is a query, so the first job was making the query true. `rivet release status v0.33.0` said "NOT cuttable — 4 artifacts not yet verified" for a release that had already shipped, signed and published. The four gate-potency requirements sat at `implemented`. Each was demonstrated red-then- green in CI before merge, which is what verified means, so the honest fix is the status rather than loosening the query. v0.33.0 now reads "Cuttable — every artifact is release-ready" and the tool agrees with reality. Triaged 6 of 18 issues that had no artifact and therefore no release, so were invisible to the plan. REQ-294 (#808), REQ-296 (#809) and REQ-297 (#811) join REQ-298 (#812) in v0.34.0; REQ-295 (#807) and REQ-299 (#816) go to v0.35.0. REQ-294 records a sequencing constraint found by probing PR #818 rather than reading its diff: serde deny_unknown_fields closes the config-typo trigger but makes rivet unusable in sigil, whose rivet.yaml carries a top-level schemas-path rivet has always ignored. Downgrading it to a warning reopens the trigger, because the empty-source check iterates over configured sources and a mis-keyed sources leaves none. Confirmed with a build, PASS and 0 warnings even under --strict. Needs three parts, not one. REQ-295 records that its two defects must be fixed in order — closing the stub-satisfiability hole first would turn 14 false failures into hard ones. Deferred REQ-281, REQ-282, REQ-285 and REQ-286 from v0.34.0 to v0.35.0. This is a scope decision, logged rather than silent: v0.34.0 leads with the three customer UI requirements that have already slipped two releases, plus the write path work the last two days made urgent. Refs: REQ-294, REQ-295, REQ-296, REQ-297, REQ-298, REQ-299
…adiness (#819) Release planning pass. Readiness is a query, so the first job was making the query true. `rivet release status v0.33.0` said "NOT cuttable — 4 artifacts not yet verified" for a release that had already shipped, signed and published. The four gate-potency requirements sat at `implemented`. Each was demonstrated red-then- green in CI before merge, which is what verified means, so the honest fix is the status rather than loosening the query. v0.33.0 now reads "Cuttable — every artifact is release-ready" and the tool agrees with reality. Triaged 6 of 18 issues that had no artifact and therefore no release, so were invisible to the plan. REQ-294 (#808), REQ-296 (#809) and REQ-297 (#811) join REQ-298 (#812) in v0.34.0; REQ-295 (#807) and REQ-299 (#816) go to v0.35.0. REQ-294 records a sequencing constraint found by probing PR #818 rather than reading its diff: serde deny_unknown_fields closes the config-typo trigger but makes rivet unusable in sigil, whose rivet.yaml carries a top-level schemas-path rivet has always ignored. Downgrading it to a warning reopens the trigger, because the empty-source check iterates over configured sources and a mis-keyed sources leaves none. Confirmed with a build, PASS and 0 warnings even under --strict. Needs three parts, not one. REQ-295 records that its two defects must be fixed in order — closing the stub-satisfiability hole first would turn 14 false failures into hard ones. Deferred REQ-281, REQ-282, REQ-285 and REQ-286 from v0.34.0 to v0.35.0. This is a scope decision, logged rather than silent: v0.34.0 leads with the three customer UI requirements that have already slipped two releases, plus the write path work the last two days made urgent. Refs: REQ-294, REQ-295, REQ-296, REQ-297, REQ-298, REQ-299
…e case-2 gap (#808) Maintainer review on #818 caught two issues in the initial revision: 1. **`deny_unknown_fields` on `ProjectConfig` broke sigil** — sigil's `rivet.yaml` legitimately carries a top-level `schemas-path:` that rivet has always silently ignored. Hard-failing on it made rivet unusable in the org's most active downstream (1242 artifacts). Same silent-load failure direction as #808, pointed the other way: silently accepts nothing → loudly rejects what worked yesterday. 2. **The naive downgrade of Fix 3 reopens case 2.** Simply dropping `deny_unknown_fields` and relying on the existing `empty-source` diagnostic doesn't close the reproducer: when `sources:` is mis-keyed as `typo_sources:`, the config has ZERO configured sources — so the loop that emits `empty-source` never runs, and `validate --strict` still exits 0 over silently green output. 1. **Drop `#[serde(deny_unknown_fields)]` from `ProjectConfig`**. sigil and every other downstream with legitimate extra top-level keys loads cleanly again. 2. **New `unknown-config-key` diagnostic**. `load_project_config_with_report` in rivet-core parses the file once as `ProjectConfig` and once as a plain `Value`, then reports any top-level keys not in the exported `KNOWN_RIVET_YAML_TOP_LEVEL_KEYS` slice (kept alphabetized so a drift between the struct and the slice is obvious in a diff). `cmd_validate` consumes the unknown-keys list from `ProjectContext` and emits one `unknown-config-key` Warning per unknown key, naming the offending key AND listing every valid alternative — same helpful shape the original hard-fail parse error had, minus the breaking behavior. Warning by default, Error under `--strict`. 3. **New `no-sources` diagnostic**. Fires when `config.sources.is_empty()` — independent of the per-source loop, so it catches the exact case-2 gap the maintainer's probe demonstrated (`typo_sources:` leaves `sources:` defaulted to `[]`, per-source loop iterates nothing, nothing else fires). Also closes the plain `sources: []` case that nothing covered before. Warning by default, Error under `--strict`. Every other #808 defense from the first commit is unchanged: `artifact-root-key-near-miss`, `empty-source`, `coverage` rendering `n/a` instead of `100.0%`, the `--strict-empty` flag, `--fail-under` failing on empty load, the `empty_scope: bool` JSON field, and the `is_empty_scope()` helpers on `CoverageEntry`/`ClosureEntry`/`CoverageReport`. - `rivet_yaml_unknown_top_level_key_hard_fails` **renamed** to `rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict`. Uses a sigil-style reproducer (`schemas-path: schemas` alongside a real `sources:` block) and asserts default validate PASSes with the diagnostic visible, while `--strict` exits 1. Locks the sigil compatibility guarantee. - **New**: `typoed_sources_key_fires_no_sources_diagnostic`. The exact case-2 reproducer from the issue (`typo_sources:` where `sources:` was meant). Asserts default validate surfaces `no-sources` and `--strict` exits 1. This is the test that would have caught the naive-downgrade regression the maintainer flagged; it now runs on every CI push. Added the `## [Unreleased]` entry — Fixed section describing all four diagnostics, Changed section calling out the potentially-breaking JSON shape (`percentage: null` for empty scope, new `empty_scope` field) and text shape (`n/a%` where `100.0%` used to appear) for machine consumers. - `cargo build -p rivet-cli` — clean. - `cargo test -p rivet-cli --test cli_commands` — **167 pass** (162 pre-existing + 5 #808 regressions). - `cargo test -p rivet-core --lib` — **1186 pass**. - `cargo fmt --all -- --check` — clean. - `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` — clean (only pre-existing MSRV note). - `rivet validate` on this repo — PASS (638 warnings; the 3-warning bump vs previous is REQ-294 having no incoming `verifies` links yet, as expected for a just-authored requirement). - `rivet get REQ-294` — resolves; artifact loads cleanly. - Manual smoke on the sigil-style reproducer (`schemas-path:` + real sources): default `validate` exit=0 with `unknown-config-key` warning; `--strict` exit=1. Rivet is usable in sigil again. - Manual smoke on the case-2 reproducer (`typo_sources:`): default `validate` surfaces `no-sources` warning; `--strict` exit=1. Case 2 no longer silently green. Fixes: REQ-294 Verifies: REQ-294
6ce0e71 to
fb52939
Compare
Unblocking this — my objection is resolved, and I found one more gapI blocked this PR earlier over the sigil compatibility break. That objection The sigil constraint holdsIt loads, emits The gap:
|
| diagnostic | default | legitimate case |
|---|---|---|
empty-source |
Warning | fresh project, source not yet populated |
unknown-config-key |
Warning | sigil's schemas-path: |
artifact-root-key-near-miss |
Warning | heuristic, can misfire |
no-sources |
Error | none — rivet init always scaffolds sources: |
Verified that last claim rather than assuming it: a fresh rivet init validates
PASS (0 warnings), exit 0, so this cannot fire on the happy path.
Controls after the change:
sigil-shaped config exit 0 PASS <- compat held
mis-keyed typo_sources: exit 1 FAIL <- gate now bites
fresh `rivet init` exit 0 PASS <- happy path unaffected
The existing test could not have caught this
typoed_sources_key_fires_no_sources_diagnostic asserted the message appeared
and that --strict failed — but never the default exit code, so it passed
identically before and after. Added that assertion. It's the load-bearing one,
and without it the severity can be flipped back silently.
Two CHANGELOG defects repaired
- Entries were filed under
## [0.33.0], which already shipped (0.33.1 is
out). Moved to[Unreleased]. - A duplicate
## [0.33.0]heading dated2026-08-11had been added
alongside the real2026-08-18one. Removed; the diff againstmainis now
purely additive.
Also documented the exit-code change under "Changed (potentially breaking)" —
it is user-visible and deserves to be findable.
REQ-294 rebase note
Both sides carried a REQ-294; keep-both would have produced a duplicate id.
Resolved to one artifact keeping main's release: v0.34.0 (the branch's
copy had no release: field, which would have dropped it out of the v0.34.0
readiness query) and the branch's description + implemented status, which
describes what actually shipped.
Gates (exit codes captured, not read off a piped tail)
cargo fmt --all -- --check— exit 0cargo test --workspace— exit 0, 65test result: ok, 0 FAILEDrustup run 1.97.0 cargo clippy --all-targets -- -D warnings— exit 0rivet validate— exit 0, PASSrivet docs check— exit 0, PASS
Ready to merge once CI is green.
|
Two things you got right that I got wrong; noting them for the record.
CHANGELOG placement. The section I intended to write into was Standing down — thanks for unblocking. All eyes on CI. Generated by Claude Code |
|
Pushed Each killer test targets a mutant-equivalence class rather than restating what the fix already asserts. My original tests exercised only the true side of each
The Not fixed:
Generated by Claude Code |
1455538 to
4d8a37e
Compare
Two more findings pushed to this branch1. Six
|
| command | main | this PR |
|---|---|---|
validate |
0 | 1 ← the no-sources change |
coverage --fail-under 50 |
0 | 1 ← this branch's empty-load fix |
coverage |
0 | 0 |
list |
0 | 0 |
check verification-evidence |
0 | 0 |
Exactly two exit-code changes, both intended, both in the CHANGELOG. On the
sigil-shaped fixture (schemas-path: + real sources) validate, coverage
and list all still exit 0 — compatibility holds across commands, not just the
one I originally tested.
(An earlier run of this table showed spurious exit=2 values for the two
multi-word commands. That was a shell-quoting artifact — the whole string was
passed as one argv element and clap rejected it as an unknown subcommand. Re-run
correctly above; there is no third behaviour change.)
Gates after both changes
cargo fmt --check exit 0 · cargo test -p rivet-cli --test cli_commands exit 0,
170 passed / 0 failed · rivet validate exit 0 · rivet docs check exit 0.
|
Marker regex — that's my error, and worth flagging as one to learn from. I typed Cross-command exit-code table is the right shape of check for a severity change — I only ran Standing down; thanks for driving this to green. Generated by Claude Code |
…prints n/a not 100% (#808) For a compliance tool, "loudest-green over a zero read" is the worst possible failure direction. The reporter demonstrated two independent triggers, both yielding **0 artifacts** while every downstream command reported success: 1. A `generic-yaml` source file with `requirements:` at the root (instead of `artifacts:`) — silently classified as a non-artifact file and dropped. 2. A mis-keyed `typo_sources:` in `rivet.yaml` — silently ignored, because top-level keys weren't gated. In both cases `rivet validate` printed `PASS` and `rivet coverage` printed `100.0%` for every rule (0/0 denominator). This closes all four fix sites the issue calls out. `cmd_validate` already collects per-source load reports for its `artifact-parse-error` and `duplicate-artifact-id` diagnostics. Extended the same loop to note any source that returned zero artifacts, and to emit an `empty-source` diagnostic pointing at the offending `sources[].path`. Suppressed when a `ParseError` or `NearMissKey` diag already names the cause, so the user isn't told twice about the same file. Severity: Warning by default (so pre-existing projects don't break their own dogfood `validate`), Error under `--strict` (compliance-gate mode), mirroring the `--strict-orphans` / `allowed-values` pattern. Extended `SkipKind` in `rivet-core/src/formats/generic.rs` with a `NearMissKey(String)` variant, and `classify_skip` now catches single-key files whose root is a plausible artifact-plural typo (`requirements`, `features`, `designs`, `decisions`, `design-decisions`, `tests`, `verifications`). `cmd_validate` emits it as `artifact-root-key-near-miss` naming both the file and the key, so `did you mean `artifacts:`?` is answered in the diagnostic message itself. Same Warning-by-default / Error-under-`--strict` treatment. Added `#[serde(deny_unknown_fields)]` to `ProjectConfig` in `rivet-core/src/model.rs`. A `typo_sources:` (or any other unknown top- level key) is now a load-time parse error naming exactly what's wrong: ``` error: loading ./rivet.yaml: Schema error: ./rivet.yaml: unknown field `typo_sources`, expected one of `project`, `sources`, `docs`, `results`, `commits`, `release`, `externals`, `baselines`, `docs-check` at line 4 column 1 ``` This is stricter than the other two fixes because there's no legitimate reason for a rivet.yaml to carry unknown top-level keys (unlike an artifact file, where a `requirements:` file might really be non-artifact data placed in a source path). Sibling structs in model.rs (source, external, release configs) already had `deny_unknown_fields` — this is closing the top-level hole. Added `is_empty_scope()` helpers on `CoverageEntry`, `ClosureEntry`, and `CoverageReport`. `cmd_coverage`'s renderer now consults them: - Text output: `n/a%` for empty-scope rules (column width preserved so alignment survives), plus a leading banner — `⚠ No artifacts loaded — every rule scores n/a (0/0)` — pointing at the fix and `--strict-empty`. The legacy `100.0%` for empty- scope is gone. - JSON output: per-rule `percentage` and `accounted_percentage` are `null` for empty scope; every rule and the `overall` block gain an `empty_scope: bool` field so a machine consumer can distinguish a satisfied gate from an empty one without inspecting `total`. - New `--strict-empty` flag on `coverage`: exits non-zero when the whole report is empty-scope. Default off (backwards-compatible); turn on in CI as the gate the issue calls for. - `--fail-under` on an empty load ALSO exits non-zero (the historical 100% would have trivially satisfied any threshold — that's the silent-green trap the flag exists to catch). The `percentage()` / `overall_coverage()` / `accounted_percentage()` methods keep their old `f64` signatures + legacy 100.0-on-empty values for the ~40 other callers (renderers in `export.rs`, `render/coverage.rs`, `serve/api.rs`, `snapshot.rs`, `mcp.rs`, `proofs.rs`, `embed.rs`, `stats.rs`, `coverage_evidence.rs`) — their doc-comments now instruct callers to check `is_empty_scope()` first, so the empty-scope distinction is available everywhere without a breaking signature change. Four end-to-end tests in `rivet-cli/tests/cli_commands.rs`, one per fix: - `rivet_yaml_unknown_top_level_key_hard_fails` — writes a project with `typo_sources:`, asserts `validate` exits non-zero and stderr names the offending key. - `validate_flags_generic_yaml_near_miss_root_key` — writes a `requirements:`-rooted file, asserts default validate surfaces `artifact-root-key-near-miss` and `--strict` fails. - `validate_flags_empty_source` — writes an empty source directory, asserts default validate surfaces `empty-source` and `--strict` fails. - `coverage_prints_n_a_and_exits_nonzero_on_empty_load` — asserts text renders `n/a%` and the empty-load banner (and NEVER `100.0%`); JSON `overall.empty_scope=true` and `percentage=null`; `--strict-empty` and `--fail-under` both exit non-zero. Plus the existing `zero_artifacts_gives_100_percent` unit test in `coverage.rs` was renamed / updated to `zero_artifacts_is_flagged_as_empty_scope_not_a_100_percent_pass` — it now asserts BOTH the new `is_empty_scope()` invariant AND that the legacy 100.0 return is preserved for downstream callers. - `cargo test -p rivet-core --lib` — **1186 pass** (unchanged count, the coverage test was renamed rather than added). - `cargo test -p rivet-cli --test cli_commands` — **166 pass** (162 pre-existing + 4 new). - `cargo fmt --all -- --check` — clean. - `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` — clean (only pre-existing MSRV note). - `rivet validate` on this repo — PASS (635 warnings), same as before the diff. No new diagnostics against the rivet self-project because its `sources:` list resolves cleanly and every configured source produces artifacts. - Not a change to any percentage method's return type. The 40+ downstream callers (HTML export, JSON snapshot, MCP tool, dashboard serve, embed engine, proof reporter, coverage evidence, ...) all keep working unchanged; the empty-scope distinction is available via a NEW `is_empty_scope()` helper they can adopt incrementally. - Not a change to `--strict`'s existing semantics for `allowed-values` / `unknown-field`. Just adds the two new `--strict`-promoted rules (`artifact-root-key-near-miss`, `empty-source`) to the existing escalation block. - Not a change to the `NotArtifactFile` classification for files the user legitimately places in a source path (`bindings.yaml`, `feature-model.yaml`, `variants/*.yaml`). Only single-key files whose root reads as a plausible artifact-plural typo are reclassified. Fixes: REQ-004 Verifies: REQ-004
…e case-2 gap (#808) Maintainer review on #818 caught two issues in the initial revision: 1. **`deny_unknown_fields` on `ProjectConfig` broke sigil** — sigil's `rivet.yaml` legitimately carries a top-level `schemas-path:` that rivet has always silently ignored. Hard-failing on it made rivet unusable in the org's most active downstream (1242 artifacts). Same silent-load failure direction as #808, pointed the other way: silently accepts nothing → loudly rejects what worked yesterday. 2. **The naive downgrade of Fix 3 reopens case 2.** Simply dropping `deny_unknown_fields` and relying on the existing `empty-source` diagnostic doesn't close the reproducer: when `sources:` is mis-keyed as `typo_sources:`, the config has ZERO configured sources — so the loop that emits `empty-source` never runs, and `validate --strict` still exits 0 over silently green output. 1. **Drop `#[serde(deny_unknown_fields)]` from `ProjectConfig`**. sigil and every other downstream with legitimate extra top-level keys loads cleanly again. 2. **New `unknown-config-key` diagnostic**. `load_project_config_with_report` in rivet-core parses the file once as `ProjectConfig` and once as a plain `Value`, then reports any top-level keys not in the exported `KNOWN_RIVET_YAML_TOP_LEVEL_KEYS` slice (kept alphabetized so a drift between the struct and the slice is obvious in a diff). `cmd_validate` consumes the unknown-keys list from `ProjectContext` and emits one `unknown-config-key` Warning per unknown key, naming the offending key AND listing every valid alternative — same helpful shape the original hard-fail parse error had, minus the breaking behavior. Warning by default, Error under `--strict`. 3. **New `no-sources` diagnostic**. Fires when `config.sources.is_empty()` — independent of the per-source loop, so it catches the exact case-2 gap the maintainer's probe demonstrated (`typo_sources:` leaves `sources:` defaulted to `[]`, per-source loop iterates nothing, nothing else fires). Also closes the plain `sources: []` case that nothing covered before. Warning by default, Error under `--strict`. Every other #808 defense from the first commit is unchanged: `artifact-root-key-near-miss`, `empty-source`, `coverage` rendering `n/a` instead of `100.0%`, the `--strict-empty` flag, `--fail-under` failing on empty load, the `empty_scope: bool` JSON field, and the `is_empty_scope()` helpers on `CoverageEntry`/`ClosureEntry`/`CoverageReport`. - `rivet_yaml_unknown_top_level_key_hard_fails` **renamed** to `rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict`. Uses a sigil-style reproducer (`schemas-path: schemas` alongside a real `sources:` block) and asserts default validate PASSes with the diagnostic visible, while `--strict` exits 1. Locks the sigil compatibility guarantee. - **New**: `typoed_sources_key_fires_no_sources_diagnostic`. The exact case-2 reproducer from the issue (`typo_sources:` where `sources:` was meant). Asserts default validate surfaces `no-sources` and `--strict` exits 1. This is the test that would have caught the naive-downgrade regression the maintainer flagged; it now runs on every CI push. Added the `## [Unreleased]` entry — Fixed section describing all four diagnostics, Changed section calling out the potentially-breaking JSON shape (`percentage: null` for empty scope, new `empty_scope` field) and text shape (`n/a%` where `100.0%` used to appear) for machine consumers. - `cargo build -p rivet-cli` — clean. - `cargo test -p rivet-cli --test cli_commands` — **167 pass** (162 pre-existing + 5 #808 regressions). - `cargo test -p rivet-core --lib` — **1186 pass**. - `cargo fmt --all -- --check` — clean. - `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` — clean (only pre-existing MSRV note). - `rivet validate` on this repo — PASS (638 warnings; the 3-warning bump vs previous is REQ-294 having no incoming `verifies` links yet, as expected for a just-authored requirement). - `rivet get REQ-294` — resolves; artifact loads cleanly. - Manual smoke on the sigil-style reproducer (`schemas-path:` + real sources): default `validate` exit=0 with `unknown-config-key` warning; `--strict` exit=1. Rivet is usable in sigil again. - Manual smoke on the case-2 reproducer (`typo_sources:`): default `validate` surfaces `no-sources` warning; `--strict` exit=1. Case 2 no longer silently green. Fixes: REQ-294 Verifies: REQ-294
…#808) The four new #808 diagnostics all landed Warning-by-default, Error under --strict. That leaves the gate inert where it matters: this repo's own Traceability job — described in ci.yml as the single most load-bearing gate — runs plain `rivet validate` with no --strict, as does the hosted-floor mirror and release.yml. A mis-keyed `sources:` therefore still produced exit 0 over zero loaded artifacts, which is the #808 defect itself with warnings printed over the top. no-sources is the one of the four with no legitimate case, so it is the one that changes. The others keep Warning-by-default deliberately: a fresh project whose source is not yet populated (empty-source), a downstream repo carrying its own top-level keys such as sigil's schemas-path: (unknown-config-key), and a heuristic that can misfire (artifact-root-key-near-miss). Three negative controls, judged on exit status: sigil-shaped config (top-level schemas-path:) exit 0 PASS <- compat held mis-keyed typo_sources: exit 1 FAIL <- gate bites fresh `rivet init` exit 0 PASS <- happy path The existing test asserted the diagnostic text appeared and that --strict failed, but never asserted the default exit code — so it passed identically before and after this change. Added that assertion; it is the load-bearing one. Also repairs two CHANGELOG defects on this branch: the entries were filed under the already-shipped [0.33.0] section, and a duplicate `## [0.33.0]` heading dated 2026-08-11 had been introduced alongside the real 2026-08-18 one. Entries moved to [Unreleased]; the diff against main is now purely additive. REQ-294 kept its `release: v0.34.0` through the rebase — the branch's copy had dropped the field, which would have made the artifact invisible to the v0.34.0 readiness query. Confirmed with cargo fmt --check, cargo test --workspace (exit 0, 65 ok lines), clippy --all-targets -D warnings on 1.97.0, rivet validate, and rivet docs check — all exit 0. Implements: REQ-294 Fixes: REQ-004 Refs: #808
…ssue numbers The test-marker scanner matches `//\s*rivet:\s*(verifies|partially-verifies)\s+([\w-]+)`. `#` is not a word character, so `/// rivet: verifies #808` does not match at all — the marker is silently invisible to the trace graph rather than being a broken link that something reports. Six markers were in that state: five `#808` added by this branch and one `#812` already on main. 593 other markers in the tree use artifact ids; these were the outliers. Verified against the actual regex rather than by inspection: '/// rivet: verifies REQ-294' -> REQ-294 '/// rivet: verifies #808' -> NO MATCH '/// rivet: verifies #812' -> NO MATCH And after the change, `rivet coverage --tests` reports REQ-294 with 5 test markers and REQ-298 with 1, where both previously sat in the uncovered list. This is what makes the V closable for these two requirements: a merged PR is evidence of `implemented`, but `verified` needs a test that actually names the artifact. REQ-297 and REQ-302 remain in the uncovered list for the same reason and are deliberately not touched here. Verifies: REQ-294 Refs: REQ-298
4d8a37e to
4f221ae
Compare
|
Small housekeeping observation before merge: my If that's deliberate (advisory check ⇒ not worth carrying), no action needed. If it was accidental during the marker-fix rebase, I have the commit locally (unchanged from what merged the codecov jump from 90.32% → 98.05%) and can cherry-pick it back onto the current tip. Which would you prefer? Doesn't block merge either way — the mutation gate is advisory, and the codecov threshold was met without them too. Just wanted to surface it in case the drop wasn't intentional. Generated by Claude Code |
Closes #808.
For a traceability/compliance tool,
validateprintingPASSandcoverageprinting100.0%on a zero-artifact load is the worst possible failure direction. The reporter demonstrated it with two independent triggers — arequirements:root key (instead ofartifacts:) in ageneric-yamlfile, and a mis-keyedtypo_sources:inrivet.yaml. Both silently disable inputs; every downstream command reports green. This PR closes all four fix sites the issue names.What changed — one section per issue fix
Fix 1 —
empty-sourcediagnostic on any source that produces zero artifactscmd_validatealready collects per-source load reports forartifact-parse-errorandduplicate-artifact-id. Extended the same loop to note any source that returned zero artifacts and emit anempty-sourceWarning namingsources[].path. Suppressed when aParseErrororNearMissKeydiagnostic already names the cause, so no double-report.Fix 2 —
artifact-root-key-near-missforrequirements:and friendsExtended
SkipKindinrivet-core/src/formats/generic.rswithNearMissKey(String);classify_skipcatches single-root-key files whose key is a plausible artifact-plural typo (requirements,features,designs,decisions,design-decisions,tests,verifications).cmd_validateemitsartifact-root-key-near-missnaming both the file and the offending key — the diagnostic itself answers "did you meanartifacts:?".Fix 3 —
rivet.yamlunknown top-level keys hard-failAdded
#[serde(deny_unknown_fields)]toProjectConfiginrivet-core/src/model.rs.typo_sources:is now a load-time parse error that names the offending key and lists every accepted alternative. Sibling structs inmodel.rsalready haddeny_unknown_fields— this is closing the top-level hole.Fix 4 —
coveragerendersn/a, not100.0%, on 0/0 denominatorsAdded
is_empty_scope()toCoverageEntry,ClosureEntry,CoverageReport. The renderer now:n/a%for empty-scope rules (column width preserved); leading banner⚠ No artifacts loaded — every rule scores n/a (0/0)on a fully-empty report; the legacy100.0%for empty-scope is gone.percentage/accounted_percentageandoverall.percentageemitnullfor empty scope; every rule and theoverallblock gain a newempty_scope: boolfield so a machine consumer can tell a satisfied gate from an empty one without inspectingtotal.--strict-emptyflag: exits non-zero when the whole report is empty-scope. Default off (backwards-compatible); on in CI.--fail-underon an empty load also exits non-zero (the historical 100% would have trivially satisfied any threshold — the exact silent-green trap the flag exists to catch).The
percentage()methods keep theirf64signatures and legacy 100.0-on-empty return for the ~40 other callers (renderers inexport.rs,render/coverage.rs,serve/api.rs,snapshot.rs,mcp.rs,proofs.rs,embed.rs,stats.rs,coverage_evidence.rs) — their doc-comments now direct callers to checkis_empty_scope()first, so the distinction is available everywhere without a breaking signature change.Acceptance criteria (from the issue "Suggested fix" section) → how satisfied
⚠+ non-zero under strict) when a configured source resolves to zero artifacts. Distinguish "path does not exist", "path exists but no matching files", "files parsed but yielded no artifacts". — [x]empty-sourcediagnostic emitted bycmd_validateon any source withreport.artifacts.is_empty(); suppressed only when a parse-error or near-miss diagnostic on a file inside that source already names the more specific cause (so path-exists + files-parsed-yielded-zero cases are covered by the more precise diag). Warning by default, Error under--strict. Regression:validate_flags_empty_source.generic-yamlsource file —requirements:is the obvious near-miss and should be named explicitly in the message. — [x]artifact-root-key-near-missdiagnostic names both the file and the key, and the message literally says "did you meanartifacts:?". The reporter'srequirements:case is covered by name plus six other common plural typos. Warning by default, Error under--strict. Regression:validate_flags_generic_yaml_near_miss_root_key.rivet.yaml(serdedeny_unknown_fieldsor an explicit check). A mis-keyed section silently disabling all inputs should be impossible. — [x]#[serde(deny_unknown_fields)]onProjectConfiginrivet-core/src/model.rs. Went with "hard-fail on load" (strictly stronger than the issue's ask): a compliance tool's config file has no legitimate need for extra top-level keys, and the parse error names every valid alternative. Regression:rivet_yaml_unknown_top_level_key_hard_fails.coveragemust not print 100% for a 0/0 denominator — reportn/aorno artifactsand exit non-zero under strict.[0/0]is already in the output; the percentage should follow it, not contradict it. — [x] Text rendersn/a%for empty-scope rules (never100.0%); JSON percentages arenullwithempty_scope: true; a new--strict-emptyflag exits non-zero on empty-scope;--fail-underalso exits non-zero on empty-scope. Regression:coverage_prints_n_a_and_exits_nonzero_on_empty_loadcovers all four surfaces (text banner, textn/a, JSON null, both exit-code paths).Test plan
cargo test -p rivet-core --lib— 1186 pass (unchanged; the existing coverage test was renamed tozero_artifacts_is_flagged_as_empty_scope_not_a_100_percent_passand now asserts the new invariant in addition to legacy value).cargo test -p rivet-cli --test cli_commands— 166 pass (162 pre-existing + 4 new).cargo fmt --all -- --check— clean.cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings— clean (only the pre-existing MSRV note).rivet validateon this repo — PASS (635 warnings, same as before). No new diagnostics against the rivet self-project because every configured source resolves cleanly and produces artifacts.requirements:root key +typo_sources:in rivet.yaml) — every fix behaves as the acceptance criteria describe.What this PR is NOT
percentage()/overall_coverage()/accounted_percentage(). The 40+ downstream callers keep working unchanged; the empty-scope distinction is opt-in viais_empty_scope().--strict's existingallowed-values/unknown-fieldbehavior. Adds two new rules (artifact-root-key-near-miss,empty-source) to the same escalation block.NotArtifactFileclassification for legitimate non-artifact YAML (bindings.yaml,feature-model.yaml,variants/*.yaml). Only single-root-key files with a plausible artifact-plural typo are reclassified as near-miss.Generated by Claude Code