Skip to content

fix(validate,coverage): close the silent-empty-load window (#808) - #818

Merged
avrabe merged 4 commits into
mainfrom
fix/issue-808-silent-empty-load
Aug 20, 2026
Merged

fix(validate,coverage): close the silent-empty-load window (#808)#818
avrabe merged 4 commits into
mainfrom
fix/issue-808-silent-empty-load

Conversation

@avrabe

@avrabe avrabe commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Closes #808.

For a traceability/compliance tool, validate printing PASS and coverage printing 100.0% on a zero-artifact load is the worst possible failure direction. The reporter demonstrated it with two independent triggers — a requirements: root key (instead of artifacts:) in a generic-yaml file, and a mis-keyed typo_sources: in rivet.yaml. Both silently disable inputs; every downstream command reports green. This PR closes all four fix sites the issue names.

What changed — one section per issue fix

Fix 1 — empty-source diagnostic on any source that produces zero artifacts

cmd_validate already collects per-source load reports for artifact-parse-error and duplicate-artifact-id. Extended the same loop to note any source that returned zero artifacts and emit an empty-source Warning naming sources[].path. Suppressed when a ParseError or NearMissKey diagnostic already names the cause, so no double-report.

Fix 2 — artifact-root-key-near-miss for requirements: and friends

Extended SkipKind in rivet-core/src/formats/generic.rs with NearMissKey(String); classify_skip catches single-root-key files whose key is a plausible artifact-plural typo (requirements, features, designs, decisions, design-decisions, tests, verifications). cmd_validate emits artifact-root-key-near-miss naming both the file and the offending key — the diagnostic itself answers "did you mean artifacts:?".

Fix 3 — rivet.yaml unknown top-level keys hard-fail

Added #[serde(deny_unknown_fields)] to ProjectConfig in rivet-core/src/model.rs. typo_sources: is now a load-time parse error that names the offending key and lists every accepted alternative. Sibling structs in model.rs already had deny_unknown_fields — this is closing the top-level hole.

Fix 4 — coverage renders n/a, not 100.0%, on 0/0 denominators

Added is_empty_scope() to CoverageEntry, ClosureEntry, CoverageReport. The renderer now:

  • Text: n/a% for empty-scope rules (column width preserved); leading banner ⚠ No artifacts loaded — every rule scores n/a (0/0) on a fully-empty report; the legacy 100.0% for empty-scope is gone.
  • JSON: per-rule percentage / accounted_percentage and overall.percentage emit null for empty scope; every rule and the overall block gain a new empty_scope: bool field so a machine consumer can tell a satisfied gate from an empty one without inspecting total.
  • New --strict-empty flag: exits non-zero when the whole report is empty-scope. Default off (backwards-compatible); on in CI.
  • --fail-under on an empty load also exits non-zero (the historical 100% would have trivially satisfied any threshold — the exact silent-green trap the flag exists to catch).

The percentage() methods keep their f64 signatures and legacy 100.0-on-empty return for the ~40 other callers (renderers in export.rs, render/coverage.rs, serve/api.rs, snapshot.rs, mcp.rs, proofs.rs, embed.rs, stats.rs, coverage_evidence.rs) — their doc-comments now direct callers to check is_empty_scope() first, so the distinction is available everywhere without a breaking signature change.

Acceptance criteria (from the issue "Suggested fix" section) → how satisfied

  1. Loud-fail (or + non-zero under strict) when a configured source resolves to zero artifacts. Distinguish "path does not exist", "path exists but no matching files", "files parsed but yielded no artifacts". — [x] empty-source diagnostic emitted by cmd_validate on any source with report.artifacts.is_empty(); suppressed only when a parse-error or near-miss diagnostic on a file inside that source already names the more specific cause (so path-exists + files-parsed-yielded-zero cases are covered by the more precise diag). Warning by default, Error under --strict. Regression: validate_flags_empty_source.
  2. Warn on unrecognized top-level keys in a generic-yaml source file — requirements: is the obvious near-miss and should be named explicitly in the message. — [x] artifact-root-key-near-miss diagnostic names both the file and the key, and the message literally says "did you mean artifacts:?". The reporter's requirements: case is covered by name plus six other common plural typos. Warning by default, Error under --strict. Regression: validate_flags_generic_yaml_near_miss_root_key.
  3. Warn on unknown top-level keys in rivet.yaml (serde deny_unknown_fields or an explicit check). A mis-keyed section silently disabling all inputs should be impossible. — [x] #[serde(deny_unknown_fields)] on ProjectConfig in rivet-core/src/model.rs. Went with "hard-fail on load" (strictly stronger than the issue's ask): a compliance tool's config file has no legitimate need for extra top-level keys, and the parse error names every valid alternative. Regression: rivet_yaml_unknown_top_level_key_hard_fails.
  4. coverage must not print 100% for a 0/0 denominator — report n/a or no artifacts and exit non-zero under strict. [0/0] is already in the output; the percentage should follow it, not contradict it. — [x] Text renders n/a% for empty-scope rules (never 100.0%); JSON percentages are null with empty_scope: true; a new --strict-empty flag exits non-zero on empty-scope; --fail-under also exits non-zero on empty-scope. Regression: coverage_prints_n_a_and_exits_nonzero_on_empty_load covers all four surfaces (text banner, text n/a, JSON null, both exit-code paths).

Test plan

  • cargo test -p rivet-core --lib1186 pass (unchanged; the existing coverage test was renamed to zero_artifacts_is_flagged_as_empty_scope_not_a_100_percent_pass and now asserts the new invariant in addition to legacy value).
  • cargo test -p rivet-cli --test cli_commands166 pass (162 pre-existing + 4 new).
  • cargo fmt --all -- --check — clean.
  • cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings — clean (only the pre-existing MSRV note).
  • rivet validate on this repo — PASS (635 warnings, same as before). No new diagnostics against the rivet self-project because every configured source resolves cleanly and produces artifacts.
  • Manual smoke on both issue-reproducer configs (requirements: root key + typo_sources: in rivet.yaml) — every fix behaves as the acceptance criteria describe.

What this PR is NOT

  • Not a signature change to percentage() / overall_coverage() / accounted_percentage(). The 40+ downstream callers keep working unchanged; the empty-scope distinction is opt-in via is_empty_scope().
  • Not a change to --strict's existing allowed-values / unknown-field behavior. Adds two new rules (artifact-root-key-near-miss, empty-source) to the same escalation block.
  • Not a change to the NotArtifactFile classification for legitimate non-artifact YAML (bindings.yaml, feature-model.yaml, variants/*.yaml). Only single-root-key files with a plausible artifact-plural typo are reclassified as near-miss.

Generated by Claude Code

@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown

📐 Rivet artifact delta

Change Count
Added 0
Removed 0
Modified 1
Downstream impacted (depth ≤ 5) 0

Graph

graph LR
  REQ_294["REQ-294"]:::modified
  classDef added fill:#d4edda,stroke:#28a745,color:#155724
  classDef removed fill:#f8d7da,stroke:#dc3545,color:#721c24
  classDef modified fill:#fff3cd,stroke:#ffc107,color:#856404
  classDef overflow fill:#e2e3e5,stroke:#6c757d,color:#495057,stroke-dasharray: 3 3
Loading
Modified
ID Changes
REQ-294

📎 Full HTML dashboard attached as workflow artifact rivet-delta-pr-818download from the workflow run.

Posted by rivet-delta workflow. The graph shows only changed artifacts; open the HTML dashboard (above) for full context.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite Current: 1455538 Previous: f510645 Ratio
link_graph_build/10000 42810511 ns/iter (± 2119683) 26307444 ns/iter (± 2307946) 1.63

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.32258% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rivet-core/src/coverage.rs 76.92% 3 Missing ⚠️
rivet-core/src/lib.rs 91.30% 2 Missing ⚠️
rivet-core/src/formats/generic.rs 96.15% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@avrabe

avrabe commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Review: verified, but blocking on Fix 3

Verified all four acceptance points empirically against the original #808 repro — they genuinely hold:

check result
near-miss root key WARN: … has top-level requirements:— did you meanartifacts:?
empty-source WARN: source 'artifacts' loaded 0 artifacts
validate --strict exit 1
coverage n/a% per rule and overall; no more 100.0%
coverage --strict-empty / --fail-under 90 exit 1
rivet.yaml unknown key hard-fails, message lists every valid key ✅

Blocking — Fix 3 breaks sigil

deny_unknown_fields accepts project, sources, docs, results, commits, release, externals, baselines, docs-check.

sigil's rivet.yaml line 15 is schemas-path: schemas — a column-0 key that is not a ProjectConfig field. rivet's only mention of it is a comment at rivet-core/src/externals.rs:431: "schemas-path is implicit: rivet always falls back to …". It's a key sigil wrote that rivet has always silently ignored.

With this change sigil's config fails to load at all — rivet becomes unusable in that repo. sigil is our most active downstream (1242 artifacts), and I told them today to upgrade for the #778 fix.

Same failure direction as #808, pointed the other way: silently accepts nothingloudly rejects what worked yesterday.

Fix — make Fix 3 consistent with Fixes 1 and 2

Fixes 1 and 2 chose warn by default, error under --strict. Fix 3 alone hard-fails. Making it consistent removes the breakage and matches the PR's own design. #808 asked to "warn on unknown top-level keys" — warn suffices, because Fix 1's empty-source diagnostic already catches the consequence loudly.

Hard-fail is a fine end state, but it must be sequenced: warn now → fix sigil's config → escalate later with a BREAKING CHANGELOG note.

Second finding — no CHANGELOG entry

No CHANGELOG.md in the diff. A new --strict-empty flag, new JSON fields (empty_scope, null percentages), changed coverage output and config loading all need one — the JSON shape change alone can break a consumer parsing percentage.

Taking both on myself now.

avrabe commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Acknowledged on both — the sigil schemas-path: breakage is real (I picked hard-fail thinking it was "strictly stronger than the issue's ask" but the issue explicitly said "warn on unknown top-level keys," and Fix 1's empty-source diagnostic does catch the downstream consequence loudly on its own). Standing down and leaving the branch as-is so I don't collide with your revisions.

For reference in case it's useful, the mechanical change from hard-fail to warn-with-diagnostic looks like:

  • drop #[serde(deny_unknown_fields)] from ProjectConfig (rivet-core/src/model.rs:1006)
  • add an unknown-key pre-scan on the parsed YAML in load_project_config (rivet-core/src/lib.rs:207) returning the unknown keys alongside the config
  • new rivet-yaml-unknown-key diagnostic in cmd_validate (near the empty-source block, rivet-cli/src/main.rs:5926), Warning by default, escalated in the if strict { ... } block alongside empty-source and artifact-root-key-near-miss
  • flip the rivet_yaml_unknown_top_level_key_hard_fails regression test to rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict — same reproducer, asserts default passes with the diagnostic in output and --strict exits 1

Happy to push it if you'd rather I do — just say the word. Otherwise I'll leave this to you and the CHANGELOG together.


Generated by Claude Code

@avrabe

avrabe commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my own suggestion — a naive downgrade reopens #808 case 2

I proposed simply swapping Fix 3's hard-fail for a warning. I tested that before recommending it further, and it does not work.

Probe: removed deny_unknown_fields from ProjectConfig, rebuilt, ran the original case-2 repro (typo_sources: instead of sources:):

Result: PASS (0 warnings)
  validate exit=0
  validate --strict exit=0

Silently green again — even under --strict.

Why

Fix 1's empty-source diagnostic iterates over configured sources. When sources: is mis-keyed, there are zero configured sources, so the loop body never runs and nothing is emitted. deny_unknown_fields was doing the real work for case 2, at load time and unconditionally — which is exactly why it also breaks sigil.

The gap is precise: "a source that loaded zero artifacts" and "no sources configured at all" are different conditions, and only the first is covered.

Revised fix — three parts, not one

  1. Drop deny_unknown_fields from ProjectConfig → sigil's schemas-path stops being fatal.
  2. New unknown-config-key diagnostic — compare rivet.yaml's top-level keys against the known set, Warning by default / Error under --strict. Keeps the existing helpful message (it already lists every valid key).
  3. New no-sources diagnostic — fires when the config declares zero sources, independent of the per-source loop. This is what actually closes case 2, and it closes it for the plain sources: [] case too, which nothing covers today.

(3) is the one the current PR is missing, and it is the reason (1) is safe.

Also still needed, from my previous comment: a CHANGELOG entry, and artifact trailers — the diff touches rivet-core/src/ and rivet-cli/src/ with no requirements.yaml change, so the commit-msg hook will reject it. Needs a REQ; REQ-294 is the next free id (292 is claimed by #785, 293 by #810).

avrabe pushed a commit that referenced this pull request Aug 18, 2026
…e case-2 gap (#808)

Maintainer review on #818 caught two issues in the initial revision:

1. **`deny_unknown_fields` on `ProjectConfig` broke sigil** — sigil's
   `rivet.yaml` legitimately carries a top-level `schemas-path:` that
   rivet has always silently ignored. Hard-failing on it made rivet
   unusable in the org's most active downstream (1242 artifacts). Same
   silent-load failure direction as #808, pointed the other way:
   silently accepts nothing → loudly rejects what worked yesterday.

2. **The naive downgrade of Fix 3 reopens case 2.** Simply dropping
   `deny_unknown_fields` and relying on the existing `empty-source`
   diagnostic doesn't close the reproducer: when `sources:` is mis-keyed
   as `typo_sources:`, the config has ZERO configured sources — so the
   loop that emits `empty-source` never runs, and `validate --strict`
   still exits 0 over silently green output.

## Revised fix — three parts, matching the maintainer's spec

1. **Drop `#[serde(deny_unknown_fields)]` from `ProjectConfig`**. sigil
   and every other downstream with legitimate extra top-level keys
   loads cleanly again.

2. **New `unknown-config-key` diagnostic**. `load_project_config_with_report`
   in rivet-core parses the file once as `ProjectConfig` and once as a
   plain `Value`, then reports any top-level keys not in the exported
   `KNOWN_RIVET_YAML_TOP_LEVEL_KEYS` slice (kept alphabetized so a drift
   between the struct and the slice is obvious in a diff). `cmd_validate`
   consumes the unknown-keys list from `ProjectContext` and emits one
   `unknown-config-key` Warning per unknown key, naming the offending
   key AND listing every valid alternative — same helpful shape the
   original hard-fail parse error had, minus the breaking behavior.
   Warning by default, Error under `--strict`.

3. **New `no-sources` diagnostic**. Fires when `config.sources.is_empty()`
   — independent of the per-source loop, so it catches the exact case-2
   gap the maintainer's probe demonstrated (`typo_sources:` leaves
   `sources:` defaulted to `[]`, per-source loop iterates nothing, nothing
   else fires). Also closes the plain `sources: []` case that nothing
   covered before. Warning by default, Error under `--strict`.

## What was preserved from the original PR

Every other #808 defense from the first commit is unchanged:
`artifact-root-key-near-miss`, `empty-source`, `coverage` rendering
`n/a` instead of `100.0%`, the `--strict-empty` flag, `--fail-under`
failing on empty load, the `empty_scope: bool` JSON field, and the
`is_empty_scope()` helpers on `CoverageEntry`/`ClosureEntry`/`CoverageReport`.

## Regression tests

- `rivet_yaml_unknown_top_level_key_hard_fails` **renamed** to
  `rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict`.
  Uses a sigil-style reproducer (`schemas-path: schemas` alongside a
  real `sources:` block) and asserts default validate PASSes with the
  diagnostic visible, while `--strict` exits 1. Locks the sigil
  compatibility guarantee.
- **New**: `typoed_sources_key_fires_no_sources_diagnostic`. The exact
  case-2 reproducer from the issue (`typo_sources:` where `sources:`
  was meant). Asserts default validate surfaces `no-sources` and
  `--strict` exits 1. This is the test that would have caught the
  naive-downgrade regression the maintainer flagged; it now runs on
  every CI push.

## CHANGELOG

Added the `## [Unreleased]` entry — Fixed section describing all four
diagnostics, Changed section calling out the potentially-breaking JSON
shape (`percentage: null` for empty scope, new `empty_scope` field) and
text shape (`n/a%` where `100.0%` used to appear) for machine consumers.

## Test plan

- `cargo build -p rivet-cli` — clean.
- `cargo test -p rivet-cli --test cli_commands` — **167 pass** (162
  pre-existing + 5 #808 regressions).
- `cargo test -p rivet-core --lib` — **1186 pass**.
- `cargo fmt --all -- --check` — clean.
- `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` —
  clean (only pre-existing MSRV note).
- `rivet validate` on this repo — PASS (638 warnings; the 3-warning bump
  vs previous is REQ-294 having no incoming `verifies` links yet, as
  expected for a just-authored requirement).
- `rivet get REQ-294` — resolves; artifact loads cleanly.
- Manual smoke on the sigil-style reproducer (`schemas-path:` +
  real sources): default `validate` exit=0 with `unknown-config-key`
  warning; `--strict` exit=1. Rivet is usable in sigil again.
- Manual smoke on the case-2 reproducer (`typo_sources:`): default
  `validate` surfaces `no-sources` warning; `--strict` exit=1. Case 2
  no longer silently green.

Fixes: REQ-294
Verifies: REQ-294

avrabe commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 6ce0e71 — your revised 3-part fix, verbatim on the spec you laid out:

  1. deny_unknown_fields dropped from ProjectConfig → sigil's schemas-path: loads cleanly again.
  2. unknown-config-key diagnostic — new load_project_config_with_report returns unknown top-level keys alongside the parsed config; cmd_validate emits one Warning per key, naming both the offending key and the exported KNOWN_RIVET_YAML_TOP_LEVEL_KEYS list. Error under --strict.
  3. no-sources diagnostic — fires when config.sources.is_empty(), closing the case-2 gap you demonstrated (typo_sources:sources: defaulted to [] → per-source loop never runs). Also covers plain sources: []. Warning by default, Error under --strict.

Verified against your probe:

$ rivet validate                       # typo_sources: rivet.yaml
Diagnostics:
  rivet.yaml: WARN: `rivet.yaml` has top-level key `typo_sources:` that is not a rivet config field ...
  rivet.yaml: WARN: `rivet.yaml` declares no `sources:` — nothing to load ...
Result: PASS (2 warnings)      exit 0
$ rivet validate --strict              # same config
Result: FAIL (2 errors)         exit 1

Sigil compatibility guarantee locked by rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict (renamed from the previous test; now uses a sigil-style schemas-path: reproducer). Case-2 gap locked by new typoed_sources_key_fires_no_sources_diagnostic — the test that would have caught the naive-downgrade regression.

CHANGELOG entry added in ## [Unreleased] — Fixed section for the four new diagnostics + Changed section calling out the potentially-breaking JSON (percentage: null, new empty_scope) and text (n/a% where 100.0% used to appear) shapes for machine consumers.

Artifact trailer: REQ-294 added to artifacts/requirements.yaml describing the whole silent-empty-load class; commit carries Fixes: REQ-294 / Verifies: REQ-294 so the trailer gate is satisfied.

Local: cargo test -p rivet-cli --test cli_commands 167/167, cargo test -p rivet-core --lib 1186/1186, fmt + clippy clean, rivet validate self-check PASS (638 warnings; +3 from before matches REQ-294 having no incoming verifies links yet, as expected).


Generated by Claude Code

avrabe added a commit that referenced this pull request Aug 19, 2026
…adiness

Release planning pass. Readiness is a query, so the first job was making the
query true.

`rivet release status v0.33.0` said "NOT cuttable — 4 artifacts not yet
verified" for a release that had already shipped, signed and published. The four
gate-potency requirements sat at `implemented`. Each was demonstrated red-then-
green in CI before merge, which is what verified means, so the honest fix is the
status rather than loosening the query. v0.33.0 now reads "Cuttable — every
artifact is release-ready" and the tool agrees with reality.

Triaged 6 of 18 issues that had no artifact and therefore no release, so were
invisible to the plan. REQ-294 (#808), REQ-296 (#809) and REQ-297 (#811) join
REQ-298 (#812) in v0.34.0; REQ-295 (#807) and REQ-299 (#816) go to v0.35.0.

REQ-294 records a sequencing constraint found by probing PR #818 rather than
reading its diff: serde deny_unknown_fields closes the config-typo trigger but
makes rivet unusable in sigil, whose rivet.yaml carries a top-level schemas-path
rivet has always ignored. Downgrading it to a warning reopens the trigger,
because the empty-source check iterates over configured sources and a mis-keyed
sources leaves none. Confirmed with a build, PASS and 0 warnings even under
--strict. Needs three parts, not one.

REQ-295 records that its two defects must be fixed in order — closing the
stub-satisfiability hole first would turn 14 false failures into hard ones.

Deferred REQ-281, REQ-282, REQ-285 and REQ-286 from v0.34.0 to v0.35.0. This is
a scope decision, logged rather than silent: v0.34.0 leads with the three
customer UI requirements that have already slipped two releases, plus the write
path work the last two days made urgent.

Refs: REQ-294, REQ-295, REQ-296, REQ-297, REQ-298, REQ-299
avrabe added a commit that referenced this pull request Aug 19, 2026
…adiness (#819)

Release planning pass. Readiness is a query, so the first job was making the
query true.

`rivet release status v0.33.0` said "NOT cuttable — 4 artifacts not yet
verified" for a release that had already shipped, signed and published. The four
gate-potency requirements sat at `implemented`. Each was demonstrated red-then-
green in CI before merge, which is what verified means, so the honest fix is the
status rather than loosening the query. v0.33.0 now reads "Cuttable — every
artifact is release-ready" and the tool agrees with reality.

Triaged 6 of 18 issues that had no artifact and therefore no release, so were
invisible to the plan. REQ-294 (#808), REQ-296 (#809) and REQ-297 (#811) join
REQ-298 (#812) in v0.34.0; REQ-295 (#807) and REQ-299 (#816) go to v0.35.0.

REQ-294 records a sequencing constraint found by probing PR #818 rather than
reading its diff: serde deny_unknown_fields closes the config-typo trigger but
makes rivet unusable in sigil, whose rivet.yaml carries a top-level schemas-path
rivet has always ignored. Downgrading it to a warning reopens the trigger,
because the empty-source check iterates over configured sources and a mis-keyed
sources leaves none. Confirmed with a build, PASS and 0 warnings even under
--strict. Needs three parts, not one.

REQ-295 records that its two defects must be fixed in order — closing the
stub-satisfiability hole first would turn 14 false failures into hard ones.

Deferred REQ-281, REQ-282, REQ-285 and REQ-286 from v0.34.0 to v0.35.0. This is
a scope decision, logged rather than silent: v0.34.0 leads with the three
customer UI requirements that have already slipped two releases, plus the write
path work the last two days made urgent.

Refs: REQ-294, REQ-295, REQ-296, REQ-297, REQ-298, REQ-299
avrabe pushed a commit that referenced this pull request Aug 20, 2026
…e case-2 gap (#808)

Maintainer review on #818 caught two issues in the initial revision:

1. **`deny_unknown_fields` on `ProjectConfig` broke sigil** — sigil's
   `rivet.yaml` legitimately carries a top-level `schemas-path:` that
   rivet has always silently ignored. Hard-failing on it made rivet
   unusable in the org's most active downstream (1242 artifacts). Same
   silent-load failure direction as #808, pointed the other way:
   silently accepts nothing → loudly rejects what worked yesterday.

2. **The naive downgrade of Fix 3 reopens case 2.** Simply dropping
   `deny_unknown_fields` and relying on the existing `empty-source`
   diagnostic doesn't close the reproducer: when `sources:` is mis-keyed
   as `typo_sources:`, the config has ZERO configured sources — so the
   loop that emits `empty-source` never runs, and `validate --strict`
   still exits 0 over silently green output.

1. **Drop `#[serde(deny_unknown_fields)]` from `ProjectConfig`**. sigil
   and every other downstream with legitimate extra top-level keys
   loads cleanly again.

2. **New `unknown-config-key` diagnostic**. `load_project_config_with_report`
   in rivet-core parses the file once as `ProjectConfig` and once as a
   plain `Value`, then reports any top-level keys not in the exported
   `KNOWN_RIVET_YAML_TOP_LEVEL_KEYS` slice (kept alphabetized so a drift
   between the struct and the slice is obvious in a diff). `cmd_validate`
   consumes the unknown-keys list from `ProjectContext` and emits one
   `unknown-config-key` Warning per unknown key, naming the offending
   key AND listing every valid alternative — same helpful shape the
   original hard-fail parse error had, minus the breaking behavior.
   Warning by default, Error under `--strict`.

3. **New `no-sources` diagnostic**. Fires when `config.sources.is_empty()`
   — independent of the per-source loop, so it catches the exact case-2
   gap the maintainer's probe demonstrated (`typo_sources:` leaves
   `sources:` defaulted to `[]`, per-source loop iterates nothing, nothing
   else fires). Also closes the plain `sources: []` case that nothing
   covered before. Warning by default, Error under `--strict`.

Every other #808 defense from the first commit is unchanged:
`artifact-root-key-near-miss`, `empty-source`, `coverage` rendering
`n/a` instead of `100.0%`, the `--strict-empty` flag, `--fail-under`
failing on empty load, the `empty_scope: bool` JSON field, and the
`is_empty_scope()` helpers on `CoverageEntry`/`ClosureEntry`/`CoverageReport`.

- `rivet_yaml_unknown_top_level_key_hard_fails` **renamed** to
  `rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict`.
  Uses a sigil-style reproducer (`schemas-path: schemas` alongside a
  real `sources:` block) and asserts default validate PASSes with the
  diagnostic visible, while `--strict` exits 1. Locks the sigil
  compatibility guarantee.
- **New**: `typoed_sources_key_fires_no_sources_diagnostic`. The exact
  case-2 reproducer from the issue (`typo_sources:` where `sources:`
  was meant). Asserts default validate surfaces `no-sources` and
  `--strict` exits 1. This is the test that would have caught the
  naive-downgrade regression the maintainer flagged; it now runs on
  every CI push.

Added the `## [Unreleased]` entry — Fixed section describing all four
diagnostics, Changed section calling out the potentially-breaking JSON
shape (`percentage: null` for empty scope, new `empty_scope` field) and
text shape (`n/a%` where `100.0%` used to appear) for machine consumers.

- `cargo build -p rivet-cli` — clean.
- `cargo test -p rivet-cli --test cli_commands` — **167 pass** (162
  pre-existing + 5 #808 regressions).
- `cargo test -p rivet-core --lib` — **1186 pass**.
- `cargo fmt --all -- --check` — clean.
- `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` —
  clean (only pre-existing MSRV note).
- `rivet validate` on this repo — PASS (638 warnings; the 3-warning bump
  vs previous is REQ-294 having no incoming `verifies` links yet, as
  expected for a just-authored requirement).
- `rivet get REQ-294` — resolves; artifact loads cleanly.
- Manual smoke on the sigil-style reproducer (`schemas-path:` +
  real sources): default `validate` exit=0 with `unknown-config-key`
  warning; `--strict` exit=1. Rivet is usable in sigil again.
- Manual smoke on the case-2 reproducer (`typo_sources:`): default
  `validate` surfaces `no-sources` warning; `--strict` exit=1. Case 2
  no longer silently green.

Fixes: REQ-294
Verifies: REQ-294
@avrabe
avrabe force-pushed the fix/issue-808-silent-empty-load branch from 6ce0e71 to fb52939 Compare August 20, 2026 05:12
@avrabe

avrabe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Unblocking this — my objection is resolved, and I found one more gap

I blocked this PR earlier over the sigil compatibility break. That objection
is now satisfied and locked by a test.
Rebased onto main (ebd253a) and
verified the three-part shape works. Detail below, plus one change I made.

The sigil constraint holds

sigil-shaped rivet.yaml (top-level `schemas-path:`)   exit 0   PASS

It loads, emits unknown-config-key as a warning, and does not fail. That was
the whole reason for softening Fix 3, and
rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict already
asserts the exit code, so it can't silently regress.

The gap: --strict-only escalation is inert where #808 was found

All four new diagnostics landed Warning-by-default / Error-under---strict.
Checked what actually runs in this repo:

  • ci.yml:299 Traceability — cargo run --release -p rivet-cli -- validate
  • ci.yml:379 hosted floor — same, no --strict
  • release.yml:292 — same

ci.yml:362 calls Traceability "the single most load-bearing gate". So on
main as this PR stood:

mis-keyed `typo_sources:`  ->  Result: PASS (2 warnings)   exit 0

Zero artifacts loaded, two warnings printed, exit 0. That is #808 with
warnings printed over the top — the gate reports green on precisely the
condition it was just taught to detect. Pipelines read exit codes, not prose.

What I changed: no-sources is now Error by default

Only that one of the four. The asymmetry is deliberate — the other three each
have a legitimate case, and no-sources does not:

diagnostic default legitimate case
empty-source Warning fresh project, source not yet populated
unknown-config-key Warning sigil's schemas-path:
artifact-root-key-near-miss Warning heuristic, can misfire
no-sources Error none — rivet init always scaffolds sources:

Verified that last claim rather than assuming it: a fresh rivet init validates
PASS (0 warnings), exit 0, so this cannot fire on the happy path.

Controls after the change:

sigil-shaped config     exit 0   PASS   <- compat held
mis-keyed typo_sources: exit 1   FAIL   <- gate now bites
fresh `rivet init`      exit 0   PASS   <- happy path unaffected

The existing test could not have caught this

typoed_sources_key_fires_no_sources_diagnostic asserted the message appeared
and that --strict failed — but never the default exit code, so it passed
identically before and after. Added that assertion. It's the load-bearing one,
and without it the severity can be flipped back silently.

Two CHANGELOG defects repaired

  1. Entries were filed under ## [0.33.0], which already shipped (0.33.1 is
    out). Moved to [Unreleased].
  2. A duplicate ## [0.33.0] heading dated 2026-08-11 had been added
    alongside the real 2026-08-18 one. Removed; the diff against main is now
    purely additive.

Also documented the exit-code change under "Changed (potentially breaking)" —
it is user-visible and deserves to be findable.

REQ-294 rebase note

Both sides carried a REQ-294; keep-both would have produced a duplicate id.
Resolved to one artifact keeping main's release: v0.34.0 (the branch's
copy had no release: field, which would have dropped it out of the v0.34.0
readiness query) and the branch's description + implemented status, which
describes what actually shipped.

Gates (exit codes captured, not read off a piped tail)

  • cargo fmt --all -- --check — exit 0
  • cargo test --workspace — exit 0, 65 test result: ok, 0 FAILED
  • rustup run 1.97.0 cargo clippy --all-targets -- -D warnings — exit 0
  • rivet validate — exit 0, PASS
  • rivet docs check — exit 0, PASS

Ready to merge once CI is green.

avrabe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Two things you got right that I got wrong; noting them for the record.

no-sources default. You're right that Warning was the wrong choice. My reasoning was "warn-by-default matches Fixes 1/2 for symmetry," but the case-2 whole point is exit-code fatality on the mis-keyed-config state, and Warning defaulted right back into the silent-green shape #808 exists to remove. Your "there is no legitimate zero-sources case" argument holds cleanly: rivet init scaffolds sources:, sources: [] is a config bug in every real project, and the other three do have legitimate defaults where mine didn't. Also, the regression I wrote asserted the message text and the --strict exit — but not the default exit — so it would have accepted a silent regression on that severity. Both defects, mine.

CHANGELOG placement. The section I intended to write into was [Unreleased]; what you saw was the rebase collapsing my entries into a shipped [0.33.0] section after 0.33.1 came out mid-work. Good catch; my Unreleased-vs-shipped placement should have been re-checked at rebase time, not inherited.

Standing down — thanks for unblocking. All eyes on CI.


Generated by Claude Code

avrabe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 1455538 — mutant-killer tests for 7 of the 8 diff-scoped survivors flagged by the advisory Mutation Testing (rivet-core, PR-diff) check.

Each killer test targets a mutant-equivalence class rather than restating what the fix already asserts. My original tests exercised only the true side of each is_empty_scope boundary, so a mutation returning that side unconditionally looked identical to the fix — that's the pattern that lets 5 of the 8 mutants survive. The new tests exercise the opposite side too.

site mutation killed by
coverage.rs:107 CoverageEntry::is_empty_scope replace with true non_empty_entries_and_report_are_not_empty_scope
coverage.rs:171 ClosureEntry::is_empty_scope replace with true closure_entry_is_empty_scope_reads_total_correctly
coverage.rs:171 ClosureEntry::is_empty_scope replace with false same (empty side of the pair)
coverage.rs:171 ClosureEntry::is_empty_scope ==!= same (both sides needed)
coverage.rs:200 CoverageReport::is_empty_scope replace with true non_empty_entries_and_report_are_not_empty_scope
lib.rs:258 load_project_config_with_report delete ! in unknown-keys filter load_report_lists_unknown_keys_and_omits_every_known_key
generic.rs:356 classify_skip ==!= in map.len() == 1 classify_skip_near_miss_key_requires_single_root_key

The test_schema in coverage.rs assigns one rule per source type, so v_closure() returns nothing and ClosureEntry::is_empty_scope is never called — that's also the codecov "3 uncovered lines in coverage.rs" surface. closure_entry_is_empty_scope_reads_total_correctly builds a schema with two rules on requirement so v_closure() actually returns an entry.

Not fixed: generic.rs:270 && → || in import_generic_directory's log-emission guard. Both operators produce the same value when warn_skips == surface; the observable difference is whether log::warn! fires, which needs a log-capture harness the crate doesn't have. Adding one for a single mutant in an advisory-mode check isn't worth it; flagged so the residue is honest rather than claiming zero survivors. If it needs to die for promotion-mode, that's a small module addition — happy to do it if you want.

cargo test -p rivet-core --lib 1202 pass (1198 pre-existing + 4 new). Fmt + clippy clean.


Generated by Claude Code

@avrabe
avrabe force-pushed the fix/issue-808-silent-empty-load branch from 1455538 to 4d8a37e Compare August 20, 2026 09:18
@avrabe

avrabe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Two more findings pushed to this branch

1. Six rivet: verifies markers were invisible to the trace graph

The scanner regex is //\s*rivet:\s*(verifies|partially-verifies)\s+([\w-]+).
# is not a word character, so /// rivet: verifies #808 does not match at
all
— it isn't a broken link that something reports, it's simply not a marker.

Checked against the regex itself rather than by eye:

'/// rivet: verifies REQ-294'  -> REQ-294
'/// rivet: verifies #808'     -> NO MATCH
'/// rivet: verifies #812'     -> NO MATCH

Six were in that state: five #808 added by this branch, one #812 already on
main. 593 other markers in the tree use artifact ids — these were the
outliers. Repointed to REQ-294 / REQ-298.

Effect, measured with rivet coverage --tests:

before:  REQ-294 and REQ-298 in "Uncovered (207)"
after:   REQ-294  5 test markers
         REQ-298  1 test marker

That is the difference between "the PR merged" and "the V is closed" — and it
mattered here, because without it these requirements could not have been flipped
to verified no matter how good the tests were.

2. Re-checked the severity change across every command, not just validate

I changed a default severity, so validate alone wasn't sufficient evidence.
Comparing this branch against main on the same mis-keyed-sources: fixture:

command main this PR
validate 0 1 ← the no-sources change
coverage --fail-under 50 0 1 ← this branch's empty-load fix
coverage 0 0
list 0 0
check verification-evidence 0 0

Exactly two exit-code changes, both intended, both in the CHANGELOG. On the
sigil-shaped fixture (schemas-path: + real sources) validate, coverage
and list all still exit 0 — compatibility holds across commands, not just the
one I originally tested.

(An earlier run of this table showed spurious exit=2 values for the two
multi-word commands. That was a shell-quoting artifact — the whole string was
passed as one argv element and clap rejected it as an unknown subcommand. Re-run
correctly above; there is no third behaviour change.)

Gates after both changes

cargo fmt --check exit 0 · cargo test -p rivet-cli --test cli_commands exit 0,
170 passed / 0 failed · rivet validate exit 0 · rivet docs check exit 0.

avrabe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Marker regex — that's my error, and worth flagging as one to learn from. I typed // rivet: verifies #808 on both PRs assuming it would round-trip to a GitHub issue reference; the scanner takes [\w-]+ which # isn't in, so all six markers were silently no-ops. Read as an artifact-ID reference but resolved to nothing, and the trace graph had no way to tell me. The lesson is the one #808 itself is about: an evidence path that looks correct and produces no diagnostic is exactly the shape that hides. Good catch and the right repoint to REQ-294 / REQ-298.

Cross-command exit-code table is the right shape of check for a severity change — I only ran validate when I flipped no-sources, so I would not have noticed if list or coverage had picked up an unintended propagation. Two intended changes, three sigil-compat holds, nothing else — that's the check I should have written.

Standing down; thanks for driving this to green.


Generated by Claude Code

claude and others added 4 commits August 20, 2026 11:21
…prints n/a not 100% (#808)

For a compliance tool, "loudest-green over a zero read" is the worst
possible failure direction. The reporter demonstrated two independent
triggers, both yielding **0 artifacts** while every downstream command
reported success:

1. A `generic-yaml` source file with `requirements:` at the root
   (instead of `artifacts:`) — silently classified as a non-artifact
   file and dropped.
2. A mis-keyed `typo_sources:` in `rivet.yaml` — silently ignored,
   because top-level keys weren't gated.

In both cases `rivet validate` printed `PASS` and `rivet coverage`
printed `100.0%` for every rule (0/0 denominator). This closes all
four fix sites the issue calls out.

`cmd_validate` already collects per-source load reports for its
`artifact-parse-error` and `duplicate-artifact-id` diagnostics. Extended
the same loop to note any source that returned zero artifacts, and to
emit an `empty-source` diagnostic pointing at the offending
`sources[].path`. Suppressed when a `ParseError` or `NearMissKey` diag
already names the cause, so the user isn't told twice about the same
file.

Severity: Warning by default (so pre-existing projects don't break
their own dogfood `validate`), Error under `--strict` (compliance-gate
mode), mirroring the `--strict-orphans` / `allowed-values` pattern.

Extended `SkipKind` in `rivet-core/src/formats/generic.rs` with a
`NearMissKey(String)` variant, and `classify_skip` now catches
single-key files whose root is a plausible artifact-plural typo
(`requirements`, `features`, `designs`, `decisions`, `design-decisions`,
`tests`, `verifications`). `cmd_validate` emits it as
`artifact-root-key-near-miss` naming both the file and the key, so
`did you mean `artifacts:`?` is answered in the diagnostic message
itself. Same Warning-by-default / Error-under-`--strict` treatment.

Added `#[serde(deny_unknown_fields)]` to `ProjectConfig` in
`rivet-core/src/model.rs`. A `typo_sources:` (or any other unknown top-
level key) is now a load-time parse error naming exactly what's wrong:

```
error: loading ./rivet.yaml: Schema error: ./rivet.yaml:
  unknown field `typo_sources`, expected one of `project`, `sources`,
  `docs`, `results`, `commits`, `release`, `externals`, `baselines`,
  `docs-check` at line 4 column 1
```

This is stricter than the other two fixes because there's no
legitimate reason for a rivet.yaml to carry unknown top-level keys
(unlike an artifact file, where a `requirements:` file might really
be non-artifact data placed in a source path). Sibling structs in
model.rs (source, external, release configs) already had
`deny_unknown_fields` — this is closing the top-level hole.

Added `is_empty_scope()` helpers on `CoverageEntry`, `ClosureEntry`,
and `CoverageReport`. `cmd_coverage`'s renderer now consults them:

- Text output: `n/a%` for empty-scope rules (column width preserved
  so alignment survives), plus a leading banner —
  `⚠  No artifacts loaded — every rule scores n/a (0/0)` — pointing
  at the fix and `--strict-empty`. The legacy `100.0%` for empty-
  scope is gone.
- JSON output: per-rule `percentage` and `accounted_percentage` are
  `null` for empty scope; every rule and the `overall` block gain an
  `empty_scope: bool` field so a machine consumer can distinguish a
  satisfied gate from an empty one without inspecting `total`.
- New `--strict-empty` flag on `coverage`: exits non-zero when the
  whole report is empty-scope. Default off (backwards-compatible);
  turn on in CI as the gate the issue calls for.
- `--fail-under` on an empty load ALSO exits non-zero (the historical
  100% would have trivially satisfied any threshold — that's the
  silent-green trap the flag exists to catch).

The `percentage()` / `overall_coverage()` / `accounted_percentage()`
methods keep their old `f64` signatures + legacy 100.0-on-empty
values for the ~40 other callers (renderers in `export.rs`,
`render/coverage.rs`, `serve/api.rs`, `snapshot.rs`, `mcp.rs`,
`proofs.rs`, `embed.rs`, `stats.rs`, `coverage_evidence.rs`) —
their doc-comments now instruct callers to check `is_empty_scope()`
first, so the empty-scope distinction is available everywhere
without a breaking signature change.

Four end-to-end tests in `rivet-cli/tests/cli_commands.rs`, one per
fix:

- `rivet_yaml_unknown_top_level_key_hard_fails` — writes a project
  with `typo_sources:`, asserts `validate` exits non-zero and stderr
  names the offending key.
- `validate_flags_generic_yaml_near_miss_root_key` — writes a
  `requirements:`-rooted file, asserts default validate surfaces
  `artifact-root-key-near-miss` and `--strict` fails.
- `validate_flags_empty_source` — writes an empty source directory,
  asserts default validate surfaces `empty-source` and `--strict`
  fails.
- `coverage_prints_n_a_and_exits_nonzero_on_empty_load` — asserts
  text renders `n/a%` and the empty-load banner (and NEVER `100.0%`);
  JSON `overall.empty_scope=true` and `percentage=null`; `--strict-empty`
  and `--fail-under` both exit non-zero.

Plus the existing `zero_artifacts_gives_100_percent` unit test in
`coverage.rs` was renamed / updated to
`zero_artifacts_is_flagged_as_empty_scope_not_a_100_percent_pass` —
it now asserts BOTH the new `is_empty_scope()` invariant AND that
the legacy 100.0 return is preserved for downstream callers.

- `cargo test -p rivet-core --lib` — **1186 pass** (unchanged count,
  the coverage test was renamed rather than added).
- `cargo test -p rivet-cli --test cli_commands` — **166 pass** (162
  pre-existing + 4 new).
- `cargo fmt --all -- --check` — clean.
- `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings`
  — clean (only pre-existing MSRV note).
- `rivet validate` on this repo — PASS (635 warnings), same as before
  the diff. No new diagnostics against the rivet self-project because
  its `sources:` list resolves cleanly and every configured source
  produces artifacts.

- Not a change to any percentage method's return type. The 40+
  downstream callers (HTML export, JSON snapshot, MCP tool, dashboard
  serve, embed engine, proof reporter, coverage evidence, ...) all
  keep working unchanged; the empty-scope distinction is available
  via a NEW `is_empty_scope()` helper they can adopt incrementally.
- Not a change to `--strict`'s existing semantics for `allowed-values`
  / `unknown-field`. Just adds the two new `--strict`-promoted rules
  (`artifact-root-key-near-miss`, `empty-source`) to the existing
  escalation block.
- Not a change to the `NotArtifactFile` classification for files
  the user legitimately places in a source path (`bindings.yaml`,
  `feature-model.yaml`, `variants/*.yaml`). Only single-key files
  whose root reads as a plausible artifact-plural typo are reclassified.

Fixes: REQ-004
Verifies: REQ-004
…e case-2 gap (#808)

Maintainer review on #818 caught two issues in the initial revision:

1. **`deny_unknown_fields` on `ProjectConfig` broke sigil** — sigil's
   `rivet.yaml` legitimately carries a top-level `schemas-path:` that
   rivet has always silently ignored. Hard-failing on it made rivet
   unusable in the org's most active downstream (1242 artifacts). Same
   silent-load failure direction as #808, pointed the other way:
   silently accepts nothing → loudly rejects what worked yesterday.

2. **The naive downgrade of Fix 3 reopens case 2.** Simply dropping
   `deny_unknown_fields` and relying on the existing `empty-source`
   diagnostic doesn't close the reproducer: when `sources:` is mis-keyed
   as `typo_sources:`, the config has ZERO configured sources — so the
   loop that emits `empty-source` never runs, and `validate --strict`
   still exits 0 over silently green output.

1. **Drop `#[serde(deny_unknown_fields)]` from `ProjectConfig`**. sigil
   and every other downstream with legitimate extra top-level keys
   loads cleanly again.

2. **New `unknown-config-key` diagnostic**. `load_project_config_with_report`
   in rivet-core parses the file once as `ProjectConfig` and once as a
   plain `Value`, then reports any top-level keys not in the exported
   `KNOWN_RIVET_YAML_TOP_LEVEL_KEYS` slice (kept alphabetized so a drift
   between the struct and the slice is obvious in a diff). `cmd_validate`
   consumes the unknown-keys list from `ProjectContext` and emits one
   `unknown-config-key` Warning per unknown key, naming the offending
   key AND listing every valid alternative — same helpful shape the
   original hard-fail parse error had, minus the breaking behavior.
   Warning by default, Error under `--strict`.

3. **New `no-sources` diagnostic**. Fires when `config.sources.is_empty()`
   — independent of the per-source loop, so it catches the exact case-2
   gap the maintainer's probe demonstrated (`typo_sources:` leaves
   `sources:` defaulted to `[]`, per-source loop iterates nothing, nothing
   else fires). Also closes the plain `sources: []` case that nothing
   covered before. Warning by default, Error under `--strict`.

Every other #808 defense from the first commit is unchanged:
`artifact-root-key-near-miss`, `empty-source`, `coverage` rendering
`n/a` instead of `100.0%`, the `--strict-empty` flag, `--fail-under`
failing on empty load, the `empty_scope: bool` JSON field, and the
`is_empty_scope()` helpers on `CoverageEntry`/`ClosureEntry`/`CoverageReport`.

- `rivet_yaml_unknown_top_level_key_hard_fails` **renamed** to
  `rivet_yaml_unknown_top_level_key_warns_and_escalates_under_strict`.
  Uses a sigil-style reproducer (`schemas-path: schemas` alongside a
  real `sources:` block) and asserts default validate PASSes with the
  diagnostic visible, while `--strict` exits 1. Locks the sigil
  compatibility guarantee.
- **New**: `typoed_sources_key_fires_no_sources_diagnostic`. The exact
  case-2 reproducer from the issue (`typo_sources:` where `sources:`
  was meant). Asserts default validate surfaces `no-sources` and
  `--strict` exits 1. This is the test that would have caught the
  naive-downgrade regression the maintainer flagged; it now runs on
  every CI push.

Added the `## [Unreleased]` entry — Fixed section describing all four
diagnostics, Changed section calling out the potentially-breaking JSON
shape (`percentage: null` for empty scope, new `empty_scope` field) and
text shape (`n/a%` where `100.0%` used to appear) for machine consumers.

- `cargo build -p rivet-cli` — clean.
- `cargo test -p rivet-cli --test cli_commands` — **167 pass** (162
  pre-existing + 5 #808 regressions).
- `cargo test -p rivet-core --lib` — **1186 pass**.
- `cargo fmt --all -- --check` — clean.
- `cargo clippy -p rivet-cli -p rivet-core --all-targets -- -D warnings` —
  clean (only pre-existing MSRV note).
- `rivet validate` on this repo — PASS (638 warnings; the 3-warning bump
  vs previous is REQ-294 having no incoming `verifies` links yet, as
  expected for a just-authored requirement).
- `rivet get REQ-294` — resolves; artifact loads cleanly.
- Manual smoke on the sigil-style reproducer (`schemas-path:` +
  real sources): default `validate` exit=0 with `unknown-config-key`
  warning; `--strict` exit=1. Rivet is usable in sigil again.
- Manual smoke on the case-2 reproducer (`typo_sources:`): default
  `validate` surfaces `no-sources` warning; `--strict` exit=1. Case 2
  no longer silently green.

Fixes: REQ-294
Verifies: REQ-294
…#808)

The four new #808 diagnostics all landed Warning-by-default, Error under
--strict. That leaves the gate inert where it matters: this repo's own
Traceability job — described in ci.yml as the single most load-bearing gate —
runs plain `rivet validate` with no --strict, as does the hosted-floor mirror
and release.yml. A mis-keyed `sources:` therefore still produced exit 0 over
zero loaded artifacts, which is the #808 defect itself with warnings printed
over the top.

no-sources is the one of the four with no legitimate case, so it is the one
that changes. The others keep Warning-by-default deliberately: a fresh project
whose source is not yet populated (empty-source), a downstream repo carrying
its own top-level keys such as sigil's schemas-path: (unknown-config-key), and
a heuristic that can misfire (artifact-root-key-near-miss).

Three negative controls, judged on exit status:

  sigil-shaped config (top-level schemas-path:)  exit 0  PASS   <- compat held
  mis-keyed typo_sources:                        exit 1  FAIL   <- gate bites
  fresh `rivet init`                             exit 0  PASS   <- happy path

The existing test asserted the diagnostic text appeared and that --strict
failed, but never asserted the default exit code — so it passed identically
before and after this change. Added that assertion; it is the load-bearing one.

Also repairs two CHANGELOG defects on this branch: the entries were filed under
the already-shipped [0.33.0] section, and a duplicate `## [0.33.0]` heading
dated 2026-08-11 had been introduced alongside the real 2026-08-18 one. Entries
moved to [Unreleased]; the diff against main is now purely additive.

REQ-294 kept its `release: v0.34.0` through the rebase — the branch's copy had
dropped the field, which would have made the artifact invisible to the v0.34.0
readiness query.

Confirmed with cargo fmt --check, cargo test --workspace (exit 0, 65 ok
lines), clippy --all-targets -D warnings on 1.97.0, rivet validate, and
rivet docs check — all exit 0.

Implements: REQ-294
Fixes: REQ-004
Refs: #808
…ssue numbers

The test-marker scanner matches `//\s*rivet:\s*(verifies|partially-verifies)\s+([\w-]+)`.
`#` is not a word character, so `/// rivet: verifies #808` does not match at all
— the marker is silently invisible to the trace graph rather than being a
broken link that something reports.

Six markers were in that state: five `#808` added by this branch and one `#812`
already on main. 593 other markers in the tree use artifact ids; these were the
outliers.

Verified against the actual regex rather than by inspection:

  '/// rivet: verifies REQ-294'  -> REQ-294
  '/// rivet: verifies #808'     -> NO MATCH
  '/// rivet: verifies #812'     -> NO MATCH

And after the change, `rivet coverage --tests` reports REQ-294 with 5 test
markers and REQ-298 with 1, where both previously sat in the uncovered list.

This is what makes the V closable for these two requirements: a merged PR is
evidence of `implemented`, but `verified` needs a test that actually names the
artifact. REQ-297 and REQ-302 remain in the uncovered list for the same reason
and are deliberately not touched here.

Verifies: REQ-294
Refs: REQ-298
@avrabe
avrabe force-pushed the fix/issue-808-silent-empty-load branch from 4d8a37e to 4f221ae Compare August 20, 2026 09:23

avrabe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Small housekeeping observation before merge: my 1455538 (the mutant-killer tests) doesn't appear to be on the current branch tip after the marker-fix force-push — git log origin/fix/issue-808-silent-empty-load shows 79b9fd4 · afafaae · 03d0861 · 4f221ae, and grepping the current tree for non_empty_entries_and_report_are_not_empty_scope, closure_entry_is_empty_scope_reads_total_correctly, load_report_lists_unknown_keys_and_omits_every_known_key, classify_skip_near_miss_key_requires_single_root_key finds 0 matches.

If that's deliberate (advisory check ⇒ not worth carrying), no action needed. If it was accidental during the marker-fix rebase, I have the commit locally (unchanged from what merged the codecov jump from 90.32% → 98.05%) and can cherry-pick it back onto the current tip. Which would you prefer?

Doesn't block merge either way — the mutation gate is advisory, and the codecov threshold was met without them too. Just wanted to surface it in case the drop wasn't intentional.


Generated by Claude Code

@avrabe
avrabe merged commit b3bc0d3 into main Aug 20, 2026
31 of 34 checks passed
@avrabe
avrabe deleted the fix/issue-808-silent-empty-load branch August 20, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent empty load: a config typo yields 0 artifacts, validate PASSes and coverage reports 100.0%

2 participants