plan(v0.58): add RQ-58-SHIPVERIFY (#1000) — we ship a verifier nobody can run - #1001
Merged
Conversation
… can run #1000, from a toolchain-wide CLI survey: `synth verify` exists in the tree and in `--help`, fails closed with an exemplary message — and is in NO artifact a user can obtain. Reproduced by the reporter on the layer build (0.55.0) AND the current release (0.57.0), whose four platform tarballs all lack the feature. THE ISSUE'S OWN ESCAPE HATCH IS DEAD, AND I MEASURED IT RATHER THAN ASSUMING. #1000 offers: "or, if the Z3 dependency makes that undesirable for the default artifact, publish a variant". There is no Z3 dependency: cargo tree -p synth-cli --features verify -> 0 z3 nodes cargo tree -p synth-cli --features verify,…/z3-solver -> 2 z3 nodes `verify = ["synth-verify"]` and nothing more. `z3-solver = ["z3"]` is a separate opt-in with `optional = true`, and even enabled it links the SYSTEM libz3 rather than bundling it (#553). Ordeal (pure Rust QF_BV) has been the default engine since v0.27.0. Shipping `--features verify` costs no Z3, no libz3, no C++ toolchain — so this is a release-workflow change, not a trade-off. README.md:98 IS THE OTHER HALF OF THE FINDING, and it is corrected here. It said the CLI `verify` feature "currently also enables the feature-gated Z3 differential oracle (statically linked)". False since v0.27.0 — and it is exactly the sentence that would make a maintainer accept #1000's escape hatch and NOT ship the feature. Prose that was true once, quietly stopped being, and stayed load-bearing on a release decision: the class the v0.57 cold review kept turning up, found again by a user asking a good question about a stale premise. Artifact scope, three parts, with the middle one carrying the weight: 1. build the released binaries (four tarballs + npm) with `--features verify` 2. a NON-VACUITY GATE — a released-artifact smoke test that actually RUNS `synth verify` on a freshly compiled module and asserts a VERDICT, not the capability-missing error. Red-first against a binary built without the feature. Without this, "we shipped the feature" regresses silently to "we shipped the help text", which is the class this whole release closes. 3. move the capability check AHEAD of the banner — today the four `Translation validation:` lines including `Strategy: Per-rule SMT verification (ASIL D path)` print BEFORE the tool discovers it cannot verify, so a log-scraper finds the ASIL-D line in a run that verified nothing. Exit code is already correct. Out of scope, named not dropped: #1000's `synthesize`-vs-`compile` description overlap, `--format json`, help-line width. Also re-graded RQ-58-MIRRORS (#993) and RQ-58-SELECT973 (#992) proposed -> implemented; both merged and the ledger had not caught up. rivet: 50 errors before AND after (unchanged); warnings +2, the standard pair every artifact carries. claim_check 49/49. Refs #1000, #242 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds RQ-58-SHIPVERIFY for #1000, and corrects the stale README sentence that kept the issue's escape hatch alive.
The problem
synth verifyexists in the tree and in--help, and fails closed with an exemplary message — but it is in no artifact a user can obtain. The reporter confirmed it on the layer build (0.55.0) and the current release (0.57.0), whose four platform tarballs all lack the feature.For a compiler whose entire argument is verified code generation, the ASIL-D translation-validation path being unobtainable is the gap worth closing first. It is also uncomfortably on-theme for v0.58: we ship a verifier nobody can run.
The escape hatch is dead — measured, not assumed
#1000 offers "or, if the Z3 dependency makes that undesirable for the default artifact, publish a variant". There is no Z3 dependency:
cargo tree -p synth-cli --features verify... --features verify,synth-verify/z3-solververify = ["synth-verify"]and nothing more.z3-solver = ["z3"]is a separate opt-in withoptional = true, and even enabled it links the system libz3 rather than bundling it (#553). Ordeal (pure Rust QF_BV) has been the default engine since v0.27.0.So shipping
--features verifycosts no Z3, no libz3, no C++ toolchain — a release-workflow change, not a trade-off.README:98 is the other half of the finding
It said the CLI
verifyfeature "currently also enables the feature-gated Z3 differential oracle (statically linked)". False since v0.27.0 — and precisely the sentence that would make a maintainer accept the escape hatch and not ship the feature.Prose that was true once, quietly stopped being, and stayed load-bearing on a release decision. Same class the v0.57 cold review kept surfacing; found this time by a user questioning a stale premise. Corrected here.
Scope, with the weight in the middle
--features verify.synth verifyon a freshly compiled module and asserts a verdict, not the capability-missing error. Red-first against a binary built without the feature. Without it, "we shipped the feature" regresses silently to "we shipped the help text".Strategy: Per-rule SMT verification (ASIL D path)prints before the tool discovers it cannot verify, so a log-scraper finds the ASIL-D line in a run that verified nothing. Exit code is already correct.Out of scope, named: #1000's
synthesize-vs-compileoverlap,--format json, help-line width.Also re-grades RQ-58-MIRRORS (#993) and RQ-58-SELECT973 (#992) →
implemented; both merged and the ledger had not caught up.Gates
rivet 50 errors before and after (unchanged); warnings +2, the standard pair every artifact carries.
claim_check49/49.Refs #1000, #242