feat(#1000): ship 'synth verify' in released artifacts + non-vacuity smoke gate (RQ-58-SHIPVERIFY) - #1002
Merged
Merged
Conversation
…ategy line from a binary that cannot verify Red-first: tests/verify_banner_1000.rs (no required-features) fails on the pre-fix ordering — the incapable binary printed all four 'Translation validation:' lines including 'Strategy: Per-rule SMT verification (ASIL D path)' before bailing. Exit code was already correct (#124); now nothing verification-shaped precedes the failure. The verify half (wired into fact-spec-oracle) pins that a capable binary still prints the banner and ends in a real verdict. Also de-stales the 'via Z3' help line (measured: 0 z3 nodes under --features verify). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
…ity smoke gate (RQ-58-SHIPVERIFY) release.yml: all four platform builds now pass --features verify (pure-Rust ordeal since #553 — re-measured: 0 z3 nodes; the z3 differential oracle stays a separate synth-verify/z3-solver opt-in). The npm channel inherits the feature: the @pulseengine/synth wrapper downloads and checksum-verifies these same tarballs at install time. Non-vacuity gate: scripts/release_verify_smoke.sh runs against the binary EXTRACTED FROM EACH PACKAGED TARBALL (aarch64-linux under qemu-user; the x86_64 macOS tarball under Rosetta on the arm64 runner) — compiles a module, runs 'synth verify' on it, and asserts a real verdict (exit 0, verdict line, synth-verify-v1 report with >=1 verified / 0 failed). Red-first, demonstrated at authoring AND pinned in CI: the fact-spec-oracle job builds a verify-less binary and asserts the script FAILS with exactly the capability error, then re-runs it green on the verify build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1000 (RQ-58-SHIPVERIFY).
synth verifyexisted in the tree and in--help, failed closed with an exemplary message — and was in no artifact any user could obtain (confirmed on 0.55.0 and 0.57.0: all four platform tarballs lacked--features verify). For a compiler whose whole argument is verified code generation, this ships the feature and gates it so it cannot silently regress to help text.1. Released binaries carry
verifyrelease.yml: all four platform builds (x86_64/aarch64Linux,x86_64/aarch64macOS) now build-p synth-cli --features verify; the stale "deliberate follow-up decision" comment is replaced with the measured basis.cargo tree -p synth-cli --features verify→ 0 z3 nodes;--features verify,synth-verify/z3-solver→ 2.verify = ["synth-verify"](pure-Rust ordeal since Adopt ordeal (pure-Rust QF_BV) in synth-verify; demote Z3 to differential oracle, then drop static-link-z3 #553/v0.27); the Z3 differential oracle remains a separate opt-in that links the system libz3 and is not enabled here.@pulseengine/synthwrapper downloads and checksum-verifies these same release tarballs at install time (npm/install.js), so it inherits the feature.cargo install synth-cliusers still choose features themselves (verifyis deliberately not made a default feature — that would change every workspace build; named residual, not silently dropped).2. The non-vacuity gate (the part that matters)
scripts/release_verify_smoke.shruns against the binary extracted from each packaged tarball (not a dev build): compiles a module with it, runssynth verifyon that module, and asserts a real verdict — exit 0, the verdict line, and asynth-verify-v1report whose summary provesverified >= 1 && failed == 0(counts, not sentence-greps). All four tarballs are executed: linux-x86_64 + both macOS natively (x86_64-darwin under Rosetta on the arm64 runner), the cross-built aarch64-linux tarball underqemu-aarch64 -L /usr/aarch64-linux-gnu.Red-first evidence (local, this branch):
And the red direction is CI-pinned, not authoring-time-only: the
fact-spec-oraclejob now builds a verify-less binary, asserts the smoke script FAILS with exactly the capability error (wrong-reason failures are also red), then re-runs it green on the verify build. A gate whose red leg is not exercised goes vacuous silently — this repo has shipped that class.3. Capability check before the banner
Previously the four
Translation validation:lines — includingStrategy: Per-rule SMT verification (ASIL D path)— printed before the tool discovered it could not verify, so a log-scraper found the ASIL-D strategy line in a run that verified nothing (exit code was already correct, #124). The check now runs before the banner: an incapable binary prints nothing verification-shaped before failing.Red-first:
tests/verify_banner_1000.rs(norequired-features; the missing-capability half runs in the plain workspace suite) fails on the pre-fix ordering — observed:ASIL-D strategy line printed by a binary that cannot verify— and passes with the fix. The#[cfg(feature = "verify")]half (wired intofact-spec-oracle) pins that a capable binary still prints the banner and ends inAll functions verified successfully..Also de-staled the
synth verifyhelp line ("via Z3" → SMT translation validation, pure-Rust ordeal) — measured above.Out of scope, named not silently dropped
#1000's
synthesize-vs-compiledescription overlap,--format json, help-line width (real, minor, not this lane's subject); makingverifya default cargo feature; README:98's stale Z3 sentence (corrected in #1001 — not duplicated here).Step 5 — witness MC/DC (#978 floors)
The reordered capability check adds one decision — in
synth-cli, which is outside the scored scope. The harness's full in-workspace closure issynth-mcdc-harness+synth-core/synth-cfg/synth-opt/synth-synthesis/synth-backend-riscv(percargo tree), andgit diff --name-only main...HEADintersects it in 0 files (Cargo.lock also unchanged, rustc pinned 1.96.1): the harness wasm this branch produces is byte-for-byte main's, so the counts cannot move.I re-derived locally anyway at rustc 1.96.1 and READ THE GAP ROWS rather than trusting the argument: every gap-row function is in the untouched crates (
validate_final_allocation_rv32,ensure_supported_target,compile_function_with_opts,static_data_addr::resolve_owner/validate_reloc_resolutions*) — none is this lane's code. Honest boundary on the local numbers: my localwitness-mcdcis 0.28.0, not the CI-pinned 0.42.0, and the local run reports 122 cond / 50 proved / 51 dead vs floors 130/57/≤50 — since the harness inputs are provably identical to main's, that local red is the unpinned local witness version measuring differently, not a delta from this change (it would reproduce identically on main). The PR'smcdc-structural-coveragejob (pinned witness 0.42.0, pinned host) is the authoritative floors gate — read its result, not my local one.Step 6 — sigil / attestation
Assessed, not defaulted to N/A: this PR changes what the released tarballs contain (a larger binary with the ordeal solver linked in). The existing attestation chain covers the new content without modification — SLSA provenance is generated per-release over
release-assets/*.tar.gz(subject digests are computed from the new bytes) and the cosign-signedSHA256SUMS.txttransitively covers them; the smoke step runs before upload/attestation and does not modify the archives. No new artifact type or build stage is introduced, and synth's native tarballs are not sigil-signed wasm (sigil is not part of this repo's release chain). Nothing to add.🤖 Generated with Claude Code
https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L