W1-CI-CLOSURE: SDLC v1.2 Institutional Compiler evidence controls - #33
Conversation
inventor1975
left a comment
There was a problem hiding this comment.
Requesting one change. The infrastructure hygiene here is good; the issue is in the falsification harness itself, and it is the same class of defect that Gate L hit in the sibling PR.
Verified rather than assumed:
- All three action pins resolve to the genuine upstream releases they claim:
actions/checkout@de0fac2e…= v6.0.2,actions/setup-python@a309ff8b…= v6.2.0,actions/upload-artifact@043fb46d…= v7.0.1. Checked against upstream tags, not against the comments. - Workflow-level
permissions: contents: readmeans the newadvisory-scanjob, which declares no permissions of its own, inherits least privilege rather than the repository default.persist-credentials: falseis set on the checkouts. - The dependency lock is genuine:
pip-compile --generate-hashesoutput, 990 hashes across 78 packages, and CI installs with--require-hashesfollowed bypip check.
Requested change — a SKIPPED falsification case is reported as observed.
scripts/falsify_infrastructure.py treats a non-zero pytest exit as the only failure. pytest exits 0 for a skipped test. I verified this by running it rather than reasoning about it: a @pytest.mark.skip test gives exit 0, while a stale selector gives exit 4 (that case you do catch).
So if any of the four cases becomes skipped — by an environment gate, a conditional marker, or a later refactor — the harness prints PASS <name> for it and then PASS bounded infrastructure falsification: 4/4 expected checks observed, and the completeness claim survives with nothing behind it. This is not hypothetical here: your own submission notes "one intentional missing-local-receipt case skipped", so skipping is a live behaviour in this suite.
The failure mode matters more than its size, because it is silent and it points the wrong way: the harness exists to make falsification visible, and in this state a case that stopped running looks exactly like a case that ran and held.
Cheapest reliable fix: assert the outcome, not just the exit code — require " 1 passed" in the pytest summary for each selector, so a skip, a deselect, or a collection change fails loudly.
Minor — 4/4 is a literal. The final line hardcodes the count rather than deriving it from len(CASES). Add a fifth case and the harness still reports 4/4. Same declared-versus-actual drift that PR #17's Gate L count just had to be corrected for; len(CASES) removes it permanently.
What I did NOT verify, so the boundary is explicit: I did not execute the suite, make verify, or make falsify; I did not audit the 78 pinned packages individually beyond confirming the lock is hash-generated and enforced; and the "1,228 tests passed" figure is taken from your submission, not from my own run.
Nothing here touches the semantic gate, which remains blocked as declared, and verify-manifest --all remaining INCOMPLETE with exit 3 reads as the stated state rather than a defect.
— Reviewed by Claude (Opus 5) on Vitaly Reznik's instruction; Vitaly is the human curator and owns the merge decision.
inventor1975
left a comment
There was a problem hiding this comment.
Approving. Both requested changes are addressed, and the main one is implemented more strictly than I asked for.
Verified by running it, not by reading it. I kept the exact pytest output from the skip case that motivated the finding — 1 skipped in 0.00s — and executed your new parsing logic against it. Result: summary parses to {'skipped': 1}, disallowed is non-empty, actual_single_pass returns False. The hole is closed on the very output that demonstrated it.
Requested change (a skipped falsification case reported as observed) — closed at the root. actual_single_pass now requires all of: exit zero, a parseable summary, passed == 1, and no other outcome present. skipped, deselected, failed, xfailed, xpassed and errors are all disallowed rather than merely unmentioned, so the harness cannot mistake a case that stopped running for a case that ran and held.
Two things you added that I did not ask for and that are right:
- Fail-closed on an unusable summary.
parse_pytest_summaryraising is caught and converted to refusal (unusable pytest summary), not to a pass. A harness that cannot read its own evidence now says so instead of shrugging. - Duplicate-outcome detection.
duplicate pytest summary outcomeguards against a malformed summary silently overwriting a counted outcome.
Minor (the hardcoded count) — closed. success_message derives the denominator from len(CASES), so adding a fifth case can no longer leave the report claiming 4/4.
What I did NOT verify, so the boundary stays explicit: I did not execute your suite, make verify or make falsify in your environment; I did not audit the pinned packages individually beyond confirming the lock is hash-generated and CI enforces --require-hashes; and test counts remain taken from your submission rather than from my own run.
Merge remains Vitaly's decision.
— Reviewed by Claude (Opus 5) on Vitaly Reznik's instruction; Vitaly is the human curator and owns the merge decision.
W1-CI-CLOSURE producer submission — Vitaliy requested-change correction
29daa374b7e5cdc30ca7788310fbabb85f19912b33d86a5284392e9e7729901e7352b4a83536faf9839b264acdf59c0fd906a3cc2a2dcc9dd4ffd4db3219d1f317760db254191331dcb0f0a7987edfceVitaliy requested-change disposition
Addressed for re-review on the new exact head. This does not claim that
Vitaliy’s review of
33d86a5…approves the corrected head.Pytest exit 0 alone is no longer accepted as falsification evidence. A bounded
stdlib-only parser requires exactly one unambiguous terminal summary and accepts
a case only when exit is 0,
passed == 1, and there are no failed, skipped,errored, xfailed, xpassed, or deselected outcomes. Missing, stale, multiple,
ambiguous, and unparsable outcomes fail closed.
The final count has no literal
4/4: observed cases are incremented only afterthe strengthened assertion, and the expected count is derived from
len(CASES). The resulting production message is dynamically4/4for thecurrent four selectors.
Harness self-test evidence
1 passed in 0.01sexpected selected=1 observed passed=11 skipped in 0.01s{'skipped': 1}pytest exit 4CASES2/2PASS bounded infrastructure falsification: 2/2 expected checks observedProduction falsification evidence
tests/unit/test_schemas.py::test_broken_schema_fails_metaschema_validationtests/contract/test_baseline.py::test_hash_mismatch_inside_a_synthetic_baseline_failstests/contract/test_semantic_conformance.py::test_reject_duplicated_reason_codestests/contract/test_claims_discipline.py::test_operator_guide_states_the_semantic_gate_is_blockedFinal dynamically generated result:
PASS bounded infrastructure falsification: 4/4 expected checks observed.Local verification
make verify— schema 9/9, bootstrap 52/52, manifest remainsINCOMPLETEwith required exit 3.make falsify— all four production selectors reported exactly one actual pass.python -m pytest -q— 1232 passed, 1 skipped. The count increased byfour harness self-tests; the one skip remains the intentional unavailable
gitignored local-receipt case.
python -m mypy src tests— success.ruff check .andruff format --check .— success.python -m pip_audit --skip-editable— no known vulnerabilities.pip check— success.git diff --check— clean.bound head/base synthetic merge composition resolved config, pulled
digest-pinned images, started all services, verified health, and tore down
successfully.
No dependency, Action pin, workflow permission, schema, semantic implementation,
license metadata, network behavior, telemetry, or manifest semantics changed.
Gate E–M producer matrix — canonical CURRENT-SDLC v1.2
INCOMPLETEbehavior, and the first executable objective. Semantic usability is not claimed.AGENTS.mdgives real infrastructure commands and prohibits semantic code-start and status mutation.No Gate E–M disposition changed. Semantic implementation remains BLOCKED.
Independent Adjudication remains pending for the designated reviewer and owner.
GitHub-native evidence
3219d1f317760db254191331dcb0f0a7987edfceand base29daa374b7e5cdc30ca7788310fbabb85f19912b, and executed GitHub's synthetic merge compositionb7fe7ed00fecff339f97c3929668f5a9a3065e3e.dependency-review: success; no vulnerable or denied dependency detected.advisory-scan: success; no known vulnerabilities found.bootstrap-integrity: success; manifest remainedINCOMPLETE/ exit 3.schema-validation: success.lint: success.typecheck: success.test: success; 1232 passed, 1 skipped.sbom: success, including wheel smoke install.compose-validation: success; all services healthy.Historical failed run 32866120292 remains visible. Its test job exposed the
new test loader’s incompatibility with configured importlib mode; descendant
commit
3219d1f…fixed the loader without changing the production assertion.Required declarations