fix(pr-management): reject empty dependency intersections - #1060
Conversation
An empty constraint intersection has no concrete failing resolution, so the prior evidence rule could mistake an uninstallable graph for compatibility. Required output fields and case-specific suggestions also need enforcement so regressions cannot pass by omission. Generated-by: Codex (GPT-5)
37fdaa1 to
4c41ccc
Compare
potiuk
left a comment
There was a problem hiding this comment.
Approving — no findings. The reasoning holds and the gap was real, so it's
worth recording what I checked rather than just "LGTM".
The logic: with an empty effective intersection no resolution exists,
so the previous requirement of "one concrete supported resolution that
still fails" was impossible to satisfy. A definitively uninstallable graph
would therefore have been classified unknown (or the finding suppressed
as unsubstantiated), which is the wrong answer for the most clear-cut kind
of breakage there is. Classifying it broken is correct.
The part that is easy to get wrong and which this PR got right: the
surrounding prose is retro-scoped to "for a non-empty effective
intersection". Without that, the pre-existing rule — "if the available
evidence does not identify a concrete failing resolution, the runtime
incompatibility claim remains unsubstantiated and must not be raised" —
would directly contradict the new empty-intersection rule. The state space
stays broken | compatible | unknown rather than growing a fourth member,
which is the right call.
What I verified:
- The new
has_required_finding_fieldsassertion actually behaves. I
compiled the pattern and ran it: it matches a complete finding, rejects
one missingfile, and rejects[]. That last point is why only the
positive cases opt in viaexpected.json, and whycase-3
(has_empty_findings) correctly does not — the opt-in-by-key mechanism
is used properly. output-spec.mdalready declares every field the assertion demands
(file,category,severity,runtime_compatibility,
mandatory_paths_checked,metadata_coverage,
supported_incompatible_resolution,reason,suggestion) with
enumerations matching the regex alternatives, so it is not asserting a
shape the prompt never asks for. It also omitsrecommended_action,
whichhas_recommended_actionalready covers — no duplicated assertion.case-6is the discriminating case, not just another case.
compat-core>=1.8.0,<2.0.0against thequery-base==2.0.0transitive
compat-core>=2.0.0is genuinely empty, and pairing it with
metadata_coverage: partialand
supported_incompatible_resolution: falseis exactly the combination the
old rules would have mis-classified asunknown. Setting the flag
falseis also right: no resolution exists to be incompatible.- The per-fixture policies are consistent with their expected
remediations. case-1 / case-2 supply a policy forbidding direct
lower-bound changes →release_marker→ "add the marker comment, do not
change the bound"; case-5 / case-6 supply the opposite →
direct_lower_bound→ "raise the bound". Deliberately varying the policy
per fixture is what makes the vendor-neutrality guarantee testable rather
than asserted, and addingsuggestiontograding-schema.json's
prose_fieldsis what enforces it — a contradictory remediation now
fails the rubric instead of going ungraded entirely. - Counts are updated in all three places that carry them: the spec
(115 → 116),tools/skill-evals/README.md, and the suite README
(step-4 5 → 6, with the description updated to name the uninstallable
case).
The spec update in tools/spec-loop/specs/pr-management-family.md is
included in the same PR, which is what AGENTS.md's spec-sync rule asks
for — the specs do not fall behind the behaviour they describe.
This review was drafted by an AI-assisted tool and
confirmed by a Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Magpie handles maintainer review:
CONTRIBUTING.md.
Summary
brokenbecause the dependency graph is uninstallable, without requiring a concrete failing version.This is a focused follow-up to PR #1048, which established the dependency-evidence contract.
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
uv run --project tools/skill-evals skill-eval tools/skill-evals/evals/pr-management-code-review/rendered all 116 cases successfully)(the dependency-compatibility suite passes 6/6 with Codex generating and Claude grading)
RFC-AI-0004 compliance
Linked issues
Follow-up to PR #1048.
Notes for reviewers
The compatibility state space remains
broken | compatible | unknown; an uninstallable graph is abrokengraph, not a fourth state. The eval assertions and prose grading use the existing skill-evals mechanisms without changing the runner.