Skip to content

fix(review-pr): fail closed on incomplete outcomes - #94

Open
Sayt-0 wants to merge 1 commit into
mainfrom
fix/review-pr-fail-closed-outcomes
Open

fix(review-pr): fail closed on incomplete outcomes#94
Sayt-0 wants to merge 1 commit into
mainfrom
fix/review-pr-fail-closed-outcomes

Conversation

@Sayt-0

@Sayt-0 Sayt-0 commented Aug 28, 2026

Copy link
Copy Markdown
Member

Summary

  • makes PR review outcomes fail closed instead of deriving success from an agent exit code or log text
  • keeps every GitHub review submission on the COMMENT event and replaces approving language with the neutral 🟢 NO FINDINGS completion label
  • validates and attributes each posted review with a trusted per-run marker, then propagates the classified outcome through the reusable workflow and check run
  • preserves low-severity findings, bounds delegation retries, and updates incremental-review and rate-limit handling for the new review markers
  • refreshes reviewer eval fixtures after model-backed canaries exposed stale or contradictory scenarios

Root cause

Several independent paths could present an incomplete run as successful:

  • an exit code of zero without a posted review triggered a synthetic LGTM fallback
  • low-severity findings were skipped during verification and could disappear before assessment
  • incomplete or inconclusive agent output was not independently classified after posting
  • the reusable workflow used job.status while continue-on-error masked review failures
  • the posting template contained an approving default and route placeholders the model had to rewrite

Behavior

Scenario Result
complete, conclusive, zero surviving findings COMMENT with ### Assessment: 🟢 NO FINDINGS
any surviving finding, including low severity COMMENT with NEEDS ATTENTION or CRITICAL
incomplete drafter coverage COMMENT with Review incomplete; no incremental checkpoint
inconclusive verification COMMENT with Verification inconclusive; no incremental checkpoint
no attributable review posted non-approving incomplete notice and non-success workflow status
malformed, conflicting, approving, or non-comment review fail closed as unverified

Safety and compatibility

  • stages immutable repository, PR, and head-SHA routing in the trusted action
  • creates and masks a cryptographically random per-run nonce
  • validates the review body and comments array in bundled TypeScript before posting
  • constructs the payload in a file before invoking gh api, with event: COMMENT hardcoded
  • rejects active APPROVE, LGTM, and No issues found wording
  • keeps historical ### Assessment: reviews compatible while rejecting the old synthetic LGTM fallback as an incremental checkpoint
  • keeps structured-output schemas provider-compatible and free of JSON Schema pattern

Validation

Check Result
Unit tests 40 files, 1413 tests passed
TypeScript typecheck passed
Biome passed
Build passed
actionlint passed
Shell integration tests 3 suites passed
YAML and eval JSON parsing passed
Model-backed targeted evals on Docker Agent v1.121.0 4/4 evals, 24/24 relevance criteria
Model-backed clean-case repetition on v1.121.0 3/3 evals, 24/24 relevance criteria
Additional security, confidence, and scope canaries 3/3 evals, 20/20 relevance criteria
Fresh-eyes and specialist review findings resolved

Model-backed eval outputs are intentionally kept outside the repository. No eval posted to a real pull request; GitHub posting behavior is exercised through the production shell harness with a mocked gh endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant