Skip to content

Effort is per station, and a build reads its own diff before asking you to - #126

Merged
unitypark merged 2 commits into
mainfrom
feat/effort-is-configurable-and-builds-get-reviewed
Aug 18, 2026
Merged

Effort is per station, and a build reads its own diff before asking you to#126
unitypark merged 2 commits into
mainfrom
feat/effort-is-configurable-and-builds-get-reviewed

Conversation

@unitypark

Copy link
Copy Markdown
Owner

Two gaps in the agent layer, found by reading it rather than by a failure.

1 · Effort was typed and then thrown away

ModelCallOptions.effort accepted all five levels and defaulted to high.
Three call sites passed the literal 'high'. Nothing configured it. And the
Claude Code path passed no effort at all — so in subscription mode the
setting specd thought it had chosen never reached the model. claude --effort
exists and takes the same five levels; specd simply never sent it.

One value for every station is wrong in both directions at once. xhigh is
the documented setting for coding and agentic work — it's what Claude Code
itself runs at — while cheap mechanical passes are where low belongs. specd
ran a build that writes code a human is about to merge at the same level as an
index run that summarises text.

Station Was Now
build, review high (CLI: nothing) xhigh
spec, ground high high
index high low

effortFor(station, project.effort) resolves it, projects.effort moves every
station at once, and NULL means "no preference", not low — a project that
never chose one must not be quietly moved off the defaults. The value now
reaches the API path, both CLI paths, and the runner job payload, so a
dispatched build is not silently cheaper than a local one.

2 · Nothing read the diff

Verify answers "do the tests pass" — which the test suite already answers.
Nobody answered "is this the change we approved, and is it any good". That
reading was left entirely to whoever opened the PR, cold.

ReviewAgent now reads the diff with read-only tools between verify and
publish, and answers in a schema: a verdict, a one-line summary, and findings
carrying path:line, a severity, and the acceptance criterion or design claim
each bears on. They render into the PR body above the acceptance criteria —
the new reading sits above what it's measured against.

Four things it had to get right:

  • Advisory, and it says so in the body. A finding does not fail a build.
    That's the line citation drift already sits on: an unrelated opinion must not
    be able to stop an approved spec from shipping. Making findings refusable is
    a house-rule decision (2.4), not a default.
  • Read-only by denial, not omission. Write and Edit are in
    --disallowed-tools and the pass runs under --permission-mode plan, so an
    over-helpful reviewer cannot quietly amend the branch it was asked to assess.
  • An empty findings list is a real answer, and a clean pass is reported
    rather than omitted — silence would be indistinguishable from a pass that
    never ran, which is the same failure as a green verify for tests that didn't
    execute. Tested.
  • It runs where the workspace is. The prompt is authored API-side so both
    paths ask the same thing, but a dispatched build reviews on the runner
    the diff is on the runner's disk and gone by the time the report arrives.
    Building this only for the in-process path would have been the same
    half-wiring as the effort bug above.

What I deliberately did not do

Replace specd's prompts with stock /plan and /design commands. The
SpecAgent's prompts are what produce EARS criteria, citations, and the four
verdicts — swapping them for generic ones would trade the thing that makes a
spec checkable for convenience. Review was the actual gap.

Verify

pnpm typecheck && pnpm test (43 API files), pnpm build, pnpm site:check.
16 new tests: the station-default invariants (including one that fails if they
ever collapse back to a single value), effortFor null handling, and the PR
rendering — severity ordering, the not-run case, the clean case, and section
order. --effort flag values and --permission-mode plan were both checked
against claude --help rather than assumed.

For the reviewer

  • Migration 0019_project_effort.sql adds a nullable column with a CHECK
    constraint. Additive; run pnpm db:migrate.
  • Every build now makes one extra model call, at xhigh. Skipped on an
    empty diff, and soft everywhere — the commits exist and verify has spoken, so
    a review that can't run costs an opinion, not the build. But it is a real
    cost increase per build, on top of build effort rising from high to xhigh.
  • The review prompt is unexercised against a real model here. Its schema
    and rendering are tested; the quality of what it finds is not, and won't be
    until it runs on a real build.
  • Effort is one dial per project, not per station. Someone who wants cheap
    grounding and expensive builds can't express that yet; effortFor is the
    seam.

…ou to

Two gaps in the agent layer, found by reading it rather than by a failure.

Effort was typed and then thrown away. ModelCallOptions.effort took all five
levels and defaulted to high; three call sites passed the literal 'high';
nothing configured it. Worse, the Claude Code path passed no effort at all, so
in subscription mode the setting specd thought it had chosen never reached the
model — `claude --effort` exists and takes the same five levels, and specd
simply never sent it.

One value for every station is wrong in both directions at once. xhigh is the
setting for coding and agentic work and is what Claude Code itself runs at,
while cheap mechanical passes are where low belongs. specd ran a build that
writes code a human is about to merge at the same level as an index run that
summarizes text. STATION_EFFORT now puts build and review at xhigh, spec and
ground at high, index at low; effortFor() resolves it; projects.effort moves
all of them, with NULL meaning no preference rather than low. The value reaches
both model paths and the runner payload, so a dispatched build is not quietly
cheaper than a local one.

Nothing read the diff. Verify answers "do the tests pass", which the test suite
already answers; nobody answered "is this the change we approved, and is it any
good". ReviewAgent now reads the diff with read-only tools between verify and
publish and answers in a schema — verdict, one-line summary, findings carrying
path:line, a severity, and the criterion or design claim each bears on. They
render into the PR body above the acceptance criteria, because the new reading
belongs above what it is measured against.

Four things it had to get right. It is advisory and says so in the body: a
finding does not fail a build, the same line citation drift sits on. It is
read-only by denial rather than omission — Write and Edit are disallowed and
the pass runs under --permission-mode plan, so an over-helpful reviewer cannot
amend the branch it was asked to assess. An empty findings list is a real
answer and a clean pass is reported rather than omitted, because silence would
be indistinguishable from a pass that never ran. And it runs where the
workspace is: the prompt is authored on the API side so both paths ask the same
thing, but a dispatched build reviews on the runner, whose disk holds the diff.

Recorded as decisions/0021, including the cost: effort is one dial per project
rather than per station, and effortFor() is the seam where that would change.
CI failed with every test passing: 506 green, one unhandled `write EPIPE` from
`local-review.ts`. `run()` shells out to `gh`, `glab` and `git`, and in the
common failure cases — no such binary, a rejected push — the child is gone
before the prompt is written. Node reports that write as an `error` on the
stdin stream rather than on the child, and with no listener it is an uncaught
exception. Whether it fires at all is a scheduling race, which is why this
passed on the four PRs since I introduced it.

The runner already knew. `apps/runner/src/claude.ts` carries this exact guard
with a comment saying CI found it there first — and the same pattern in the two
API-side spawns never got it. Both now match the runner's shape rather than
swallowing everything: EPIPE and ERR_STREAM_DESTROYED are the expected ones and
say nothing, anything else lands in stderr, and neither crashes a process over
a prompt the callee was never going to read.

The regression test drives `run()` directly — the race is a property of that
function, and reproducing it through `openLocalReview` would depend on which
binaries the machine happens to have. Writing 8 MB to a command that exits at
once loses the race every time instead of occasionally; without the guard it
fails with the identical `write EPIPE` CI reported.
@unitypark
unitypark merged commit 092c996 into main Aug 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant