Skip to content

Wire post-swap monitor into live 2-policy serving (rollback was dead code) - #301

Merged
rylinjames merged 1 commit into
mainfrom
fix/wire-post-swap-rollback-and-docker-smoke-lowercase
Aug 12, 2026
Merged

Wire post-swap monitor into live 2-policy serving (rollback was dead code)#301
rylinjames merged 1 commit into
mainfrom
fix/wire-post-swap-rollback-and-docker-smoke-lowercase

Conversation

@rylinjames

Copy link
Copy Markdown
Collaborator

Summary

  • pro/post_swap_monitor.py and pro/rollback.py were real, well-designed, unit-tested-in-isolation code, but neither was imported anywhere outside pro/__init__.py and their own test files. No CLI path or live server code reached them. If a promoted policy regressed in production, nothing in the running system would notice or auto-rollback.
  • Adds PolicyCrashTracker.force_drain() / clear_forced_drain() as an explicit external-override mechanism, kept distinct from the existing crash-count-derived drain path so a monitor trip doesn't corrupt crash telemetry those counters also drive.
  • Wires PostSwapMonitor + RollbackHandler into runtime/server.py's 2-policy /act path: T1 (safety-clamp) and T3 (safety-violation webhook count) use signals already computed per-request (result["guard_summary"]["clamp_count"], result["safety_violations"]); a trip calls dispatcher.force_drain() (same mechanism an organic crash-count drain already uses) and RollbackHandler.execute() for the audit trail.
  • Adds --rollback-sensitivity and --post-swap-baseline-clamp-rate to tether serve.
  • T2 (action cosine-similarity to the previous model) is intentionally left unwired live. Computing it per request would mean running both policies on every call, doubling inference cost. PostSwapMonitor already supports cos_to_previous_model=None for exactly this case ("T2 not checked this episode"), so this is documented, not silently faked.
  • Does not include the docker-smoke.yml fix for the same lowercase-registry-name root cause found while diagnosing this (broken since ~2026-05-10). The token used to push this branch lacks the workflow OAuth scope required to modify .github/workflows/*. Sending that fix to the repo owner directly as a patch to apply.

Test plan

  • Full suite: pytest tests/, 3192 passed, 100 skipped, 0 failed (baseline before this change: 3183/100/0, the delta is exactly the 9 new tests below, no regressions)
  • 6 new tests in test_two_policy_dispatcher.py: force_drain overrides routing without touching crash counters, persists across multiple predicts, clear_forced_drain restores normal routing, rejects unknown slots, and takes priority over an organic crash-derived drain
  • 3 new tests in test_post_swap_rollback_integration.py: PostSwapMonitor + RollbackHandler composed against a real TwoPolicyDispatcher (not the existing rollback tests' _StubRouter, which encodes a router.set_active() API PolicyRouter never actually implements), proves T2 and T3 trips actually drain the monitored slot on the next live request, and that sub-sensitivity-threshold trips correctly do not fire

🤖 Generated with Claude Code

…owercase bug

post_swap_monitor.py and rollback.py existed as real, unit-tested-in-isolation
code but were never imported outside pro/__init__.py and their own test
files -- no CLI path or live server code reached them, so an auto-rollback
on production regression was effectively dead code. Adds a force_drain
override to PolicyCrashTracker (kept distinct from crash-count-derived
drains so it doesn't corrupt that telemetry), wires PostSwapMonitor +
RollbackHandler into runtime/server.py's 2-policy /act path using T1
(safety-clamp) and T3 (safety-violation webhook count) signals already
computed per-request, and adds --rollback-sensitivity /
--post-swap-baseline-clamp-rate CLI flags. T2 (action cosine-similarity to
the previous model) is intentionally left unwired live -- computing it per
request would double inference cost by running both policies on every
call -- and is documented as such rather than faked.

NOTE: this PR does NOT include the docker-smoke.yml fix for the same root
cause (raw docker pull against ghcr.io/github.repository is mixed-case and
therefore invalid). That fix is ready but blocked on push: this session's
git token lacks the `workflow` OAuth scope GitHub requires to push changes
under .github/workflows/. Delivered out-of-band to the repo owner as a
patch to apply directly.

New tests: 6 in test_two_policy_dispatcher.py covering force_drain,
clear_forced_drain, and priority-over-organic-drain; 3 in a new
test_post_swap_rollback_integration.py proving PostSwapMonitor and
RollbackHandler compose correctly against a REAL TwoPolicyDispatcher (the
existing rollback tests only exercised RollbackHandler against a stub
router matching an API PolicyRouter never actually implements). Full
suite: 3192 passed, 100 skipped, 0 failed (was 3183/100/0 before this
change).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rylinjames
rylinjames merged commit e6a9dff into main Aug 12, 2026
7 checks passed
@rylinjames
rylinjames deleted the fix/wire-post-swap-rollback-and-docker-smoke-lowercase branch August 12, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant