fix(aarch64): validate the ret-dispatch resume PC and complete the inline-switch outgoing handoff (#576, #607)

[ryanbreen]

Summary

Six places on aarch64 load a stored resume PC and transfer to it. Five validated the value and
redirected to idle_loop_arm64 when it failed; the sixth — the ret-based kernel dispatch fed by
take_inline_ret_dispatch_info and consumed by aarch64_ret_to_kernel_context, the one tried
first at both dispatch sites — did not. Round 1 gave it a validation. The gate then showed the
validation was the wrong shape: it was an address-space check, so it closed the PC == 0 face and
left the PC == <some other kernel address> face of the same consumer fully open. Round 2 makes the
predicate a kernel-text-range check off the linker's own bounds and raises the assembly floor to
match. Along the way this PR also closes one place that leaves a thread mis-classified as
inline-saved, and completes an outgoing-thread handoff that one fallback arm left half-done.

Closes #607 — with DEVIATION D5 attached (below) and with the note that the RCA's second
commit-2 bullet, publishing previous_thread in schedule_from_kernel, was measured harmful 3/3
and is correctly absent from this PR.

Not closed here.

• aarch64: rare EL1 INSTRUCTION_ABORT (ELR=0, FAR=0) during process spawn on the boot_tests strict gate (~1/80 boots) #576 — no close claim on this PR. What is proven is signature-scoped: the ret consumer can no
  longer transfer to a PC outside kernel text, by two independent nets, each mutation-proven; the
  FAR=0 ELR=0 bucket read 0/350 in round 1 with its tolerance removed; and the empirical
  confirmation R38(c) asked for (refusals > 0 in production with zero aarch64: rare EL1 INSTRUCTION_ABORT (ELR=0, FAR=0) during process spawn on the boot_tests strict gate (~1/80 boots) #576-shape aborts) has not
  been obtained — every production-profile gate reads RET dispatch refused: 0 marker lines. The
  close wording is deliberately left to the confirm slot.
• aarch64: EL1 instruction abort at PC=0 with ESR=0x8600000d (IFSC=0xd, permission fault) — NOT #576's 0x86000005, 1/25 max-profile SS-gate boots #626 — stays open, not claimed. Its capture carries spsr=0x200023c5 with DAIF I+F set and
  every resume-PC consumer clears those one instruction before transferring, so it was not reached
  from a dispatch transfer. An honest comment saying that is on the issue.
• aarch64: ret-dispatch transfers to a kernel-STACK resume PC — [INSTRUCTION_ABORT] FAR==ELR==x30==x29, ESR=0x8600000e IFSC=0xe, 3/200 clean SS-gate boots on both CPU profiles #635 — filed by this PR for the face the round-1 gate produced, and not closed by it. See
  "What round 2 does and does not settle".
• aarch64: a live thread id is ERETed as a userspace PC ([PC_ALIGN] ELR=<tid>, from_el0=1) — one face across two preserved serials, EL0 resume PCs are unvalidated #633 — the [PC_ALIGN] ELR=<tid> face; an EL1 exemplar found in this PR's starved gate is
  appended to it, which widens it from the EL0-only face it was filed as.
• aarch64: disagreeing INSTRUCTION_ABORT record pairs in one serial score UNATTRIBUTED on the SS gate (~2/400) #613 — this PR's starved gate produced one more exemplar of the disagreeing-record-pair face;
  cross-referenced on the issue.

Commits

c3b0133d	test — three feature-gated self-tests (legs K, E, S). They report FAIL on this commit, on purpose.
2d7ce786	fix — validate the resume PC on the ret-based kernel dispatch path (#576)
ddd03a11	fix — complete the outgoing-thread handoff in the inline-switch fallback (#607)
6b3e143d	test — gate the ret-dispatch census, attribute #626, drop the #576 tolerance
c9c75c3b	docs — preserve the ten round-1 serials
972a0832	fix — round 2: admit only kernel text as a ret-dispatch resume PC (#576, #635); legs T and F
a092e8a4	test — make the floor leg's verdict depend on whether the boot faulted
71cceff0	docs — preserve the round-2 serials and the four round-1 gate boots that failed

What changed

2d7ce786. is_kernel_addr is lifted out of restore_kernel_context_inline into one file-level
resume_pc_is_dispatchable, so both consumers share one authority.
take_inline_ret_dispatch_info returns None when it fails — before any of its mutations, so a
declined dispatch leaves the thread exactly as it found it — and the dispatch falls through to the
ERET arm, where the existing redirect-to-idle already handles it. No guard relaxed, no dispatch
added, nothing freed twice: a refusal routes onto a path that already exists. It also bumps
RET_DISPATCH_REFUSALS / RET_DISPATCH_REFUSED_TID and prints, capped at 8 emissions. It also
mirrors a floor into aarch64_ret_to_kernel_context, matching the one aarch64_enter_exception_frame
has carried all along. And ExecSchedCommit::apply clears the inline-schedule classification when it
replaces t.context, through a single new authority Thread::clear_inline_schedule_state.

ddd03a11. The scheduler_ptr-null fallback arm of inline_schedule_trampoline finishes the
outgoing thread's transaction the way the normal arm does: normalise elr_el1 = x30, requeue under
the same two-part condition, clear previous_thread unconditionally.

6b3e143d. The service-sequence gate's 576 bucket is no longer excluded from the FAIL
condition. 0x0 0x0 0x8600000d gets its own field-exact, still-failing 626 bucket — naming, not
tolerating. [RET_DISPATCH_REFUSED: is counted per boot and per profile, reported and never gated.

972a0832 — round 2. The predicate stops being an address-space test:

// before: addr >= 0xFFFF_0000_0000_0000 || (addr >= 0x4008_0000 && addr < 0xC000_0000)
// after:  4-byte aligned && inside [__kernel_text_start, __kernel_text_end), in either alias

__kernel_text_start / __kernel_text_end are already exported by arch_impl/aarch64/linker.ld:57,78
and bracket .text.vectors + .text and nothing else; exception.rs:100-118 is the existing
precedent for reading a linker bound from Rust. Disassembly of the built kernel-aarch64 shows the
bounds materialised PC-relatively (adr for the start after linker relaxation, adrp+add for the
end), so the window is automatically the alias of whichever mapping is executing — the high-half HHDM
alias on QEMU, the identity-mapped physical alias on Parallels, where the loader enters kernel_main
at a physical address. A PC spelled in the counterpart alias is admitted through a second text-sized
window rather than refused for its spelling. Live window on this build:
[0xffff000040400000, 0xffff000040600000).

The assembly floor is raised to the same lower bound — adrp x16 / add x16, :lo12: / cmp x1, x16
against __kernel_text_start, replacing cmp x1, #0x1000. x16 is scratch there and is what the
neighbouring assembly in the same file already uses; x19-x30 and SP are live and untouched.

Both nets are register-only arithmetic — no lock, no allocation, no logging on any always-taken path.
addr_of! on an extern static is an address computation, not a load. The census line gains an
explicit :pc= field naming the refused value; no existing field is renamed or removed, so the gate's
grep -cF and both ratchets that read the line are unaffected.

[RET_DISPATCH_REFUSED:tid=10:pc=0xffff000054265f00:cpu=2:x30=0xffff000054265f00:elr=0xffff0000405468c0:sp=0xffff000054265f00:has_started=1:bis=1:priv=1]

a092e8a4. Leg F's verdict was armed && fired, and the kernel survives its substituted dispatch
in both configurations the leg is meant to separate — so it printed PASS on both sides of its own
mutation. It now also reads any_fatal_postmortem_captured(), a two-line accessor over the array
exception.rs already maintains, and prints it as fatal=.

Evidence

60 s boots, -cpu cortex-a72 -smp 4, soft-float target. Every serial is in
docs/planning/teardown-unification/607-576-serials/, with a README section per round.

Round 1 — legs K, E, S

leg	test commit c3b0133d	HEAD
K — ret dispatch with a zero saved resume PC	refused=0:FAIL and [FATAL_REGS] label=INSTRUCTION_ABORT cpu=2 esr=0x8600000d far=0x0 elr=0x0	refused=1:refused_tid=10:PASS, no abort, [BOOT_TESTS:PASS]
E — the ExecSchedCommit classification hole	victim_tid=1212:inline_left_set=1:cleared=0:FAIL	inline_left_set=0:cleared=1:PASS
S — the dropped outgoing thread (#607)	stranded=1:FAIL (2/3 boots; the third fired=0:FAIL, refusing to pass by omission)	stranded=0:PASS (3/3)

round-1 mutation	result
Rust refusal removed, assembly floor kept	refused=0:FAIL, no abort, [BOOT_TESTS:PASS] — the floor contains it; the Rust arm is what names the producer
both removed	refused=0:FAIL and [INSTRUCTION_ABORT] FAR=0x0 ELR=0x0 ESR=0x8600000d, [BOOT_TESTS:FAIL:1]

Round 2 — legs T and F

Leg T writes the designated victim's own saved sp into context.x30 at leg K's hook: a value the
round-1 predicate accepts and the round-2 predicate refuses. Leg F substitutes
resume_pc = 0x0100_0000 at the two call sites of aarch64_ret_to_kernel_context, after the
predicate has accepted the real value — the window the assembly floor exists for, and a constant
above the old #0x1000 bound and below __kernel_text_start, so the raise is the only variable.

round-2 mutation	net removed	result
1	the Rust text-range bound, reverted to round 1's address-space bound	leg T refused=0:refused_tid=0:FAIL and [INSTRUCTION_ABORT] FAR=0xffff000054265f00 ELR=0xffff000054265f00 ESR=0x8600000e, spsr=0x20000305, x30 == the same value — #635's field set from a single-field write
2	the raised assembly floor, lowered back to cmp x1, #0x1000	leg F fatal=1:FAIL and [FATAL_REGS] label=INSTRUCTION_ABORT cpu=3 esr=0x8600000d far=0x1000000 elr=0x1000000 four lines after the substitution

leg	at 972a0832
T	[RET_DISPATCH_REFUSED:tid=10:pc=0xffff000054265f00:…], refused=1:refused_tid=10:PASS, no abort — 3 boots
F	fatal=0:PASS, no abort at 0x1000000, [BOOT_TESTS:PASS]
control (boot_tests, neither feature)	0 aborts, 0 refusals, 0 [TEST:…:FAIL:…], [BOOT_TESTS:PASS]

Mutation 1 also shows x16=0xffff000040400000 in the fatal dump — the raised assembly floor loading
its bound and correctly letting a stack address through, since a stack address is above kernel text.
That is the direct evidence that the assembly floor structurally cannot catch this face and the Rust
predicate must.

Scoped honestly: mutation 1's capture reproduces the transfer and the resulting fault
(ESR=0x8600000e, IFSC=0xe, FAR == ELR == x30, spsr=0x20000305, from_el0=0) — not the
producer's write pattern. #635's field captures additionally have x29 == x30 and six consecutive
stack slots across x19-x27; a single-field write cannot make those.

Collateral, disclosed: 1 of leg T's 3 green boots also carries
[TEST:process:kernel_stack_ownership_oracle:FAIL] with slot_alloc_delta=1000:slot_free_delta=1001
— one more free than alloc inside that test's census window, which is what a thread allocated before
the window and reaped inside it looks like; the leg's own victim is such a thread and its verdict line
lands inside that test. The same boot reads two_owner=0, zero_owner=0, drop_refused_live=0,
frame_balance=0. The other 2 greens and the disarmed control are clean.

What round 2 does and does not settle

Does: the ret-based kernel dispatch can no longer transfer to a resume PC outside kernel text.
Two independent nets, each proven load-bearing by its own single-variable mutation, and a refusal
names the thread instead of killing the machine.

Does not: close #635. One boot on the fixed tree, after its own [BOOT_TESTS:PASS], produced
the family again (legF-green-floor-contained.txt:716,
FAR=ELR=0xffff000054276f28 ESR=0x8600000e IFSC=0xe). Its register file is a different producer from
the three gate captures:

	the three gate captures	the post-fix one
x29	== x30 == the faulting PC	0x1
x19-x27	six consecutive stack slots	ordinary live values
sp vs PC	PC is sp + 0x2c0	PC is sp + 0x38

x30 == sp + 0x38 with a garbage x29 is an ordinary compiled epilogue (ldp x29, x30, [sp, #0x38])
reloading an overwritten saved-LR slot and returning to it. No dispatch helper participates and no
resume-PC predicate can see it. The discriminator is written up on #635 so a later gate can attribute
a hit to the right producer instead of scoring the union.

Gates — every run, with its real verdict

Round 1, on c9c75c3b:

gate	boots	verdict
strict kernel-merge, mac, 3 × 20	60	PASS — 60/60, no red boots of any field
service-sequence 25/profile	50	PASS — 50/50 GREEN, every named bucket 0, UNATTRIBUTED = 0
service-sequence clean 100/profile	200	FAIL — 197/200; 3 UNATTRIBUTED, all three the ESR=0x8600000e kernel-stack-PC face, both CPU profiles → now filed as #635, serials in-repo
service-sequence starved 50/profile (14 host yes hogs, nice -n 19)	100	FAIL — 99/100; 1 UNATTRIBUTED, a disagreeing record pair = #613, pre-adjudicated → cross-referenced on the issue, serial in-repo
run-aarch64-full-test.sh --rebuild --boot-tests-only	1	PASS — 107/107
production-profile boot	1	PASS — 0 crash markers, seam absent
Parallels, fresh epoch-named VM × 3	3	PASS — bsshd listening, 0 fault markers, all VMs stopped
beast x86 custody	5	PASS — 5/5, every pinned oracle literal exact
beast x86 run-boot-parallel 4 × 5	20	FAIL — 17/20; 2 garbage-fd sys_read livelocks = #630, 1 loopback_recv_wake reader-EOF (unfiled, aarch64-only diff, no main baseline)
eleven structural suites	—	PASS at HEAD and at every commit

Round 2, on 71cceff0:

gate	boots	verdict
thirteen structural suites (297 tests)	—	PASS
aarch64 production profile, aarch64-breenix-kernel.json	—	PASS, zero Breenix warnings
aarch64 boot_tests profile	—	PASS, zero Breenix warnings
run-aarch64-full-test.sh --boot-tests-only --rebuild	1	PASS — 107/107
production-profile boot	1	PASS — bsshd listening, futex seam absent, 0 crash markers
service-sequence 25/profile	50	FAIL — 48/50 GREEN, 2 UNATTRIBUTED, both attributed below

The two round-2 gate reds, by field signature:

profile / boot	record	attribution
max, boot 11	[PC_ALIGN] ELR=0x5 FAR=0x5 from_el0=0 cpu=1, x30=0x5, x29=0x1f, and DISPATCH_TRACE[0] U old=5	#633 — third exemplar, and the one that shows the value is this CPU's own outgoing tid at the preceding dispatch. Appended to the issue.
cortex-a72, boot 7	[INSTRUCTION_ABORT] FAR=0xffff000040800000 ELR=0xffff000040800000 ESR=0x8200000e IFSC=0xe from_el0=1, victim SIGSEGVs	#637 — newly filed. A userspace thread resumed at a kernel address: EC=0x20 (abort from a lower EL), a second value shape in the "EL0 resume PCs are unvalidated" gap #633 names.

Both serials are preserved in-repo, with the run log.

Neither is a refusal, and neither can be one. Across all 50 boots there are zero
[RET_DISPATCH_REFUSED: lines and zero WARN: bad elr … redirecting to idle lines, so neither of
this PR's predicates returned false even once in the run — round-2's dispatch behaviour is identical
to round-1's here, and these two reds cannot be caused by the narrowing. 0x5 is below both
versions of the assembly floor, so that transfer did not come through aarch64_ret_to_kernel_context
at all; from_el0=1 puts the other one on a path this PR does not touch. Round 1's 25/profile run was
50/50 on the same script, so at these rates the two runs are consistent with the same background.

One process note, since it affects how the numbers above should be read: an earlier 25/profile run
was discarded at boot 9, not reported. run-aarch64-service-sequence-gate.sh:146-153 documents
that a concurrent cargo build hardlinks a different feature set onto the one
target/aarch64-breenix-kernel/release/kernel-aarch64 path between boots, and that the gate's guard
only runs at startup; I started the production-profile boot test — which builds without
boot_tests — while that gate was mid-run. The kernel was rebuilt with --features boot_tests and
the whole gate re-run with nothing else touching the tree. The reported run is the clean one.

Open before merge

0. The service-sequence gate is not clean on this branch. 48/50, both reds attributed to filed
   issues (aarch64: a live thread id is ERETed as a userspace PC ([PC_ALIGN] ELR=<tid>, from_el0=1) — one face across two preserved serials, EL0 resume PCs are unvalidated #633, aarch64: a userspace thread is resumed at a kernel address — [INSTRUCTION_ABORT] ESR=0x8200000e IFSC=0xe from_el0=1, FAR==ELR==0xffff000040800000, 1/50 SS-gate boots #637) and neither reachable by this PR's predicates — but the gate's own verdict is
   FAILED and this PR does not ask for that to be waived.
1. No main baseline. Nothing ran on main at comparable N, so whether aarch64: ret-dispatch transfers to a kernel-STACK resume PC — [INSTRUCTION_ABORT] FAR==ELR==x30==x29, ESR=0x8600000e IFSC=0xe, 3/200 clean SS-gate boots on both CPU profiles #635's 1.5% is
   pre-existing or was raised by ddd03a11's strand-to-ret-dispatch conversion is not measured. The
   binding rule: main >= 1 = pre-existing, with the rate noted on aarch64: ret-dispatch transfers to a kernel-STACK resume PC — [INSTRUCTION_ABORT] FAR==ELR==x30==x29, ESR=0x8600000e IFSC=0xe, 3/200 clean SS-gate boots on both CPU profiles #635; main == 0 = caused by
   ddd03a11, disclosed plainly here. An equal-N control on main is the next gate slot's.
2. The wider acceptance run. A 300-boot leg with the refusal census proven fireable — and, per the
   finding above, attributed by register-file shape rather than by raw ESR/IFSC, or it scores a
   producer this PR never claimed.
3. Round-1 review items F5-F11, carried forward: F5 no checked-in runner for the oracles and their
   FAIL cannot fail a gate; F6 leg S's stranded == 0 can pass vacuously; F7 report_strand prints a
   global instead of its argument; F8 the census can only report min(refusals, 8); F9
   RET_DISPATCH_REFUSED_TID is a last-wins slot; F10 orphaned "tolerated" prose in
   run-aarch64-arma609-arm.sh; F11 PRIOR_NAMED_INSTRUCTION_ABORT_ARMS + 1 is an equality, not a
   floor.
4. The x86 loopback_recv_wake reader-EOF failure from round 1's beast run is still unfiled.

DEVIATION D5 — leg S suppresses one recovery path, deliberately and test-only

Under strand_inject_live_outgoing only, the widened arm clears previous_thread for the CPU when
the stimulus engages, because fix_exception_cleanup_cpu_state() would otherwise opportunistically
re-enqueue the dropped thread and mask the very transaction under test. That backstop is
opportunistic, not guaranteed — #607 was observed in the field at 1/50 despite it — so suppressing
it isolates the fallback arm's own completion. It also means leg S measures that completion, not the
end-to-end field rate. #607's close carries this disclosure.

One design bullet was measured harmful and is not here

The RCA's outgoing-handoff design had a second hunk: publish
cpu_state[cpu].previous_thread = Some(old_id) in schedule_from_kernel, symmetric with the IRQ
path. Measured in isolation on top of the containment commit:

hunk	strand census, all three self-test profiles
the fallback-arm completion	stranded=0:running_shape=0:ready_shape=0, 0/3 boots
the previous_thread publication	stranded=1:ready_shape=1, 3/3 boots, first strand a user thread at state=Ready, dwell_ms 2037/2035/2001

Unlike the IRQ path, the cooperative path has no DEFERRED_REQUEUE slot a later scheduler entry
always drains, so while the marker is up a wake publishes the thread Ready and declines to enqueue
it, and once the trampoline clears the marker without requeueing there is no durable owner left. It is
not in this PR, and ddd03a11's message records the measurement rather than the intention.

Ratchets

Every structural check that moved got tighter, none looser.
inline_schedule_trampoline_retains_the_redundant_elr_guard pinned a bare count of one normalisation;
it now derives the trampoline's arms from the source and requires every arm to carry exactly one,
so a future third arm without it trips the check. serial_line_atomicity_structure's two-way
unlocked-writer census admits the new writers at their exact counts with the rationale inline: they
run under the scheduler lock on a dispatch path where the locked writer is unavailable, all of them
are one-shot or emission-capped rather than periodic, and [RET_DISPATCH_REFUSED: is a gate census
and never a gate condition, so a torn line can under-count a reported number but cannot flip a
verdict. Same class as the check_inline_save_resume_point / check_inline_eret_resume_pc entries
already there.

🤖 Generated with Claude Code