aarch64: kernel_stack_ownership_oracle can red on a concurrent free — its alloc/free equality is measured over GLOBAL pool counters, so a stack allocated before the window and dropped inside it reads as slot_balance=-1 (1/452 boots)

[ryanbreen]

Signature

docker/qemu/run-aarch64-arma609-arm.sh --expect clean --boots 100 --profile cortex-a72 --starved
(10 CPU hogs at nice -n 19, QEMU at nice -n 19, IOPS throttle 2000), branch
fix/609-early-kthread-dispatch @ d28407ae — 1 boot in 100 (boot 69):

[KSTACK_OWNER_ORACLE:aarch64:creation_rows=1000:creation_owned=1000:one_owner=1000:two_owner=0:zero_owner=0:
 fork_rows=1:fork_owned=1:slot_returns_exact_one=1:slot_alloc_delta=1000:slot_free_delta=1001:slot_balance=-1:
 frames_mapped_delta=0:frames_released_delta=0:frame_balance=0:frame_used_delta=0:frame_used_bounded=1:
 live_checks=1108:live_refusals_production=0:live_refusals_injected=1:drop_refused_live=0:pte_overwrite_refusals=0:...]
[TEST:process:kernel_stack_ownership_oracle:FAIL:ownership stress slot allocation/free equality failed]

Preserved serial (in-repo):
docs/planning/teardown-unification/609-serials/kstack-ownership-slotbalance-neg1-starved-arma-boot69.txt.

The measurement is global; the assertion is local

kernel_stack_ownership_oracle_test (kernel/src/tracing/providers/teardown.rs) samples
process-wide pool counters either side of its 1000-iteration stress loop:

let stress_before = crate::memory::kernel_stack::kernel_stack_pool_counters();   // :5162
for iteration in 0..OWNERSHIP_STRESS_ITERATIONS { ... }                          // :5169-5218
let stress_after  = crate::memory::kernel_stack::kernel_stack_pool_counters();   // :5220
measurements.slot_alloc_delta = after.slots_allocated - before.slots_allocated;  // :5224
measurements.slot_free_delta  = after.slots_freed     - before.slots_freed;      // :5227
... if slot_alloc_delta != slot_free_delta { FAIL }                              // :5662

KSTACK_SLOTS_ALLOCATED / KSTACK_SLOTS_FREED are global atomics
(kernel/src/memory/kernel_stack.rs:16-17), bumped by every pooled-stack allocation and by every
KernelStack::drop that returns its slot (kernel_stack.rs:239). So any pooled stack allocated
before stress_before and dropped inside the window contributes +1 to slot_free_delta with no
matching alloc — which is exactly the observed arithmetic: alloc_delta = 1000 (precisely
OWNERSHIP_STRESS_ITERATIONS), free_delta = 1001, slot_balance = -1.

Boot 69's own serial shows that concurrency is live inside the window by design — two other
subsystems' markers print between this test's START and its result line:

[TEST:process:kernel_stack_ownership_oracle:START]
[STRAND_INJECT_ORACLE:aarch64:legA_exercised=1:legA_recovered=1:legB_exercised=1:legB_recovered=1:stranded=0]
[CREATION_LOCK_ORDER:INJECTED:PM_HELD]
[KSTACK_OWNER_ORACLE:...:slot_balance=-1]

The boot-test executor runs subsystem kthreads concurrently, and deferred reclamation of a terminated
thread's stack runs on whichever CPU reaches the reaper.

It is not an over-free — checked, not assumed

• two_owner=0 and one_owner=1000: custody moved from the row to the published copy exactly once on
  all 1000 iterations, so no stress row and its published copy both hold the same slot. A double free
  from the loop itself is excluded by the loop's own accounting.
• KSTACK_SLOTS_FREED is incremented once per KernelStack::drop that gets past the live-slot guard
  (kernel_stack.rs:230-245); each KernelStack value drops once, so an extra increment requires an
  extra owner, not a repeated free of one owner.
• drop_refused_live=0, live_refusals_production=0, pte_overwrite_refusals=0: the release-mode
  live-slot guard never fired in that boot, which is what a slot handed out while still live would
  trip.
• The rest of the boot is intact: reclaim_progress_gate:PASS, retirement_fence_gate:PASS,
  PT_RETIRE_ORACLE ... lost=0, and the only failing test in the boot is this one.

The same arithmetic already has a known, non-defect cause

At c9c4322b (before the I2 reorder in this same round) the census-widening probe — a real kthread
spawned by a concurrently-running subsystem — produced the identical shape on 5 of 12 boots
(slot_alloc_delta=1000:slot_free_delta=1001:slot_balance=-1,
/tmp/breenix_arma609_greencensus_20260821T223355Z-2429). That interference was root-caused and
removed by ordering the probe after this oracle. Boot 69 is the same arithmetic with a different
concurrent thread as the source.

Rate, and what the branch did and did not change

corpus	boots	slot_balance != 0
post-fix, all preserved corpora (clean 100+100, green legs 12+12+12, starved 16+100, plus the R37 re-run 100)	452	1
pre-branch corpora on this machine (main-powered control 200, force609 arms 110, pre-fix arm-A legs 43)	353	0

The R37 re-run of the identical starved leg (100 boots, same hogs, same profile, same commit) was
100/100 GREEN with slot_balance=0 on every boot. 1/452 vs 0/353 does not separate the two rates
(P(0 events | p = 1/452, n = 353) ≈ 0.46).

This branch does change reclamation cadence: run_deferred_reclamation() was hoisted out of
schedule_from_kernel and is now the first statement of every idle_loop_arm64 iteration, plus an
explicit call immediately before every external schedule_from_kernel call. That drains pending stack
frees at least as promptly as before and never later, so it can shift when a pending free lands
relative to this oracle's window — but it cannot manufacture a free, and the counters above show none
was manufactured.

What to do about it (and what not to)

The oracle's equality is only sound if no other pooled-stack owner is dropped inside its window, and
the kernel does not guarantee that — production reclamation is asynchronous by design. Options, in
order of preference:

1. Scope the measurement to the oracle's own slots (tag or record the indices the stress loop
   allocates and prove those return), so the assertion measures what its failure string claims.
2. Failing that, measure alloc_delta - free_delta against an explicitly recorded count of foreign
   frees inside the window (a counter of drops whose allocation predates stress_before), so a
   foreign free is reported rather than confused with an imbalance.

Do not widen the tolerance to |slot_balance| <= 1: that is a FAIL condition weakened to fit a
measurement artifact, and it would hide a genuine single-slot leak — which is precisely the class this
oracle was built for (#579 / Tranche-2 P4).

Provenance

Branch fix/609-early-kthread-dispatch @ d28407ae, R37 amend slot (T3-G PR1 r2 review finding
R2-B2, first red leg). Host: Mac, -M virt,gic-version=3 -cpu cortex-a72 -m 512 -smp 4,
boot_tests,arm_a_609 profile of aarch64-breenix-kernel.json.