fix(core): fence lock renewal and release by generation - #93
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Eviction was already fenced, but renewal and release both read the owner record, awaited, then acted on the lock by pathname. A holder stalled past the expiry could overwrite a successor's record or delete a successor's live lock, admitting two concurrent holders — the outcome the eviction fencing exists to rule out. Fence both paths by owner generation and serialize release against an in-flight renewal, so a stale operation can only affect its own record.
2754324 to
f425bd4
Compare
|
Updated to Cubic found two real bugs after this PR had already passed independent cross-family review, and one of them was a liveness regression this branch introduced — worth stating plainly rather than burying in a thread.
The fix inverts the default rather than patching each path: continuation now happens unless we deliberately decided to stop. All ten exits of the renewal attempt were enumerated and classified — four stop, six continue — because the bug existed precisely because one exit silently skipped the continuation. Also fixed: Declined, with reasoning in-thread: an "atomic generation-conditional mutation" for the residual pre-write window. That primitive does not exist on a plain filesystem — it is the constraint this module is built around. The window is now bounded and documented in-source instead: two winners require both a Gate: 1072 pass / 0 fail (1064 on |
Fixes #90.
Eviction was already fenced. Renewal and release were its unswept twins: both read the owner record, awaited, then acted on the lock by pathname. Branched from
9bf8f4c, independent of #87.The three interleavings
writeOwner()lands unconditionally over P2's record. Two processes now believe they hold the lock.rmremoves P2's live lock and admits a third holder.release()can land its write afterwards, recreating an orphan lock nobody owns that blocks contenders until TTL.All three violate the invariant the eviction fencing exists to hold: the filesystem offers atomic claim but no atomic conditional-delete, so the acceptable worst case is zero winners, never two.
Approach
Both paths now go through the existing eviction-marker primitive as a generation fence, and release awaits any in-flight renewal.
ownerIdremains the generation and the on-disk format is unchanged, so older builds and other processes reading the same file are unaffected.The diff is large relative to the fix because the marker helpers were hoisted for reuse. Eviction semantics are untouched — the reviewer verified the four eviction fences survive 1-for-1 (old 236/240/243/249 → new 274/278/281/287, same checks, same order) and that
src/tests/review-fixes.test.tsis unmodified.Honest severity
Each interleaving needs the holder to stall past the 120s TTL between the check and the act — realistically severe event-loop starvation. Worth fixing because it breaks a stated invariant, not because it is likely to fire.
A testability seam, called out deliberately
Forcing the post-check/pre-write race required a new
onStep('renewal-write-fenced')seam in production code. It is guarded (if (options.onStep)), so nothing is awaited when unhooked.Review flagged that as a footgun rather than a bug: anyone later installing a hook there for telemetry would reopen the two-winner race. So the fence is now re-checked after the seam, immediately before the write — the invariant is structural rather than dependent on nobody hooking that point. A regression test pins that re-check specifically.
The extra
ownsEvictionMarker()call is one more filesystem read per renewal (interval ≥1s). It is fail-closed: a transient read error aborts the renewal, the lock expires naturally, and the result is zero-winner-then-one-winner — the degradation this design already accepts.Tests
1069 pass / 0 fail(1064 on9bf8f4c),tscclean.An earlier revision's proof was invalid and was redone. It red-first'd against
9bf8f4c, where the tests hung at 1s because that commit lacks the seams — a missing-API red, not a behavioural one. The proof is now fault injection on the current fencing code, with each RED a concrete assertion about observed state:ownerIdownerIdoverwrittenPlus 3,000 plain contended rounds electing exactly one owner. That stress seeds a stale lock only, so it exercises happy-path eviction under contention rather than the crashed-evictor branch — that branch stays covered by the existing unchanged tests.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Fences refresh-lock renewal and release by generation to prevent a stalled holder from overwriting a successor or deleting a live lock. Previously both validated ownership then acted by pathname; now both run under the eviction-marker fence, and release waits for any in-flight renewal, so the worst case is zero winners, never two.
withEvictionMarker,ownsEvictionMarker,recoverStaleEvictionMarker); eviction semantics unchanged.ownerIdand that the lease is unexpired, emitrenewal-owner-confirmed, re-check the marker before and afteronStep('renewal-write-fenced'), then write; emitrenewal-finishedregardless. If the marker is busy, emitrenewal-marker-unavailableand reschedule; transient errors also reschedule.ownerId, re-check the marker, then remove the lock; bounded retries recover a stale marker; never delete by pathname without the marker. Emitsrelease-owner-confirmed.ownerIdgeneration are unchanged; no migration. One extra fs read per renewal (fail-closed on transient errors).Written for commit f425bd4. Summary will update on new commits.