Deterministic simulation of multi-worker computations#810
Open
frankmcsherry wants to merge 2 commits into
Open
Deterministic simulation of multi-worker computations#810frankmcsherry wants to merge 2 commits into
frankmcsherry wants to merge 2 commits into
Conversation
A SimNet/SimAllocator pair mirroring ProcessAllocator's framed, serializing data path, with all inter-worker delivery held in per-(source, target) FIFO streams that a driver releases explicitly. All workers live on one thread, so an execution is a pure function of the sequence of step/deliver decisions. Self-directed messages bypass the pending queues: real allocators surface them at the next receive(), and gating them would admit schedules no deployment can produce. Fault injection (per-stream FIFO violation, message loss) is segregated and documented as oracle-validation tooling rather than legal-schedule exploration. Adds a Sim variant to the Allocator enum; the change is additive, with match arms in the existing dispatch and no changes to existing variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
timely::simulate hosts several workers on the calling thread, communicating through streams whose delivery the caller controls via a sequence of Decisions (step a worker; deliver messages on a stream). Executions are pure functions of the decision sequence, and any sequence is valid, so schedule traces are recordable, replayable, and shrinkable by construction. Workers run without a timer, so time-based rescheduling degrades to immediate activation and nothing reads the wall clock. Tests include multi-worker barriers asserting progress safety under seeded random schedules, and a property-testing grind whose oracles check frontier safety, conservation, quiescence, and determinism, with fault-injection meta-tests confirming the oracles detect actual protocol violations. This harness produced the single-worker deterministic reproduction of TimelyDataflow/differential-dataflow#801 (fixed by differential-dataflow#802) after the bug had resisted reproduction under thread-contention alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deterministic simulation of multi-worker computations
Adds
timely::simulate: several workers hosted on the calling thread, communicating through per-(source, target)FIFO streams whose delivery the caller controls. An execution is a pure function of the applied sequence ofDecisions (step a worker; deliver up to N messages on a stream), and any decision sequence is valid — releases clamp to what is pending — so schedule traces are recordable, replayable, and shrinkable by construction.Two commits:
SimNet/SimAllocator, mirroringProcessAllocator's framed, serializing data path so simulated runs exercise the byte-oriented path. Self-directed messages bypass the pending queues (real allocators surface them at the nextreceive(); gating them would admit schedules no deployment can produce). Fault injection — per-stream FIFO violation and message loss — is segregated and documented as oracle-validation tooling, not legal-schedule exploration. TheAllocatorenum gains aSimvariant; the change is additive.Simulation/Decisiondriver and tests: multi-worker barriers asserting progress safety (no round notified twice, rounds in order) under seeded random schedules, and a property-testing grind whose oracles check frontier safety, conservation, quiescence, and same-seed determinism, with fault-injection meta-tests confirming the oracles detect actual violations (message loss is caught in every affected run).Why this is worth carrying in-tree
This harness produced the deterministic reproduction of TimelyDataflow/differential-dataflow#801 — a batch-loss bug in DD's join that had resisted reproduction except as a ~20% flake under CPU contention with three workers. Under the simulator the bug reproduced on a single worker with no chaos at all, deterministically on every seed, which is what made the bisection to first divergence (and the fix, differential-dataflow#802) tractable. The follow-up plan is a DD regression test running the #801 computation under seeded schedules against a sequential oracle, once this is released.
Workeris concrete over theAllocatorenum, so the allocator must live in-tree; the module is public and documented as testing infrastructure whose surface may evolve.Validation
The
#[ignore]dchaos_big_grindtakesGRIND_RUNSfor manual deep soaks; a scheduled CI job for it is a possible follow-up, deliberately not part of this PR.🤖 Generated with Claude Code