benchmarks: durdir benchmark that evaluates the efficacy and performance of DurDir by sairajp-rewind · Pull Request #907 · agent-substrate/substrate

Sairaj Pokale (sairajp-rewind) · 2026-08-13T01:47:35Z

Benchmark DurDir

Fixes #673

Adds a DurDirUser load-generation workload that exercises the DurableDir
suspend/resume loop end to end, verifies every served byte against a SHA-256
digest, and emits separable latency percentiles for each step.

The loop

per VU, first iteration:  create -> resume -> WriteDisk -> ReadDisk+verify
steady state:             suspend -> resume -> ReadDisk+verify (cold)
                                  -> ReadDisk+verify (warm)
                                  -> WriteDisk (overwrite, TRUNCATE)

Under onCommit: Data the container cold-boots from the OCI image and process
memory is discarded, so a matching digest after resume can only have come from
the restored DurableDir. That is the durability assertion.

Results

All six scenarios ran on GKE/gvisor at 1 VU for 1m each: Data vs Full
snapshot scope, explicit vs implicit resume, and a 5/10/64 MiB size sweep.
Zero failures across every run, so every served byte matched its digest in
all six.

Snapshot growth over repeated overwrites: SuspendActor latency stayed flat
across consecutive overwrite-and-suspend cycles on the same DurableDir volume.
The loop overwrites with WRITE_MODE_TRUNCATE, so the file is exactly X bytes
after every write and the captured directory contents are the same size every
cycle. Nothing accumulates across suspends. That is the growth question the
issue asks about.

Latency percentiles per step are in the run artifacts.

Change surface

glutton: WriteDisk returns size + sha256; new ReadDisk with a
READ_MODE_DIGEST_ONLY mode for measuring restore cost without paying wire
transfer; disk RPCs exposed over HTTP mode.
manifests: two new ActorTemplates, glutton-durdir-{data,full}, with a
durableDir volume and onPause: Full / onCommit: {Data,Full}.
boomer: shared actor-lifecycle plumbing extracted from the ping task, then
a DurDirUser task on top of it; a general resume_mode knob.
harnesses: --workload selects the task at deploy time; durdir.py stub,
typed dynconfig flags, six nightly scenarios.

The two boomer commits above are incremental extractions made as the second
workload landed. The final package layout for internal/benchmarking/boomer
lands as a follow-up PR.

Testing

go build ./..., go test -race ./cmd/benchmarking/... ./internal/benchmarking/...
clean, no race warnings.
hack/verify-all.sh: all nine checks pass. Python protos regenerated, tree
clean.
Both ActorTemplates reach Ready; golden snapshots confirmed in the bucket.
Manual durability proof: 200 MiB payload written, suspended, resumed, and
re-read with matching sha256 across every read. Process memory discarded and
container cold-booted in between.
Scale validation: DurableDir persistence verified up to 1 GiB with 0
failures.
Regression gate: glutton_baseline_5_users ran 919 requests with 0
failures and 7 ms ping latency post-rebase. No regression on the existing
benchmark.

Deliberately out of scope

Image size over time: ActorSnapshot has no size field, so there is
nothing for a client to read. The issue permits deferring this. SuspendActor
latency is the available proxy; atelet.snapshot.size is the server-side one.
boomer package restructure: landing as a follow-up so this PR's files stay
reviewable in place.
RAM-backed variant: glutton already has WriteRAM; small follow-up.

Max Smythe (maxsmythe)

Thank you for this.

Left some comments.

Also, we should avoid sending 4k line PRs where possible -- they take multiple hours to review closely.

Can we avoid refactoring anything under internal/benchmarking/boomer for now? I don't think the abstractions are quite right and the refactoring is a large portion of the delta. We can implement ~3-4 tests and then figure out what the best refactor is.

Max Smythe (maxsmythe) · 2026-08-13T03:57:16Z

+            help="Size in bytes written to /var/lib/glutton/bench-data (default 8 MiB)",
+            include_in_web_ui=True,
+        )
+        parser.add_argument(


If this applies to any workload, it should probably live in its own Python file

Split out. --resume-mode now lives in common/resume_mode.py

Max Smythe (maxsmythe) · 2026-08-13T04:03:04Z

+# if those disagree the stats rows describe traffic that was never sent.
+# The value must name a stub in locust/tests/, a --workload case in
+# cmd/benchmarking/boomer-glutton/main.go, and an entry in runner.py's
+# BOOMER_WORKLOADS dict.


TODO: look to reduce # of sources of truth

Max Smythe (maxsmythe) · 2026-08-13T04:04:07Z

    return args


+def boomer_workload(test_file: str) -> str | None:


It would be better to test for Python workloads... I suspect we will no longer be adding those, so it will be a static list. A cleanup work item will be to remove those tests.

This will also remove one place workload lists need to be kept in alignment

Went with a single static map from stub filename to user class:

BOOMER_USER_CLASSES = {"glutton.py": "GluttonUser", "durdir.py": "DurDirUser"}

needs_boomer is membership in it and the same entry supplies --user-class, so the filename and the class name stay in one place rather than two.
Kept the check on the boomer side rather than the Python side because runner.py needs the class name string either way

Max Smythe (maxsmythe) · 2026-08-13T04:10:48Z

+  python3 runner.py -f tests/durdir.py -t 1m -u 1 --name durdir_run --dest /tmp/bench \
+    --durdir-template glutton-durdir-data --resume-mode explicit --durdir-file-size-bytes 8388608
+  ```
+* **Interactive Web UI**: The stack runs one workload per deployment, chosen at deploy time:


I don't think "workload" is correct... that is the name given to the ActorTemplate and workers.

One "test type"? "Virtual user type"?

Max Smythe (maxsmythe) · 2026-08-13T04:13:49Z

+  * `glutton-durdir-data` (default): Attaches a durable data directory under `/var/lib/glutton` without memory snapshot restore.
+  * `glutton-durdir-full`: Attaches a durable data directory and performs a full memory snapshot restore.
+
+#### File-Size Sweeps


I don't think we need to document that we can run multiple tests by adding multiple tests. Also, I think setting the read mode to digest would be orthogonal to "sweep".

I'd remove this section.

Max Smythe (maxsmythe) · 2026-08-13T04:38:45Z

 	cfg          *Config
 	actorName    string
 	hostHeader   string
+	userClass    string


I like this idea, though we will probably need some design/refactoring to make this truly generic across all users. We'll tackle that more on a later test.

Max Smythe (maxsmythe) · 2026-08-13T04:39:28Z


+// resume brings the actor up before the next request.
+//
+// The first resume is always explicit, whatever the mode. It is the only


This is far too much context for a comment. Documents the code and likely to rot quickly.

Max Smythe (maxsmythe) · 2026-08-13T04:48:34Z

+	user           *gluttonUser
+	expectedDigest string
+	fileSize       int64
+	readMode       gluttonpb.ReadMode


Things that are stored as dynamic config should not be persisted to the class, they should be read from the dynamic config, so behavior can change dynamically (might need to have some read atomicity to avoid breaking behavior due to time aliasing)

Fixed, including the aliasing half.

Max Smythe (maxsmythe) · 2026-08-13T04:50:52Z

+	slog.Info("configured durdir file size", slog.Int64("bytes", fileSize), slog.String("template", tmpl))
+
+	if readMode == gluttonpb.ReadMode_READ_MODE_DATA && fileSize > 64*1024*1024 {
+		slog.Warn("durdir_file_size_bytes > 64MB with durdir_read_mode=data may hit router or Envoy timeouts; set durdir_read_mode=digest",


No need to warn about this... if we are hitting timeouts that would be the point of the benchmark, to find out.

Max Smythe (maxsmythe) · 2026-08-13T05:11:21Z

 #     its own per-worker diagnostics at /metrics on :8001 (aggregate stats
 #     flow through the master via boomer.RecordSuccess).
+#
+# ${BENCHMARK_WORKLOAD} is substituted by benchmarking/locust/deploy.sh


Comment describes too many internal details

Benjamin Elder (BenTheElder) · 2026-08-13T18:30:49Z

let's squash this on merge (and more PRs, generally, unless there's really useful commit history)

Sairaj Pokale (sairajp-rewind) requested a review from Max Smythe (maxsmythe) August 13, 2026 01:47

Max Smythe (maxsmythe) requested changes Aug 13, 2026

View reviewed changes

Sairaj Pokale (sairajp-rewind) force-pushed the benchmarks/durdir-673 branch from 170d93b to b7e91e5 Compare August 13, 2026 22:14

Sairaj Pokale (sairajp-rewind) added 7 commits August 13, 2026 23:20

glutton: return size and digest from WriteDisk, add ReadDisk

3626dbd

glutton: expose disk RPCs over HTTP mode

398598a

benchmarking: add DurDir-backed glutton ActorTemplates

fbac3bd

boomer: add the DurDir benchmark task

1af3a4f

boomer: fail loudly on an invalid dynconfig value

a46a1a2

benchmarking: wire the DurDir user class into both harnesses

198523e

benchmarking: add DurDir nightly scenarios and docs

98916bc

Sairaj Pokale (sairajp-rewind) force-pushed the benchmarks/durdir-673 branch from b7e91e5 to 98916bc Compare August 13, 2026 23:33

		return args


		def boomer_workload(test_file: str) -> str \| None:

Uh oh!

Conversation

Sairaj Pokale (sairajp-rewind) commented Aug 13, 2026

Benchmark DurDir

The loop

Results

Change surface

Testing

Deliberately out of scope

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants