benchmarking: add SLO-knee throughput post-processor - #883
Open
Alex Bulankou (AlexBulankou) wants to merge 1 commit into
Open
benchmarking: add SLO-knee throughput post-processor#883Alex Bulankou (AlexBulankou) wants to merge 1 commit into
Alex Bulankou (AlexBulankou) wants to merge 1 commit into
Conversation
Compute throughput-at-SLO-knee from a concurrency sweep of locust runs. For each op (Type_Name) in a build tag's sweep, the knee is the maximum sustained RPS among sweep points whose chosen latency percentile stays under a ceiling AND whose failure ratio is within tolerance. This maps directly onto the spec-doc substrate-row throughput axes (@<1s / @<5s == --ceiling-ms 1000 / 5000) with p50/p95 read straight off each point. - Pure stdlib over runner.py's stats.jsonl schema (no locust/pandas/numpy), so it runs offline over saved run artifacts. - Offered concurrency (-u user count) is absent from stats.jsonl; the knee is computed from (rps, latency) pairs and does not need it. Supplying a name->users map via --users-by-name / --tests-yaml labels each point's concurrency so the knee's location is interpretable. - Failure-ratio guard (default <=1%) prevents a fast-failing high-RPS run from being selected as usable capacity. - 18 unit tests + synthetic stats.jsonl fixtures (clean ceiling-crossing sweep, fast-failing exclusion, SLO-never-met, percentile normalization, multi-metric grouping, CLI e2e). Runnable via pytest or a built-in stdlib runner (no pytest dependency required). Signed-off-by: Alex Bulankou <alexbu@google.com>
a4-a4s1
Bot
force-pushed
the
a4s1/benchmarking-slo-knee-postproc
branch
from
August 12, 2026 09:15
281e4ee to
e41255f
Compare
Collaborator
Author
|
Design question from exercising this against live sweep output: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pure-stdlib post-processor over
runner.py'sstats.jsonl. For each gRPC op in a build tag's concurrency sweep it computes the SLO-knee throughput: the max sustained RPS among sweep points whose chosen latency percentile stays under a ceiling AND whose failure ratio is within tolerance. Past the knee, offered load keeps climbing but latency has blown past the SLO, so that extra throughput isn't usable capacity.Gives the sweep a single honest "throughput @ pXX<SLO" number per op (e.g.
--ceiling-ms 1000/5000) instead of eyeballingstats_history.csv.Offline-only:
benchmarking/analysis/ships tests + 3 syntheticstats.jsonlfixtures —python3 -m pytest analysis/ -qruns 18/18 with no cluster.Note: the fixtures are the repo's first
.jsonlfiles, so this also adds.jsonlto the data-extension exemption inhack/util/verify-boilerplate.py(alongside.json) —verify-all.shruns clean with it.