benchmark: refuse to publish a run that mostly answered 5xx - #1252
Open
MDA2AV wants to merge 1 commit into
Open
Conversation
There was no error gate anywhere. save_result wrote the JSON unconditionally and status_5xx was recorded but never acted on, so a run could fail almost entirely and still publish a number. That is worse than it sounds, because an error is cheaper to serve than a real response: a server that fails fast scores HIGHER than one that works. php-fpm published 294,864 rps for a baseline-h2 run in which 1,462,506 of 1,474,323 responses - 99.2% - were 5xx, and that number counts toward its composite. save_result now computes the 5xx share of all responses and refuses to write the result above HTTPARENA_MAX_ERROR_PCT, default 5. The container log is still written so the run can be diagnosed, and any stale JSON for that profile/conns is removed. Seven already-published runs are above the threshold and are dropped here: php-fpm baseline-h2-1024 99.2% was 294864 rps symfony-spawn-franken baseline-h2-256 72.9% was 28243 rps symfony-spawn-franken static-h2-1024 56.5% was 52582 rps symfony-spawn-franken static-h2-256 47.3% was 63518 rps symfony-spawn-franken baseline-h2-1024 36.5% was 97817 rps rage api-16-1024 15.2% was 9825 rps rage api-4-256 8.8% was 13999 rps Those three frameworks now score 0 on the dropped profiles, the same as any entry that does not publish one. The 24 other runs carrying some 5xx are all under 5% and are left alone; the worst of them is hono-bun's fortunes at 3.7%. badge parity ok - 751 ranks match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the hono-bun log in #1251.
There is no error gate anywhere.
save_resultwrites the JSONunconditionally;
status_5xxis recorded and never acted on, so a run can failalmost entirely and still publish a number.
That is worse than it sounds, because an error is cheaper to serve than a real
response — a server that fails fast scores higher than one that works.
php-fpmpublishes 294,864 rps for abaseline-h2run in which1,462,506 of 1,474,323 responses — 99.2% — were 5xx, and that number counts
toward its composite score.
Fix
save_resultcomputes the 5xx share of all responses and refuses to write theresult above
HTTPARENA_MAX_ERROR_PCT(default 5). The container log is stillwritten so the run can be diagnosed, and any stale JSON for that profile/conns is
removed.
Blast radius
Seven already-published runs are above the threshold and are dropped here:
Those three frameworks now score 0 on the dropped profiles, the same as any entry
that does not publish one. php-fpm and symfony-spawn-franken lose most of their
h2 numbers, which is the intended outcome — those runs were not results.
The 24 other runs carrying some 5xx are all under 5% and are left alone; the
worst is hono-bun's
fortunesat 3.7%.site/leaderboard/data.jsregenerated;check_badge_parity.jspasses —751 ranks match.
Judgement call
5% is the threshold the codebase already documented in a comment that no longer
matched any code. If you would rather draw the line somewhere else, it is one env
var, and re-running the purge is a one-liner.
🤖 Generated with Claude Code