Skip to content

benchmark: refuse to publish a run that mostly answered 5xx - #1252

Open
MDA2AV wants to merge 1 commit into
mainfrom
fix/reject-failed-runs
Open

benchmark: refuse to publish a run that mostly answered 5xx#1252
MDA2AV wants to merge 1 commit into
mainfrom
fix/reject-failed-runs

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Follow-up to the hono-bun log in #1251.

There is no error gate anywhere. save_result writes the JSON
unconditionally; status_5xx is recorded and never acted on, so a run can fail
almost entirely and still publish a number.

That is worse than it sounds, because an error is cheaper to serve than a real
response
— a server that fails fast scores higher than one that works.

php-fpm publishes 294,864 rps for a baseline-h2 run in which
1,462,506 of 1,474,323 responses — 99.2% — were 5xx, and that number counts
toward its composite score.

Fix

save_result computes the 5xx share of all responses and refuses to write the
result above HTTPARENA_MAX_ERROR_PCT (default 5). The container log is still
written so the run can be diagnosed, and any stale JSON for that profile/conns is
removed.

Blast radius

Seven already-published runs are above the threshold and are dropped here:

framework profile err% was
php-fpm baseline-h2-1024 99.2% 294,864 rps
symfony-spawn-franken baseline-h2-256 72.9% 28,243 rps
symfony-spawn-franken static-h2-1024 56.5% 52,582 rps
symfony-spawn-franken static-h2-256 47.3% 63,518 rps
symfony-spawn-franken baseline-h2-1024 36.5% 97,817 rps
rage api-16-1024 15.2% 9,825 rps
rage api-4-256 8.8% 13,999 rps

Those three frameworks now score 0 on the dropped profiles, the same as any entry
that does not publish one. php-fpm and symfony-spawn-franken lose most of their
h2 numbers
, which is the intended outcome — those runs were not results.

The 24 other runs carrying some 5xx are all under 5% and are left alone; the
worst is hono-bun's fortunes at 3.7%.

site/leaderboard/data.js regenerated; check_badge_parity.js passes —
751 ranks match.

Judgement call

5% is the threshold the codebase already documented in a comment that no longer
matched any code. If you would rather draw the line somewhere else, it is one env
var, and re-running the purge is a one-liner.

🤖 Generated with Claude Code

There was no error gate anywhere. save_result wrote the JSON unconditionally and
status_5xx was recorded but never acted on, so a run could fail almost entirely
and still publish a number.

That is worse than it sounds, because an error is cheaper to serve than a real
response: a server that fails fast scores HIGHER than one that works. php-fpm
published 294,864 rps for a baseline-h2 run in which 1,462,506 of 1,474,323
responses - 99.2% - were 5xx, and that number counts toward its composite.

save_result now computes the 5xx share of all responses and refuses to write the
result above HTTPARENA_MAX_ERROR_PCT, default 5. The container log is still
written so the run can be diagnosed, and any stale JSON for that profile/conns
is removed.

Seven already-published runs are above the threshold and are dropped here:

  php-fpm                baseline-h2-1024   99.2%   was 294864 rps
  symfony-spawn-franken  baseline-h2-256    72.9%   was  28243 rps
  symfony-spawn-franken  static-h2-1024     56.5%   was  52582 rps
  symfony-spawn-franken  static-h2-256      47.3%   was  63518 rps
  symfony-spawn-franken  baseline-h2-1024   36.5%   was  97817 rps
  rage                   api-16-1024        15.2%   was   9825 rps
  rage                   api-4-256           8.8%   was  13999 rps

Those three frameworks now score 0 on the dropped profiles, the same as any
entry that does not publish one. The 24 other runs carrying some 5xx are all
under 5% and are left alone; the worst of them is hono-bun's fortunes at 3.7%.

badge parity ok - 751 ranks match.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant