feat(ruby): eval suite — Ruby ports of all 77 bench tasks + runner - #2861
Draft
filip-michalsky wants to merge 6 commits into
Draft
feat(ruby): eval suite — Ruby ports of all 77 bench tasks + runner#2861filip-michalsky wants to merge 6 commits into
filip-michalsky wants to merge 6 commits into
Conversation
- evals/harness.rb + evals/runner.rb: bench-tier harness mirroring packages/evals — tasks auto-discovered from evals/tasks/<category>/, fresh Browserbase session per run (one shared extension upload per invocation: per-session uploads trip the POST /v1/extensions rate limit), self_heal: true like the TS evals, concurrency/trials/targets CLI, JSONL results + per-category summary - evals/tasks: 1:1 semantic ports of all 77 bench tasks (40 act, 25 extract, 12 observe) — same URLs, same assertions; zod schemas as JSON Schema hashes (z.string().url() => format: "uri", which the extension resolves to absolute URLs); framework helpers (normalizeString, Jaro-Winkler compareStrings, matchingSelector) inlined where the TS imported them - SDK: act/observe/extract gain locator:/ignore_locators: (protocol options TS/Python already exposed; the eval tasks use them) with page-ownership validation, and options now encode through their wire models; RBS updated; 3 new unit tests (123 runs total) Results on Browserbase (Model Gateway): 73/77 pass. The 4 remaining are not Ruby defects — probed individually: extract_apartments (Akamai Access Denied) and observe_taxes (bot interstitial) are IP-walled for this infra, observe_vantechjournal's hardcoded expected xpath no longer exists on the live site, and act/vantechjournal is a navigation-timing race shared with the TS task (the SDK clicks the right link and the navigation completes; the task samples page.url before it commits). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This was referenced Aug 30, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rom the repo root) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…old CI runners) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e the byte assertion) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… cold CI runners) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the AP-2857 parity stack:
Stack: … ← #2858 (M6a) ← #2859 (M6c) ← #2860 (M6d) ← this PR (evals)
Ruby ports of the entire bench-tier eval set from
packages/evals/tasks/bench/— all 77 tasks (40 act, 25 extract, 12 observe), each a 1:1 semantic port (same URLs, same assertions) — plus a runner (packages/sdk-ruby/evals/): auto-discovery, fresh Browserbase session per run, one shared extension upload per invocation (per-session uploads trip thePOST /v1/extensionsrate limit),self_heal: truelike the TS harness, trials/concurrency/target selection, JSONL results + per-category summary.SDK changes the port forced
act/observe/extractgainlocator:/ignore_locators:— protocol options TS/Python already exposed and the eval tasks use — with page-ownership validation; call options now encode through their wire models (ActOptions/ObserveOptions/ExtractOptions). RBS updated; suite at 123 runs / 807 assertions.Results (Browserbase + Model Gateway)
The 4 failures were probed individually and are not Ruby defects:
extract/extract_apartments— apartments.com returns Akamai Access Denied to this infraobserve/observe_taxes— 1040.com serves a bot-check interstitial; the form never rendersobserve/observe_vantechjournal— site drift: the hardcoded expected XPath no longer exists on the live pageact/vantechjournal— navigation-timing race shared with the TS task: the SDK clicks the correct link and navigation completes (verified:page.urlreads/recommendations3s later), but the task samplespage.urlbefore the navigation commitsPorting note that generalizes:
z.string().url()fields must reach the wire as"format": "uri"(zod's JSON-Schema interchange) for the extension to resolve element references into absolute URLs — plain"type": "string"returns reference ids.Out of scope: dataset-backed suites (WebVoyager, OnlineMind2Web, WebTailBench), agent-tier tasks, Braintrust reporting.
🤖 Generated with Claude Code