feat(benchmark): Vertex AI multi-model benchmark generation (#6913 foundation)#9638
Open
MarkusNeusinger wants to merge 11 commits into
Open
feat(benchmark): Vertex AI multi-model benchmark generation (#6913 foundation)#9638MarkusNeusinger wants to merge 11 commits into
MarkusNeusinger wants to merge 11 commits into
Conversation
Foundation for #6913 (LLM benchmark): generate the same (spec, library) implementation with multiple models served by Google Vertex AI — Gemini, Claude on Vertex, and Model Garden partners — under the existing Workload Identity Federation auth, so adding a model to the comparison is just another model id string. - automation/benchmark/: provider-routing Vertex client (AnthropicVertex for claude-*, OpenAI-compatible chat-completions endpoint for gemini-* and publisher-prefixed Model Garden ids), prompt builder reusing the exact pipeline context (base rules, style guide, library rules, spec), single-shot generation with render-error feedback, per-model result.yaml (provider, attempts, latency, tokens, canvas gate). - benchmark-generate.yml: workflow_dispatch over comma-separated model ids; results uploaded as artifacts + job summary table; nothing is committed to plots/ and the catalog pipeline is untouched. - Python libraries only in v1 (run on the workflow interpreter via the existing lib-* extras); R/Julia/JS reuse the setup actions later. - Unit tests for routing, prompt assembly, code extraction, CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an experimental Vertex AI–backed benchmark harness to generate the same (spec, library) implementation across multiple models (Gemini, Claude on Vertex, and Model Garden partners), with results captured as workflow artifacts for future benchmarking work (#6913).
Changes:
- Introduces
automation/benchmark/CLI + helpers for prompt assembly, provider routing (Anthropic vs Vertex OpenAI-compatible endpoint), generation retries, and rendering validation. - Adds a
benchmark-generate.ymlworkflow to dispatch multi-model runs and summarize per-model outcomes in the job summary. - Adds docs + changelog entry and new unit tests for provider routing, prompting, and CLI helpers.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
automation/benchmark/__init__.py |
Defines the benchmark module purpose/scope. |
automation/benchmark/generate.py |
CLI entrypoint for per-model generation + render loop + result.yaml writing. |
automation/benchmark/prompting.py |
Builds benchmark-mode prompts and extracts code from model responses. |
automation/benchmark/runner.py |
Executes generated Python implementations per theme and performs the canvas gate check. |
automation/benchmark/vertex_client.py |
Routes model ids to the right Vertex surface (Anthropic transport vs OpenAI-compatible endpoint). |
.github/workflows/benchmark-generate.yml |
Workflow dispatch to run benchmarks per model, upload artifacts, and write a summary table. |
tests/unit/automation/benchmark/__init__.py |
Creates the benchmark unit test package. |
tests/unit/automation/benchmark/test_generate.py |
Tests model slugging, CLI parsing, and allowed Python library list. |
tests/unit/automation/benchmark/test_prompting.py |
Tests prompt assembly, repair prompt behavior, and code extraction logic. |
tests/unit/automation/benchmark/test_vertex_client.py |
Tests provider routing, URL formation, and OpenAI-style content extraction. |
docs/workflows/benchmark.md |
Documents routing rules, inputs/outputs, and one-time GCP prerequisites. |
docs/workflows/overview.md |
Adds a pointer to the new benchmark workflow documentation. |
CHANGELOG.md |
Adds an Unreleased entry describing the benchmark foundation work. |
… tests Copilot review follow-ups on #9638: - The render subprocess executes LLM-generated code; it now runs with an allowlisted environment (PATH/HOME/locale/tmp only) instead of inheriting CI credentials (GCP/WIF, GitHub token). - Output directories slug the normalized model id so equivalent inputs ('gemini-2.5-pro' vs 'google/gemini-2.5-pro') share one folder. - Unit tests for check_canvas, render_env scrubbing, and run_python_implementation success/failure paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Copilot round-2 follow-ups on #9638: - Render subprocess runs with `python -I` (ignore PYTHON* env, user site-packages, script dir on sys.path) as defense in depth on top of the env allowlist; venv site-packages still resolve via pyvenv.cfg. - The always()-gated job-summary step now checks that the venv and benchmark-results/ exist before running, so an early workflow failure isn't obscured by a secondary summary error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
…ocstring Copilot round-3 follow-ups on #9638: - result.yaml keeps input/output tokens as null until a provider actually reports usage; a publisher omitting usage no longer records 0 ("free") and skews comparisons. Zero remains a real report. - resolve_provider docstring now states the intentional behavior: any publisher/model id routes to the OpenAI-compatible endpoint (Model Garden publishers keep growing); unknown publishers fail at call time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Copilot round-4 follow-up on #9638: HOME passed through the render env allowlist, giving LLM-generated code a path to ~/.config (gcloud credentials, tool configs) on CI runners. HOME now points at a per-render scratch directory inside the output dir; matplotlib's config/cache land there too. The second round-4 comment (add -P because -I allegedly keeps the script dir on sys.path) is factually wrong — verified empirically: under -I, sys.path[0] is the stdlib zip and a script named json.py imports stdlib json cleanly, while the non-isolated run reproduces the self-import collision. -I implies -P on Python >= 3.11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
…ilure retry Copilot round-5 follow-ups on #9638, both real bugs: - run_python_implementation deletes plot-{theme}.png before rendering, so a PNG left by a previous attempt can no longer make a later attempt look successful when it wrote nothing. - When extract_code_block fails, the repair prompt now feeds back THIS attempt's raw response instead of a stale earlier extraction, so the model sees what it actually returned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Copilot round-6 follow-ups on #9638: - The output contract said "no file system" while requiring the script to save plot-{theme}.png — contradictory instructions cost benchmark success rates. Reworded: no shell / no second turn stays; the PNG save is the one permitted file-system interaction. - Vertex client raises a clear RuntimeError with install instructions if google-auth is missing (it normally arrives transitively via google-cloud-storage, but the client imports it directly). - Local-run docs install google-auth explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Copilot round-7 follow-ups on #9638: - canvas_ok now requires BOTH theme renders to hit a canonical canvas size — a dark render with layout drift was previously masked by an on-target light render. - The lenient unfenced-code fallback in extract_code_block stays (the benchmark measures plot quality, not markdown discipline), but result.yaml now records code_fenced so contract compliance is measurable per model instead of silently ignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…ch gate) codecov/patch flagged 65.97% diff coverage (target 80%): the provider transports and the CLI main loop were untested. Adds mocked tests for - generate() routing + latency stamping, - _generate_anthropic (fake AnthropicVertex: text-block filtering, usage, region/project wiring), - _generate_openapi (payload/headers, usage parsing, non-200 error), - _access_token via fake ADC credentials, - main() end-to-end with a scripted fake client: success, extraction- failure retry, provider error, render-failure exhaustion — asserting the persisted result.yaml each time. Benchmark module coverage: 98%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Copilot round-9 follow-ups on #9638: - --max-attempts / --max-tokens / --render-timeout reject 0/negative values at parse time instead of silently writing an ambiguous result.yaml (attempts: 0, no error) from a loop that never ran. - Job-summary error bullets flatten multi-line tracebacks to one line (capped at 500 chars) so the Markdown list stays intact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Foundation for #6913 (LLM benchmark): generate the same (spec, library) implementation with multiple models served by Google Vertex AI — Gemini, Claude on Vertex, and Model Garden partners (Llama, Mistral, Qwen, DeepSeek, …) — so models can be compared side-by-side on identical inputs. One GCP authentication (the repo's existing Workload Identity Federation) covers every model family: adding a model to the comparison is just another model id string, no new secrets or SDK integrations.
What's included
automation/benchmark/— provider-routing Vertex client:claude-*/anthropic/*→ Anthropic SDK Vertex transport (AnthropicVertex)gemini-*/google/*and publisher-prefixed Model Garden ids (meta/*,mistralai/*, …) → Vertex's OpenAI-compatible chat-completions endpointmax_attempts) — deliberately model-neutral: no model gets an agent harness another lacksresult.yaml: provider, success, attempts, LLM latency, token usage, canvas-gate result.github/workflows/benchmark-generate.yml—workflow_dispatchover comma-separated model ids; per-model results uploaded as artifacts + a job-summary comparison table. Fails only when no model produces a working render (individual failures are benchmark signal, not CI errors).docs/workflows/benchmark.md(model routing, inputs, output layout, one-time GCP prerequisites) + pointer inoverview.md; CHANGELOG entry.Scope decisions (v1)
lib-*extras); R/Julia/JS reuse the existing setup actions in a later iteration.plots/— benchmark output never touches the catalog, metadata, or GCS; artifacts only. Persistence +/benchmarksite integration + scoring via the review rubric are the follow-up phases from LLM Benchmark: Compare model performance across specs and libraries on the site #6913.One-time GCP setup needed before first run
gcloud services enable aiplatform.googleapis.comroles/aiplatform.userVerification
pytest tests/unit/automation— 216 passed (30 new)ruff check ./ruff format --check— cleanpython -m automation.benchmark.generate --helpsmoke-tested; workflow YAML parse-checked.github/workflows/changes have no local verification loop — the workflow itself is only observable on a real dispatch, which also needs the GCP setup above.Refs #6913
🤖 Generated with Claude Code
https://claude.ai/code/session_01FppGJc99ryEDtBVn4jdewJ
Generated by Claude Code