update sdk with new adds

[luke-e-schaefer]

Summary — v0.19.0: EvaluationV2, benchmark paradigm

Brings the SDK to parity with the Nucleus EvaluationV2 UI. Evaluation creation is benchmark-scoped only — dataset/slice-scoped creation (an unreleased earlier iteration of this branch) was removed before release as the platform moved to the benchmark paradigm.

Added

• Benchmarks. create_benchmark() (members from item_ids, (dataset_id, ref_id) items pairs, a slice_id, or a dataset_id — exactly one required; membership frozen at creation), list_benchmarks(), get_benchmark(), update_benchmark(), delete_benchmark(), list_benchmark_items(), plus the Benchmark resource (refresh() / update() / delete() / items() / create_evaluation_v2()) in nucleus/benchmark.py.
• Benchmark evaluations. create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...) — every benchmark item is scored (uncovered items count as FN, keeping leaderboard scores comparable). EvaluationV2 exposes benchmark_id, rollup_groups, exclusion_rules, exclusion_stats. No only_items_with_predictions / eval-scope slice_id (invalid in the benchmark paradigm; the server rejects the former).
• Rollup groups. RollupGroup (class_name + labels) as the primary label configuration on benchmark eval create and presets (create/update_evaluation_v2_preset(rollup_groups=...), mutually exclusive with legacy allowed_label_matches; parsed from both key casings). Preset seeding prefers rollup_groups, falling back to legacy matches.
• Exclusion rules. MetadataExclusionRule / LabelExclusionRule / BoxAreaExclusionRule via exclusion_rules on eval create and presets.
• Presets. list/create/update/delete_evaluation_v2_preset + EvaluationV2Preset resource; preset= seeds benchmark eval creation (explicit args override).
• Results. EvaluationV2.charts() (mAP, per-class AP, confusion matrix, PR/F1 curves, TIDE, AP by size) and examples() (paginated TP/FP/FN; match_type optional) with EvaluationV2FilterArgs filtering; cancel() / retry(); Dataset.evaluation_label_schema().
• Benchmark leaderboards. leaderboard_ranking(metric_type, benchmark_ids, ...) (metrics: MAP_50, MAP_50_95, AP_SMALL/MEDIUM/LARGE, PRECISION, RECALL, F1; scope/collapse) and leaderboard_f1_curve(benchmark_ids, ..., top_n=5).
• Filter schema discovery. EvaluationV2.filter_schema() / get_evaluation_v2_filter_schema() return the evaluation's filter vocabulary (gt_labels, pred_labels, metadata_fields with inferred value types).
• New DTOs: EvaluationV2FilterSchema, LeaderboardRankingEntry, LeaderboardF1CurveEntry, BenchmarkItemsPage.

Removed (relative to earlier commits on this branch — never released)

• create_evaluation_v2 (dataset/slice-scoped), create_evaluations_v2_batch, BatchEvaluationResult, only_items_with_predictions. Reading existing evals (including old slice-scoped ones) still works.

Fixed

• CI build_test pylint failure (pre-existing W0718 on the per-job catch) — suppressed with a black-stable # pylint: disable-next line.

Tests / Version

• tests/test_benchmarks.py, tests/test_leaderboard.py, plus rollup/benchmark coverage in the eval/preset test files (59 unit tests, mock-based).
• pyproject.toml → 0.19.0; single consolidated CHANGELOG entry (the unreleased 0.18.9 section was folded in).

Server dependency: the leaderboard and filter-schema methods call new REST mirrors (POST /nucleus/leaderboard/ranking, POST /nucleus/leaderboard/f1Curve, GET /nucleus/evaluationsV2/:id/filterSchema) that ship separately in scaleapi. Unit tests pass regardless; live calls 404 until that deploys. All benchmark CRUD / benchmark-eval / rollup-group / preset endpoints are already live (DE-8209).

🤖 Generated with Claude Code

Greptile Summary

This PR introduces the benchmark evaluation paradigm for v0.19.0: frozen benchmark sets, benchmark-scoped evaluations, rollup groups, exclusion rules, evaluation presets, cancel/retry, leaderboard ranking, and filter-schema discovery. The former NucleusClient.create_evaluation_v2 is intentionally removed (noted in the CHANGELOG as an unreleased surface that was dropped in favour of the benchmark paradigm) and replaced by create_benchmark_evaluation_v2.

• Benchmarks: full CRUD (create_benchmark, list_benchmarks, get_benchmark, update_benchmark, delete_benchmark, list_benchmark_items) plus the Benchmark resource with create_evaluation_v2().
• Evaluation V2 upgrades: charts() switched from GET to POST (fixing the backend route mismatch), match_type in examples() made optional, cancel()/retry()/filter_schema() added, and EvaluationV2 gains benchmark_id, rollup_groups, slice_id, exclusion_rules, and exclusion_stats.
• Presets & leaderboards: EvaluationV2Preset with full CRUD and preset-seeding on eval create; leaderboard_ranking() and leaderboard_f1_curve() backed by new server endpoints (REST mirrors not yet deployed — unit tests pass regardless).

Confidence Score: 5/5

Safe to merge — the benchmark paradigm is well-tested (66 unit tests), the breaking removal of create_evaluation_v2 is explicitly documented in the CHANGELOG, and no existing functionality is silently changed.

All changes are additive or intentionally breaking with a major-minor version bump and explicit CHANGELOG documentation. The three findings are minor inconsistencies (an or vs is not None guard, an orphaned test helper, and non-Optional required DTO fields) that are unlikely to manifest in practice.

nucleus/evaluation_v2_preset.py (inconsistent null-guard on allowed_label_matches) and tests/test_evaluation_v2_presets.py (orphaned _stub_create helper).

Important Files Changed

Filename	Overview
nucleus/init.py	Large addition of benchmark CRUD, benchmark evaluation create, leaderboard, preset management, and filter-schema methods; create_evaluation_v2 removed (replaced by create_benchmark_evaluation_v2); _UNSET sentinel imported at module level (minor namespace leak).
nucleus/benchmark.py	New Benchmark dataclass with CRUD delegation to NucleusClient; from_json correctly maps benchmark_id → id; all instance methods guard against missing _client.
nucleus/evaluation_v2.py	Adds RollupGroup, parse_rollup_groups, parse_allowed_label_matches, _parse_json_field; EvaluationV2 gains rollup_groups, benchmark_id, slice_id, exclusion_rules, exclusion_stats; charts() correctly switched from GET to POST; cancel()/retry()/filter_schema() added; match_type in examples() made optional.
nucleus/evaluation_v2_preset.py	New EvaluationV2Preset with camelCase+snake_case fallback parsing; allowed_label_matches uses or fallback (inconsistent with is not None used for rollup_groups/exclusion_rules).
nucleus/evaluation_v2_exclusions.py	New file with MetadataExclusionRule, LabelExclusionRule, BoxAreaExclusionRule dataclasses; serialization is clean and server-side validation is relied on for bound checks.
nucleus/data_transfer_object/evaluation_v2.py	Adds gt_area_range/slice_ids filter fields, EvaluationV2FilterSchema (all fields required, may fail on not-yet-computed evaluations), and new leaderboard/benchmark DTOs.
tests/test_evaluation_v2_presets.py	New test file; contains orphaned _stub_create helper and empty section stubs from a removed feature iteration.
tests/test_benchmarks.py	Comprehensive unit tests for benchmark CRUD, evaluation create, preset seeding, and mutual-exclusion guards; all critical paths covered.
tests/test_leaderboard.py	Unit tests for leaderboard ranking, F1 curve, and filter-schema endpoints; request payloads and response parsing verified.
nucleus/connection.py	Adds patch() method mirroring post(), correctly delegating to make_request with requests.patch.
nucleus/dataset.py	Adds evaluation_label_schema() delegating to dataset/{id}/labelSchema via GET; straightforward addition.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant User
    participant NucleusClient
    participant Server

    User->>NucleusClient: "create_benchmark(name, slice_id=...)"
    NucleusClient->>Server: POST /benchmarks
    Server-->>NucleusClient: benchmark_id bm_1
    NucleusClient-->>User: Benchmark

    User->>NucleusClient: create_evaluation_v2_preset(name, rollup_groups)
    NucleusClient->>Server: POST /evaluationV2Presets
    Server-->>NucleusClient: id prev_1
    NucleusClient-->>User: EvaluationV2Preset

    User->>NucleusClient: create_benchmark_evaluation_v2(bm_id, run_id, preset)
    NucleusClient->>Server: POST /benchmarks/bm_id/evaluationsV2
    Server-->>NucleusClient: evaluation_id evalv2_1
    NucleusClient->>Server: GET /evaluationsV2/evalv2_1
    Server-->>NucleusClient: EvaluationV2 row
    NucleusClient-->>User: EvaluationV2

    User->>NucleusClient: evaluation.wait_for_completion()
    loop Poll until terminal status
        NucleusClient->>Server: GET /evaluationsV2/evalv2_1
        Server-->>NucleusClient: status computing
    end
    Server-->>NucleusClient: status succeeded

    User->>NucleusClient: "evaluation.charts(iou_threshold=0.5)"
    NucleusClient->>Server: POST /evaluationsV2/id/charts
    Server-->>NucleusClient: EvaluationV2Charts

    User->>NucleusClient: leaderboard_ranking(MAP_50, benchmark_ids)
    NucleusClient->>Server: POST /leaderboard/ranking
    Server-->>NucleusClient: LeaderboardRankingEntry list
    NucleusClient-->>User: List LeaderboardRankingEntry

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant User
    participant NucleusClient
    participant Server

    User->>NucleusClient: "create_benchmark(name, slice_id=...)"
    NucleusClient->>Server: POST /benchmarks
    Server-->>NucleusClient: benchmark_id bm_1
    NucleusClient-->>User: Benchmark

    User->>NucleusClient: create_evaluation_v2_preset(name, rollup_groups)
    NucleusClient->>Server: POST /evaluationV2Presets
    Server-->>NucleusClient: id prev_1
    NucleusClient-->>User: EvaluationV2Preset

    User->>NucleusClient: create_benchmark_evaluation_v2(bm_id, run_id, preset)
    NucleusClient->>Server: POST /benchmarks/bm_id/evaluationsV2
    Server-->>NucleusClient: evaluation_id evalv2_1
    NucleusClient->>Server: GET /evaluationsV2/evalv2_1
    Server-->>NucleusClient: EvaluationV2 row
    NucleusClient-->>User: EvaluationV2

    User->>NucleusClient: evaluation.wait_for_completion()
    loop Poll until terminal status
        NucleusClient->>Server: GET /evaluationsV2/evalv2_1
        Server-->>NucleusClient: status computing
    end
    Server-->>NucleusClient: status succeeded

    User->>NucleusClient: "evaluation.charts(iou_threshold=0.5)"
    NucleusClient->>Server: POST /evaluationsV2/id/charts
    Server-->>NucleusClient: EvaluationV2Charts

    User->>NucleusClient: leaderboard_ranking(MAP_50, benchmark_ids)
    NucleusClient->>Server: POST /leaderboard/ranking
    Server-->>NucleusClient: LeaderboardRankingEntry list
    NucleusClient-->>User: List LeaderboardRankingEntry

Loading

_{Reviews (7): Last reviewed commit: "Remove deprecated dataset/slice-scoped e..." | Re-trigger Greptile}