Add extensible free-space motion benchmark

[yuecideng]

Description

This PR implements the first three benchmark steps discussed in #437:

• refactors the monolithic NeuralPlanner benchmark into configuration, adapter registry, runner, scenario, metric, artifact, aggregation, and reporting modules;
• adds free-space-common smoke and coverage suites with deterministic Franka case manifests;
• uses cuRobo as the primary baseline and keeps IK interpolation and TOPPRA as optional diagnostic baselines;
• leaves NMG as an explicit unsupported adapter stub while making its internal pos_eps and rot_eps independently configurable from suite-owned external validation thresholds;
• recomputes ordered waypoint, motion-validity, joint-limit, path-quality, latency, throughput, and memory metrics outside the planner timing boundary;
• writes resolved configuration, environment metadata, case manifest, numeric JSONL, aggregates, and one Markdown report containing exactly three tables;
• adds an explicit cuRobo backend preparation hook so construct, prepare, cold-plan, and warm-plan costs are measured separately.

Atomic-task remains out of scope for this PR and can be added through the extension points in a follow-up.

Dependencies: adds psutil>=5.9 for process RSS measurement.

Partially addresses #437.

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

Not applicable; this change provides a CLI benchmark and generated Markdown/JSON artifacts.

Validation

• black . — 580 files unchanged
• black --check --diff --color ./
• python -m compileall -q embodichain/lab/sim/planners/curobo/curobo_planner.py scripts/benchmark/__main__.py scripts/benchmark/planners/neural_planner tests/benchmark/planners/test_motion_generation_benchmark.py
• pytest -q tests/benchmark tests/sim/planners/test_curobo_planner.py -m 'not gpu and not slow' — 51 passed, 3 deselected
• manual minimal end-to-end runs: cuRobo on CUDA, IK interpolation on CPU with B=1/2, and TOPPRA on CPU

The nightly-scale coverage matrix was not run locally.

Checklist

• I have run the black . command to format the code base.
• I have made corresponding changes to the documentation
• I have added tests that prove the feature works.
• Dependencies have been updated, if applicable.

[Copilot]

Pull request overview

This PR replaces the previous monolithic NeuralPlanner benchmark with a new, extensible motion-generation benchmark framework centered on free-space cases and cuRobo as the primary baseline. It introduces suite-driven configuration, a registry-based adapter/scenario architecture, a generic runner with lifecycle timing phases, external validation metrics (ordered waypoint matching + motion-validity gate), and standardized artifacts/reporting (including a single Markdown report with exactly three tables).

Changes:

• Refactor benchmark implementation into modular motion-generation components (config/registry/runner/scenarios/metrics/aggregation/artifacts/reporting) and wire a new embodichain benchmark motion-generation CLI subcommand.
• Add deterministic free-space-common smoke/coverage suite YAMLs and a free_space scenario provider that generates stable manifests and start-state bins.
• Extend planners to support the benchmark lifecycle and cleanup: add CuroboPlanner.prepare_backend() for explicit backend warmup timing, and add ToppraPlanner.close() so adapters can reliably release worker pools.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/benchmark/planners/test_neural_planner_benchmark.py	Removes tests tied to the deleted monolithic NeuralPlanner benchmark implementation.
tests/benchmark/motion_generation/test_motion_generation_benchmark.py	Adds comprehensive unit tests covering ordered waypoint matching, outcome/aggregation semantics, reporting constraints (3 tables), adapter lifecycle, and runner gating.
tests/benchmark/motion_generation/init.py	Introduces test package for the new motion-generation benchmark suite.
scripts/benchmark/planners/neural_planner/run_benchmark.py	Removes the legacy monolithic NeuralPlanner benchmark runner.
scripts/benchmark/motion_generation/init.py	Adds package marker for the new motion-generation benchmark modules.
scripts/benchmark/motion_generation/run_benchmark.py	Adds the new CLI-compatible entry point supporting suite selection and overrides.
scripts/benchmark/motion_generation/config.py	Adds suite/config models, YAML loading, hashing utilities, and validation for free-space track parameters.
scripts/benchmark/motion_generation/registry.py	Adds registries for planner adapters and scenario providers.
scripts/benchmark/motion_generation/runner.py	Adds a generic benchmark runner with lifecycle timing, capability gating, artifact writing, aggregation, and report generation.
scripts/benchmark/motion_generation/aggregation.py	Adds aggregation logic for Time & Memory, Success & Metrics, and Leaderboard tables with case-macro averaging semantics.
scripts/benchmark/motion_generation/reporting.py	Adds Markdown reporting that always renders exactly three tables with stable schemas and formatted values.
scripts/benchmark/motion_generation/artifacts.py	Adds reproducibility artifacts: resolved suite YAML, environment metadata, case manifest, JSON aggregates, and JSONL trial records.
scripts/benchmark/motion_generation/metrics/init.py	Exposes metric helpers for performance timing and trajectory validation.
scripts/benchmark/motion_generation/metrics/performance.py	Adds timed-call measurement including CPU RSS (psutil), GPU allocator deltas, and optional peak GPU tracking.
scripts/benchmark/motion_generation/metrics/stats.py	Adds shared nearest-rank percentile helper.
scripts/benchmark/motion_generation/metrics/trajectory.py	Adds ordered waypoint matching, external motion-validity metrics, joint-limit checks, and path metrics.
scripts/benchmark/motion_generation/scenarios/init.py	Registers built-in scenarios for benchmark discovery.
scripts/benchmark/motion_generation/scenarios/base.py	Defines the ScenarioProvider contract for track case generation.
scripts/benchmark/motion_generation/scenarios/free_space.py	Adds deterministic free-space case generation for free-space-common (seed-stable manifests, bins, shapes).
scripts/benchmark/motion_generation/planners/init.py	Registers built-in planner adapters for benchmark discovery.
scripts/benchmark/motion_generation/planners/base.py	Defines PlannerAdapter lifecycle contract and metadata hashing/export.
scripts/benchmark/motion_generation/planners/curobo.py	Adds cuRobo adapter with explicit prepare phase and empty-world configuration.
scripts/benchmark/motion_generation/planners/ik_interpolate.py	Adds sequential IK + interpolation diagnostic adapter with per-env failure handling.
scripts/benchmark/motion_generation/planners/toppra.py	Adds IK+TOPPRA diagnostic adapter with optional dependency gating and explicit planner cleanup.
scripts/benchmark/motion_generation/planners/neural.py	Adds explicit unsupported NMG adapter stub with configurable internal eps values.
scripts/benchmark/motion_generation/suites/smoke.yaml	Adds smoke suite YAML for free-space-common with deterministic Franka manifests.
scripts/benchmark/motion_generation/suites/coverage.yaml	Adds coverage suite YAML expanding batch/path/bin/seed matrix.
scripts/benchmark/motion_generation/README.md	Adds usage and scope documentation for the new benchmark suite.
scripts/benchmark/motion_generation/BENCHMARK_DESIGN.md	Updates benchmark design doc references to the new module layout and CLI entry point.
scripts/benchmark/curobo_extraction/run_benchmark.py	Updates module run instruction to the new package path.
scripts/benchmark/curobo_extraction/init.py	Adds package marker for cuRobo extraction microbenchmark.
scripts/benchmark/main.py	Replaces the old planners-neural-planner subcommand with motion-generation and wires parser arguments.
pyproject.toml	Adds psutil>=5.9 dependency for process RSS measurement used in benchmark tooling.
embodichain/lab/sim/planners/toppra_planner.py	Adds explicit close() method and routes __del__ through it for reliable pool teardown.
embodichain/lab/sim/planners/curobo/curobo_planner.py	Adds prepare_backend() API to separate backend materialization/warmup from first plan timing.
docs/source/guides/cli.md	Updates CLI docs to reference the new motion-generation benchmark command.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[greptile-apps]

Greptile Summary

The PR introduces an extensible free-space motion-generation benchmark with configurable planner adapters, deterministic scenarios, external trajectory metrics, aggregation, and reproducible reports.

• Adds cuRobo, IK interpolation, TOPPRA, and unsupported NMG adapters behind a registry.
• Adds smoke and coverage suites plus lifecycle, latency, quality, memory, and leaderboard reporting.
• Exposes the benchmark through embodichain benchmark motion-generation.

Confidence Score: 4/5

The PR is not yet safe to merge because the documented motion-generation command cannot load its built-in suites from an installed distribution.

The suite loader requires package-relative smoke.yaml and coverage.yaml files, while the current package discovery and package-data configuration omit the benchmark package and its YAML resources.

Files Needing Attention: scripts/benchmark/motion_generation/config.py, setup.py, pyproject.toml

Important Files Changed

Filename	Overview
scripts/benchmark/motion_generation/config.py	Defines suite loading, validation, and hashing for the new benchmark.
scripts/benchmark/motion_generation/runner.py	Coordinates planner lifecycle phases, benchmark cases, trials, and external metric collection.
scripts/benchmark/motion_generation/aggregation.py	Produces case-weighted performance, success, quality, and leaderboard datasets.
scripts/benchmark/motion_generation/scenarios/free_space.py	Generates deterministic Franka free-space cases and waypoint manifests.
scripts/benchmark/motion_generation/reporting.py	Renders the aggregated benchmark results into the required three-table report.
scripts/benchmark/main.py	Replaces the neural-planner command with the extensible motion-generation CLI.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    CLI[Motion-generation CLI] --> CFG[Suite configuration]
    CFG --> SCN[Scenario and case manifest]
    CFG --> REG[Planner registry]
    SCN --> RUN[Benchmark runner]
    REG --> RUN
    RUN --> MET[External metrics]
    MET --> AGG[Aggregation]
    AGG --> ART[JSONL and JSON artifacts]
    AGG --> REP[Three-table Markdown report]

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into feat/motion-gen..." | Re-trigger Greptile}

[greptile-apps]

Built-in suites missing from packages

When the benchmark runs from an installed distribution, load_suite resolves smoke and coverage to package-relative YAML files that the packaging configuration does not include, causing the documented command to fail with FileNotFoundError.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/motion_generation/config.py
Line: 267-273

Comment:
**Built-in suites missing from packages**

When the benchmark runs from an installed distribution, `load_suite` resolves `smoke` and `coverage` to package-relative YAML files that the packaging configuration does not include, causing the documented command to fail with `FileNotFoundError`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[Copilot]

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/benchmark/main.py:123

• The legacy subcommand planners-neural-planner is no longer registered in the unified benchmark CLI. Issue #437 explicitly calls out keeping that entry point; removing it is a breaking change for existing scripts/docs. Consider keeping planners-neural-planner as a deprecated alias that forwards to the new motion-generation runner.

    motion_generation_parser = subparsers.add_parser(
        "motion-generation",
        help="Benchmark free-space motion generation with cuRobo as baseline.",
    )
    add_parser_arguments(motion_generation_parser)

scripts/benchmark/motion_generation/artifacts.py:69

• _git_commit() currently runs git rev-parse HEAD in the current working directory. When the benchmark is executed from another git repo, this can record an unrelated commit hash (or misleadingly succeed outside the EmbodiChain checkout). It’s safer to scope the command to the repository root (when available) and only return a hash on success.

def _git_commit() -> str | None:
    """Return the current repository commit without failing outside git."""
    result = subprocess.run(
        ["git", "rev-parse", "HEAD"],
        check=False,

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

Suppressed comments (1)

scripts/benchmark/motion_generation/metrics/trajectory.py:182

• _resample_joint_path() returns only a single sample when the input trajectory has length 1, even if validation_samples > 1. This can both (a) make FK validation under-sample the path and (b) raise torch.stack shape errors when other envs have longer trajectories (because some resampled paths will be (1, dof) while others are (validation_samples, dof)).

def _resample_joint_path(positions: torch.Tensor, sample_count: int) -> torch.Tensor:
    """Resample one joint path at uniform cumulative joint-arc length."""
    positions = torch.as_tensor(positions)
    if positions.shape[0] == 0:
        return positions
    if positions.shape[0] == 1 or sample_count <= 1:
        return positions[:1]