Skip to content

Add RL-friendly velocity and effort action commands - #462

Draft
yuecideng wants to merge 1 commit into
mainfrom
fix/control-command-actions
Draft

Add RL-friendly velocity and effort action commands#462
yuecideng wants to merge 1 commit into
mainfrom
fix/control-command-actions

Conversation

@yuecideng

Copy link
Copy Markdown
Contributor

Description

This PR adds a stable flat Box policy contract that routes action-term slices to typed qpos, qvel, and qf robot commands.

Key changes:

  • Support single and mixed control terms on explicit, disjoint joint_ids or control_part selections.
  • Validate batched shapes and finite values, then clip physical commands to robot limits by default.
  • Hold effort commands across every physics substep during action decimation.
  • Preserve the exact raw flat policy action in RL, reward, rollout, trajectory, and dataset paths.
  • Add optional tanh-bounded Gaussian policies with the Jacobian-corrected log probability used by PPO and GRPO.
  • Keep bare env.step(tensor) as the legacy qpos path while accepting explicit direct command mappings.
  • Document action ranges, drive-mode caveats, mixed-control configuration, and bounded-policy behavior.

Compatibility notes:

  • A single qpos action term retains its historical tensor return for task reward code.
  • Built-in simulator RL enables action squashing automatically only for normalized [-1, 1] manager bounds.
  • Custom manager bounds leave automatic squashing disabled and emit guidance.

Dependencies: None.

Issue: None linked.

Type of change

  • Enhancement (non-breaking change which improves an existing functionality)

Validation

  • black --check --diff --color ./ — 849 files clean.
  • Focused Gym, manager, simulation, policy, evaluation, and rollout tests — 128 passed, 1 deselected.
  • Standalone shared-rollout regression — 1 passed.
  • RL routing, APG, differentiable collector, and trainer regressions — 23 passed.
  • Real CPU simulation qvel/qf rollout — 1 passed.
  • make -C docs html — succeeded. The build reports 668 existing repository warnings; warning-as-error remains blocked by that baseline, with no new warnings from the changed action pages.

Screenshots

Not applicable.

Checklist

  • I have run the black . formatting gate for the code base.
  • I have made corresponding changes to the documentation.
  • I have added tests that prove the enhancement works.
  • Dependencies have been updated, if applicable (none required).

Expose flat policy actions that route to qpos, qvel, and qf commands, including mixed joint groups and substep-held efforts. Keep RL rollouts in raw policy space and add bounded Gaussian support, documentation, and regression coverage.
@yuecideng yuecideng added enhancement New feature or request gym robot learning env and its related features rl Features related to reinforcement learning physics Things related to physics labels Aug 6, 2026
@yuecideng
yuecideng requested a review from acrlw August 6, 2026 08:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request gym robot learning env and its related features physics Things related to physics rl Features related to reinforcement learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant