Add Newton physics backend support - #276
Draft
yuecideng wants to merge 124 commits into
Draft
Conversation
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Jietao Chen <chenjietao@dexforce.com> Co-authored-by: Yueci Deng <dengyueci@qq.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: yuanhaonan <yuanhaonan@dexforce.top>
…entation (#247) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: chenjian <chenjian@dexforce.com> Co-authored-by: yuecideng <dengyueci@qq.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ration (#239) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Chen Jian <mtfl1996@outlook.com> Co-authored-by: chenjian <chenjian@dexforce.com> Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Yueci Deng <dengyueci@qq.com>
Co-authored-by: WaferLi <63717327+WaferLi@users.noreply.github.com> Co-authored-by: liwenfeng <liwenfeng@dexforce.top> Co-authored-by: chenjian <chenjian@dexforce.com> Co-authored-by: daojun <lookangela@qq.com> Co-authored-by: Chen Jian <mtfl1996@outlook.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Pin the physics-backend capability contract in a single source of truth (BACKEND_CAPABILITIES table) and assert, headlessly: - each backend's supports_* / can_disable_manual_update flags match the table; - every add_robot/add_soft_object/add_cloth_object/add_rigid_object_group capability guard raises NotImplementedError iff its flag is False; - the matrix covers every capability flag and every concrete backend. So flipping a flag or adding a backend fails loudly instead of silently changing which add_* methods are gated. Mirrors the lifecycle-test pattern (bare SimulationManager via object.__new__ with a fake physics back-ref). Co-Authored-By: Claude <noreply@anthropic.com>
…ical_attr Articulation.set_link_physical_attr called dexsim's set_physical_attr per link unconditionally. On Newton, set_physical_attr only mirrors onto link metadata (consumed at the next scene rebuild) — it does not push to the live model. So runtime per-link mass overrides applied via set_link_physical_attr (and via link_attrs overrides) did not take effect on Newton until a rebuild, unlike the dedicated set_mass which already branches on is_newton_backend -> set_link_mass. Fix: on the Newton backend, additionally call set_link_mass(local_name, mass) after the metadata mirror, mirroring the dedicated set_mass. set_link_mass pushes to model.body_mass/body_inv_mass + notify_model_changed(BODY_PROPERTIES) when READY (and to the builder when BUILDER). Friction/restitution/contact_offset remain rebuild-time-only for articulation links (no live per-link API on Newton, consistent with the Phase 3 deferred per-link contact params). Test: TestArticulationNewton.test_set_link_physical_attr_mass_live_on_newton verifies a runtime per-link mass override round-trips through get_mass. Co-Authored-By: Claude <noreply@anthropic.com>
set_mass / set_friction / set_inertia had an `is_ready` batch fast-path (which routes through the view and works on Newton) but their not-ready `else` paths unconditionally called the PhysX-bound MeshObject getters/setters (get_physical_body().set_mass / set_dynamic_friction / set_mass_space_inertia_tensor). Those methods are NOT Newton-patched, so on a Newton entity they hit the wrong backend (broken if ever reached before finalization). Harden the not-ready paths: on Newton, mirror the single field onto the link meta PhysicalAttr (consumed at the next finalize), tolerating desc-native- spawned objects (attrs.newton set) that carry no meta attr via a new _get_newton_attr_or_none helper. The default-backend path is unchanged. This is the safe, verifiable subset of the is_newton_scene sweep: the genuinely removable "reaches around the view" cases were already fixed in Phases 2-3 (set_attrs -> _set_newton_attrs, desc-native spawn). The remaining is_newton_scene branches are legitimate backend-specific lifecycle fallbacks (BUILDER-state entity dynamics, not-ready meta reads, static-object paths where self._data is None) that don't map to the batch-oriented RigidBodyViewBase ABC without extending its semantics. Verified: Newton rigid physical_attributes + desc-native spawn + headless physics_attrs/parity (38 passed); default CPU+CUDA rigid (44 passed, no regression). Co-Authored-By: Claude <noreply@anthropic.com>
Captures the implementation plan for the two outstanding Newton PR targets: arena cloning at finalize for multi-env parallel simulation, and DifferentiableEmbodiedEnv via a Warp-tape -> torch.autograd bridge for APG. Targets 1-3 are recorded as already-done with a pointer to design/newton-backend-design.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
10-task TDD plan covering: arena-clone-at-finalize (Targets 4), DifferentiableEmbodiedEnv + Franka APG example (Target 5), and branch cleanup. Each task has exact file paths, code, and test commands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prep for clone-at-finalize multi-env. Tracks whether source arena has been replicated into peer arenas for the current Newton finalize cycle; cleared by invalidate() so topology mutations trigger re-clone. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This reverts commit 86ead01.
Code inspection during execution revealed that EmbodiChain's spawn path (spawn_rigid_object_entities / spawn_articulation_entities) already does prototype-then-clone across all arenas at spawn time via dexsim's clone_actor_to (Newton-patched), Newton views already accept multi-entity lists, and existing Newton tests (TestRigidObjectNewton with NUM_ARENAS=2, test_spawn_clones_distinct_entities, test_newton_native_attrs_desc_native_spawn) already pass. Target 4 needs no work. Drops old Tasks 1-4 (_arenas_cloned flag, arena_index>0 guard, clone-at-finalize, multi-env body-id resolution). Renumbers remaining tasks: old 5→1, 6→2, 7→3, 8→4, 9→5, 10→6. Rewrites self-review to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
create_differentiable_stepper and create_gradient_rollout are thin passthroughs to NewtonManager. Both raise on the default backend. Backs the new embodichain.lab.sim.diff package (next commit). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The without-grad test was failing before reaching the delegator because
the default mujoco_warp solver rejects empty Newton scenes. Adding
solver_cfg={"solver_type":"semi_implicit"} isolates the test to the
delegator's grad guard — the behavior we actually want to verify.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New embodichain.lab.sim.diff package: NewtonStepFunc (autograd.Function) wraps DifferentiableStepper inside a wp.Tape, tape_context is the low-level context manager for advanced kernels, differentiable_step is the convenience wrapper. Foundation for DifferentiableEmbodiedEnv.
Newton-only EmbodiedEnv subclass that wraps step() in a Warp tape via NewtonStepFunc. Subclasses implement _apply_action_kernel and _read_outputs; the base class handles validation, auto-reset on done, and the autograd bridge.
End-to-end APG smoke task built on DifferentiableEmbodiedEnv with a Warp action-to-control kernel and a Warp reward kernel computed inside the tape. Fixes NewtonStepFunc.forward to call obs_reward_fn inside the open tape so reward carries gradient back to action. Adds a _make_step_fn hook on DifferentiableEmbodiedEnv so subclasses can swap in an FK-only grad path (the semi_implicit solver does not propagate grad through joint_target_pos to body_q; the FK bypass matches the reference APG env's workaround). Verifies the autograd bridge with one-iteration loss reduction. URDF resolved from newton.utils.download_asset with explicit override. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New agent_context topic 'differentiable-env' covers the APG contract (DifferentiableEmbodiedEnv + NewtonStepFunc bridge), the required NewtonPhysicsCfg, the subclass _apply_action_kernel / _read_outputs contract, the "reward must be inside the tape" rule, and the FK-bypass workaround used by the Franka example. Registered in MAP.yaml. design/newton-backend-design.md marks Targets 4 (multi-env) and 5 (differentiable env) Done with pointers to the implementation plan and the new topic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
9 tasks
…s-backend # Conflicts: # docs/source/api_reference/embodichain/embodichain.lab.sim.cfg.rst # docs/source/features/interaction/preview_asset.md # docs/source/guides/configuration.md # docs/source/overview/sim/sim_manager.md # embodichain/lab/gym/envs/base_env.py # embodichain/lab/gym/utils/gym_utils.py # embodichain/lab/scripts/preview_asset.py # embodichain/lab/sim/cfg.py # embodichain/lab/sim/objects/articulation.py # embodichain/lab/sim/objects/cloth_object.py # embodichain/lab/sim/objects/rigid_object.py # embodichain/lab/sim/sim_manager.py # embodichain/learning/rl/train.py # embodichain_tasks/embodichain_tasks/special/franka_reach_apg.py # examples/sim/demo/pick_up_cloth.py # examples/sim/gizmo/gizmo_camera.py # examples/sim/gizmo/gizmo_object.py # examples/sim/gizmo/gizmo_robot.py # examples/sim/gizmo/gizmo_scene.py # examples/sim/gizmo/gizmo_w1.py # examples/sim/planners/neural_planner.py # examples/sim/sensors/batch_camera.py # examples/sim/sensors/create_contact_sensor.py # examples/sim/solvers/differential_solver.py # examples/sim/solvers/opw_solver.py # examples/sim/solvers/pink_solver.py # examples/sim/solvers/pinocchio_solver.py # examples/sim/solvers/pytorch_solver.py # examples/sim/solvers/srs_solver.py # examples/sim/utility/workspace_analyzer/analyze_cartesian_workspace.py # examples/sim/utility/workspace_analyzer/analyze_joint_workspace.py # examples/sim/utility/workspace_analyzer/analyze_plane_workspace.py # scripts/tutorials/gym/modular_env.py # scripts/tutorials/gym/random_reach.py # scripts/tutorials/sim/create_cloth.py # scripts/tutorials/sim/create_rigid_object_group.py # scripts/tutorials/sim/create_scene.py # scripts/tutorials/sim/create_softbody.py # scripts/tutorials/sim/gizmo_robot.py # scripts/tutorials/sim/motion_generator.py # scripts/tutorials/sim/srs_solver.py # tests/gym/utils/test_gym_utils.py # tests/sim/objects/test_rigid_object.py # tests/sim/workspace/test_sim_utils.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds Newton backend support to the simulation stack.