skill-vla-rbnx

Robonix package for the Vision-Language-Action (VLA) client skill — LLM-callable entry point for end-to-end neural-network manipulation on the Piper arm. Owns the skill/vla/* namespace.

Catalog name: robonix.skill.vla.openvla.

Overview

Closed-loop VLA control pipeline:

Camera topics ─► observe ─► VLA Server (inference) ─► safety filter ─► /arm/joint_command ─► piper_ctl ─► arm
                    ▲                                                                            │
                    └──────────────────── joint_states feedback ◄─────────────────────────────────┘

Key architecture decision: actions are sent directly to primitive-agilex-piper-arm-rbnx (/arm/joint_command), bypassing MoveIt entirely. VLA outputs are already planned trajectories — re-planning through MoveIt would destroy learned behavior and add 3–5 s latency per step.

Capability surface

Contract	Mode	Transport	Source / handler
`robonix/skill/vla/driver`	rpc	gRPC	`Driver(CMD_INIT, config_json)` — lifecycle gate
`robonix/skill/vla/execute`	rpc	MCP	`Execute(instruction) → success/failure` (user_invocable — LLM can call directly)

Unit convention

Must match the dataset builder exactly:

Joints 1..6: dataset SDK units = 0.001° (rad = SDK / 57324.840764).
Gripper (j7): dataset SDK units = 2 · µm (meters = SDK / 2_000_000; primitive-agilex-piper-arm-rbnx multiplies meters by 1e6 then by gripper_val_mutiple=2 before CAN).

Conflict with the vertical-grasp pipeline

skill-vla-rbnx monopolizes the joint command topic for closed-loop inference at action_hz, which collides with service-piper-moveit-rbnx's execute_grasp. In the same deploy manifest only one of the two skill stacks should be active at a time:

vertical-grasp: enable service-object-detect-rbnx / service-grasp-pose-rbnx / service-piper-moveit-rbnx / skill-pick-rbnx, disable this package;
VLA demo: enable this package, disable the four above.

Setup

# Clone
git clone https://github.com/syswonder/skill-vla-rbnx.git

# Build
bash scripts/build.sh

# Requires (on target robot):
# - ROS humble + rclpy
# - vla_server_rbnx running (provides inference)
# - primitive-orbbec-dabai_dcw-camera-rbnx running (provides camera images)
# - primitive-agilex-piper-arm-rbnx running (accepts joint commands)

Layout

skill-vla-rbnx/
├── package_manifest.yaml
├── capabilities/
│   └── skill/vla/{driver,execute}.v1.toml
├── vla_client/                 Python package: atlas bridge + action pipeline
└── scripts/
    ├── build.sh                rbnx codegen
    └── start.sh                source ROS, exec atlas_bridge

Driver-init lifecycle & activation

Init validates the VLA server URL and other config. Activation resolves camera topics via atlas (when use_atlas_discovery=true), then declares skill/vla/execute on atlas. The control loop only runs while Execute() is in progress.

Safety

Since MoveIt is bypassed, the internal safety filter provides:

Joint limits hard clip — prevents exceeding physical joint bounds.
Rate limiting — prevents single-step jumps that could damage hardware.
Gripper range clip — keeps gripper commands in valid range.

Emergency reset: if available, calls manipulation/reset (MoveIt joint-space) to park the arm safely.

Config (passed via `Driver(CMD_INIT, config_json)`)

Defaults are in package_manifest.yaml → config_defaults:

vla_server_url:       "http://localhost:8777"      # Direct URL fallback
use_atlas_discovery:  true                         # Discover vla_server via atlas

# Topics
full_image_topic:     /camera/color/image_raw
wrist_image_topic:    /wrist_camera/color/image_raw
joint_states_topic:   /arm/joint_states_single     # piper_ctl publishes proprio here
joint_cmd_topic:      /arm/joint_command            # piper_ctl subscribes here for joint ctl

image_resize:  [256, 256]                          # Resize before sending to VLA
action_hz:     10.0                                # Control frequency
timeout_s:     60.0                                # Default execution timeout
joint_state_dim: 7

# ── Dataset action range (must match piper_grasp_2cam_dataset_builder.ACTION_MIN / MAX) ──
# Joints 1..6 in dataset SDK units (0.001°); gripper in "2 µm" units.
action_min: [66563.0,  -2921.0, -151557.0,  -53279.0, -37605.0,  -94743.0, -3000.0]
action_max: [132905.0, 194064.0,     712.0,  103398.0,  71141.0,  119408.0, 88500.0]

# ── Hardware joint limits (safety clip — applied AFTER de-normalization) ──
# Conservative superset of dataset bounds so dataset values pass through unclipped.
# Gripper hard limits: [0, 80 mm] = [0, 80000] in 2 µm-style ⇒ [0, 160000].
hw_action_min: [-160000.0, -160000.0, -160000.0, -160000.0, -160000.0, -160000.0,      0.0]
hw_action_max: [ 160000.0,  160000.0,  160000.0,  160000.0,  160000.0,  160000.0, 160000.0]

# ── Per-step rate limit (in dataset SDK units) ──
# 5000 SDK units = 5° per step  → at 10 Hz = 50 °/s.
# Gripper: 20000 = 10 mm per step.
max_delta_per_step_sdk: [5000.0, 5000.0, 5000.0, 5000.0, 5000.0, 5000.0, 20000.0]

enable_safety_filter: true

# ── JointState publish parameters ──
# piper_ctl reads velocity[6] as global velocity % (1-100); 0 → default 30.
joint_velocity_pct: 30.0
# piper_ctl reads effort[6] as gripper effort; clipped to [0.5, 3].
gripper_effort: 1.0

use_atlas_discovery=true overrides the four topic strings via atlas resolution.
enable_safety_filter=false bypasses hardware-limit clipping — dev only.

Build / run standalone

bash scripts/build.sh
ROBONIX_ATLAS=127.0.0.1:50051 \
    bash scripts/start.sh

Verification

rbnx caps | grep skill/vla
# expect: vla_client  robonix.skill.vla.openvla  INITIALIZED
#   robonix/skill/vla/driver   (rpc/grpc)
#   robonix/skill/vla/execute  (rpc/mcp, user_invocable)

# End-to-end via Pilot LLM:
rbnx ask "put the block on the shelf"

vs `skill-pick-rbnx`

	`skill-pick-rbnx`	`skill-vla-rbnx`
Method	YOLO/VLM + geometry + MoveIt planning	End-to-end neural VLA
Control	MoveIt (3–5 s / step)	Direct joint command (10 Hz)
Safety	MoveIt collision detection	Internal filter
Use case	Simple pick-and-place	Complex language-guided manipulation

Coupling with neighbours

Upstream primitive-orbbec-dabai_dcw-camera-rbnx — RGB stream(s).
Upstream primitive-agilex-piper-arm-rbnx — joint proprio + joint command topic.
No hard dep on other services.

License

This package: MulanPSL-2.0.