Robonix Package Catalog
Back to packages

robonix.skill.vertical_grasp_object

Detect objects on the table (YOLO cubes, or open-vocabulary via a VLM), then grasp and place them (with optional grasp angle and gripper aperture) — MCP tools for the D1 dexterous hand.

README

skill-vertical-grasp-object-rbnx

中文版

robonix.skill.vertical_grasp_object — detects objects on a table with a head RGB-D camera (YOLO-OBB for coloured cubes, or a VLM for open-vocabulary detection of anything else), then picks / places them vertically with a D1 6-DOF arm and five-finger dexterous hand, and can re-check the result.

Call semantics, parameters, constraints, errors and the recommended closed loop are covered in CAPABILITY.md (read at registration and handed to Atlas; pilot retrieves it through read_capability_doc). This file is about deployment and builds.

Capabilities (8 MCP tools + lifecycle)

Contract Transport What it does
robonix/skill/vertical_grasp_object/driver gRPC lifecycle
.../detect_objects MCP VLM open-vocabulary detection from a natural-language description; returns a base-frame position + grasp yaw. Preferred in general
.../detect_cubes MCP YOLO-OBB detection of coloured cubes, same schema. Preferred for cubes, and the fallback when no VLM is configured
.../pick_cube MCP move to a position, grasp and hold; optional grasp yaw and aperture
.../place_cube MCP move to a position and release; parks out of view on success
.../stack_cubes MCP one call for "stack colour A on colour B": detect → pick → place
.../put_cube_in_container MCP one call to drop a cube of a given colour into the fixed container spot
.../sort_cubes MCP one call to sort every cube on the table into its colour's fixed spot
.../verify_grasp MCP VLM re-check of whether a pick / place landed — the basis for closed-loop retries

position is base-frame metres as "x,y" / "x,y,z", or a named location. Both ASCII and full-width commas are accepted.

Runtime dependencies

The skill is a pure robonix consumer: on_activate discovers the primitives below through atlas and drives them over gRPC throughout. It never opens the serial link, the CAN bus, or the RealSense itself.

Primitive Contracts used
robonix.primitive.beingbeyond.d1.arm arm/get_state, arm/move_joint, arm/set_head
robonix.primitive.beingbeyond.d1.hand hand/move_joint, hand/get_state, hand/info
robonix.primitive.beingbeyond.d1.camera camera/snapshot

IK/FK, YOLO inference and the hand-eye projection are local pure compute (object_detect + block_grasp ship with the package; FK/IK come from the beingbeyond_d1_sdk wheel). detect_objects and verify_grasp additionally need an OpenAI-compatible VLM endpoint; without one those two tools are unavailable and the other six keep working.

Assets (required, not shipped)

Two robot-specific files are needed — see models/README.md:

File What it is Where it comes from
best.pt YOLO-OBB cube-detection weights (~113 MB) train it yourself with the cube-detection chain
handeye_calib.npz camera→base hand-eye homography + head pose + table Z run the hand-eye calibration on this robot

The calibration is tied to the camera mount and the table height; re-calibrate after remounting the head camera or changing the table height.

Because this package is normally fetched by a deployment manifest's url: into rbnx-boot/cache/, its models/ dir starts empty on every fresh fetch. Keep the two files in the deployment repo and point model_path / calib_path at them by absolute path.

Configuration

Fields, units, defaults and failure conditions are in config.spec. The commonly tuned ones:

  • pick_z / block_height — grasp height and one cube's height (what stack_cubes uses for its release point).
  • table_z_offset — a constant correction to the calibrated table Z (negative when picks land uniformly too high across the whole table).
  • model_path / calib_path — empty falls back to the package's own ./models/ defaults; an absolute path is used as-is.
  • vlm_base_url / vlm_api_key / vlm_model — empty falls back to the env vars VLM_BASE_URL / VLM_API_KEY / VLM_MODEL. The model must accept image input.
  • vlm_grasp_height — the VLM gives no depth, so this is the assumed grasp Z.
  • verify_match_radius — the radius within which verify_grasp counts an object as being "at" the queried position.
  • grasp_feedback — whether to judge grasp success from the finger angles. Set it false when that feedback misfires, otherwise sort_cubes opens the hand and drops a cube it was actually holding.

Build and run

bash scripts/build.sh   # rbnx codegen --mcp: gRPC stubs + typed MCP Request/Response classes
rbnx caps -v | grep vertical_grasp_object
rbnx chat               # e.g. "stack the red cube on the blue one"

Skill-kind packages stay INACTIVE after rbnx boot; the executor fires CMD_ACTIVATE on the first MCP call. That is the expected state, not a startup failure.

scripts/start.sh defaults to $HOME/miniconda3/envs/bb_d1_robonix/bin/python3; override with VERTICAL_GRASP_OBJECT_PYTHON. That env needs Python 3.10 (the SDK wheel is cp310 + manylinux_2_17_x86_64) with beingbeyond_d1_sdk (≥ 0.2.0, for FK/IK — the wheel ships in the robot-beingbeyond-d1 deployment repo under tools/func_verify/lib/), ultralytics, numpy/scipy, robonix_api and grpcio. There is no cleanup to do, so no scripts/stop.sh is provided.

Safety

  • On a new robot, run detection only at first (detect_objects / detect_cubes) and sanity-check the returned coordinates before commanding a grasp.
  • Before grasping, make sure there are no hands or fragile items in the workspace. A pick_z set too low presses the fingers into the table.
  • Detection first parks the arm at HOME to clear the camera's view — that is real motion.
  • The skill does no collision checking; the caller has to sequence around existing stacks on the table.

Layout

package_manifest.yaml   config.spec   CAPABILITY.md   README.md   README_CN.md
scripts/{build.sh, start.sh}
capabilities/*.v1.toml + capabilities/lib/vertical_grasp_object/srv/*.srv
vertical_grasp_object_skill/{main.py, controller.py, primitive_clients.py, detector.py, vlm_detector.py}
object_detect/    # YOLO-OBB detection + geometry
block_grasp/      # IK/FK, coordinate transforms, grasp motion
models/           # best.pt + handeye_calib.npz (robot-specific, not shipped)

License

MulanPSL-2.0