---
title: t09_humanoid_pick_ball
canonical_url: "https://www.modelscope.cn/datasets/staritup/t09_humanoid_pick_ball"
md_url: "https://www.modelscope.cn/datasets/staritup/t09_humanoid_pick_ball.md"
repository: staritup/t09_humanoid_pick_ball
last_updated: 2026-10-23
license: apache-2.0
storage_size: "61 MB"
downloads: 96
stars: 0
---

# t09_humanoid_pick_ball

> t09_humanoid_pick_ball - staritup 在 ModelScope 开源的数据集。AI4Robot task T09, pick up a ball from the floor and place it on a table, Unitree G1 humanoid (Isaac Sim 5.1 / Isaac Lab 2.3.2): code, G1/Agile robot assets, textures and a sample video. v1.0.

staritup/t09_humanoid_pick_ball 是 ModelScope 魔搭社区上的数据集，存储大小 61 MB，采用 apache-2.0 许可。

- **Repository**: staritup/t09_humanoid_pick_ball
- **License**: apache-2.0
- **Storage size**: 61 MB
- **Downloads**: 96
- **Stars**: 0
- **Last updated**: 2026-10-23

Source: https://www.modelscope.cn/datasets/staritup/t09_humanoid_pick_ball

---

# T09 Pick Up a Ball from the Floor: Unitree G1 humanoid, v1.0

Simulation environment for AI4Robot task T09. A Unitree G1 humanoid (29 DoF, three-finger hands) starts 1.2-2.2 m from a coloured
ball lying on the floor of a closed room. It must walk to the ball, crouch, pick it up, stand up, carry it to a fixed table, place it
inside the coloured square goal area on the table top and end standing still. The agent (a large model) sees three RGB cameras
(head, left wrist, right wrist) with their intrinsics and world poses, plus the robot's own state (pelvis pose, palm poses, hand
closure, crouch level). It controls the robot with four command kinds: `base` (walk), `squat`, `eef` (move a palm) and `hand`.
Walking uses Isaac Lab's official Agile lower-body policy (frozen), crouching a scripted squat controller and the arms differential
IK; nothing is trained here.

Version v1.0, frozen 2026-10-23. Isaac Sim 5.1 + Isaac Lab 2.3.2.

## Files

| File | Content |
|---|---|
| `t09-humanoid-pick-ball-v1.0-code.zip` | `t09-humanoid-pick-ball-v1.0/`: `repo/` (the AI4Robot framework with T09: task, embodiment, tests, run scripts) and this `README.md`; `repo/` below means `t09-humanoid-pick-ball-v1.0/repo/` |
| `t09-humanoid-pick-ball-v1.0-robot.zip` | `assets/isaac/...`: the Unitree G1 robot and the Agile walking policy (NVIDIA Isaac assets, with the G1 licence file; see "Assets and licences"); the same files as T22 v1.2's robot zip |
| `t09-humanoid-pick-ball-v1.0-assets.zip` | `assets/t22_look/`: the two texture sets T09 uses for the floor and the table (6 PNG maps) |
| `episode=s0_l2-success=1-score=100.0.mp4` | sample video: the scripted solution on seed 0, layout 2 |
| `README.md` | this file (also inside the code zip) |
| `SHA256SUMS.txt` | checksums of the files above |

## Where T09 lives in `repo/`

- `ai4robot/tasks/t09_humanoid_pick_ball/`: scene build (room, table, pooled balls, goal areas), per-episode generation (`layout.py`,
  pure numpy), judge and progress score (`progress.py`), scripted solution (`solve.py`, `grasp.py`), `config.yaml` (all thresholds,
  the budget and the agent texts), `TASK.md` (task statement), `DESIGN.md` and `RUN.md` (design notes, results and run notes).
- `ai4robot/embodiments/unitree_g1_loco/`: the T09 humanoid. Agile policy for standing and walking, scripted squat controller
  (`squat.py`, `squat_table.json`), arm IK with a blocked-hand guard, three-finger hands, command checks. The "## Agent manual"
  section of `interface.md` is what the agent receives as `manual.txt`; `IMPLEMENTATION.md` holds the maintainer notes.
- `ai4robot/embodiments/unitree_g1_walk/`: the walking G1 embodiment it builds on (shared with T22's humanoid).
- `tests/test_t09_layout.py`, `tests/test_t09_progress.py`: offline tests (no Isaac needed); `test_core.py`, `test_g1_stuck.py`: framework tests.
- `t09_prep/`: `t09_run.py` (check / scripted solution / null solution), `t09_live.py` (live agent mode, records a video), `t09_video.py`
  (the recorder; also replays an episode from its command list), Windows helpers (`run_py.ps1`, `start_eval_t09.ps1`, `live_step_t09.ps1`,
  `batch.ps1`), `t09_astra_prompt_template.txt` (the agent prompt for one episode, same format as T22's `g1_astra_prompt_template.txt`),
  `isolation/` (agent account setup and check), `diag/` (replay, hostile-input and app-restart tests), `release/` (the build script of
  this package and its validation record), `SQUAT_NOTES.md` / `GRASP_NOTES.md` and the development scripts behind the numbers in those notes.

## Setup

1. A Python environment with Isaac Sim 5.1 and Isaac Lab 2.3.2 (see `repo/README.md` and `repo/docs/SETUP.md`), plus `scipy`,
   `imageio-ffmpeg` (every video, including live mode's default recording) and `pytest`, which `pyproject.toml` does not list.
2. Unzip the robot zip and the assets zip into the same folder and point `AI4R_ASSETS` at the resulting `assets/`. (The robot part can
   also be fetched with `python assets/download.py isaac/unitree_g1 isaac/g1_agile_policy`; the textures exist only in the assets zip.)
3. Set `PYTHONPATH` to `repo/` and run the scripts from `repo/`. On Windows also set `PYTHONUTF8=1`: the framework's `load_cfg` opens
   the UTF-8 task config without an encoding (shared-layer fix listed for the maintainers). Physics runs on the CPU (`--device cpu`, the
   scripts' default).

On Windows, `t09_prep/run_py.ps1 <script> [args]` does all of this: it takes the interpreter from `AI4R_PYTHON`, the assets from
`AI4R_ASSETS` and the Kit cache folder from `AI4R_SIMCACHE` (defaults are the build machine's folders), sets `PYTHONUTF8=1` and
`PYTHONPATH`, and accepts the NVIDIA Omniverse EULA (`ACCEPT_EULA=Y`, `OMNI_KIT_ACCEPT_EULA=YES`, `PRIVACY_CONSENT=N`) for you.
Windows blocks `.ps1` scripts by default (and scripts unpacked from a downloaded zip under RemoteSigned), so start the helpers with
`powershell -NoProfile -ExecutionPolicy Bypass -File ...`; the bypass applies to that one call only.

```
python t09_prep/t09_run.py check --seed 0                               # loads, builds the 3 episodes of a seed, start state is legal
python t09_prep/t09_run.py solve --seeds 0 --layouts 0,1,2 --tag dev    # scripted solution (reads ground truth), one process
python t09_prep/t09_run.py null  --seeds 0 --layouts 0,1,2              # do-nothing solution: 0 success, 0 score
python -m pytest -q tests/test_t09_layout.py tests/test_t09_progress.py
powershell -NoProfile -ExecutionPolicy Bypass -File t09_prep\run_py.ps1 t09_prep\t09_run.py check --seed 0  # the same, Windows wrapper
```

Reproducing the calibration needs one fresh process per episode (episodes run later in one process drift from a fresh run, and live
mode always starts a fresh process):

```
powershell -NoProfile -ExecutionPolicy Bypass -File t09_prep\batch.ps1 -Seeds 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19 -Layouts 0,1,2 -Tag <name> -PerEpisode -Parallel 6
```

Results go to `repo/runs/t09/solve_<name>_s<seed>_l<layout>/results.json`.

Replaying an episode as a video: `t09_video.py --seed S --layout L --cmds <file> --out <dir>`. `<file>` is a live episode's
`commands.jsonl` (each command's feedback is compared with the recording; `reproduced` must be true) or a JSON file holding a list of
command dicts (nothing to compare against). Add `--no-pre-step` when the commands come from `t09_run.py` (scripted solution,
calibration; its `results.json` keeps the last 60 commands as `cmd_log[i].cmd`), and not for live episodes.

Cameras use RTX. On Windows `t09_live.py`, `t09_video.py` and `t09_run.py --cams` start Kit with DirectX 12 (needed on the machine
this was built on); `AI4R_DX12=0` turns that off, and on Linux it is off.

## Agent interface (local live mode)

`t09_live.py --seed S --layout L --live <dir> --out <out> [--idle-timeout 7200] [--video 1]` builds the episode and exchanges files
with the agent:

- `<dir>/instruction.txt`: the task rules (robot-neutral) and, on its last line, `Instruction for this episode: ...` (one sentence
  naming the ball colour and the goal colour, as in RoboDojo);
- `<dir>/manual.txt`: the robot's manual (the "## Agent manual" section of the embodiment's `interface.md`);
- `<dir>/obs_0000/` (the start) and `<dir>/obs_0001/`, `obs_0002/`, ... after each command (k with 4 digits):
  `cam_head.png`, `cam_left_wrist.png`, `cam_right_wrist.png` (640x480 RGB with sensor noise; no depth),
  `obs.json` (RoboDojo layout: instruction, per camera `K`, `pos_w`, `quat_w_ros`; palm poses and hand closure) and `state.json`
  (pelvis pose, start pose, fallen, crouch level, pelvis height, palm poses, hand closure, budget used, feedback of the last command);
- the agent writes `<dir>/cmd_0001.json`, `cmd_0002.json`, ... (k with 4 digits, starting at 1, no gaps: the environment only waits
  for the next number; `obs_<k>/` answers `cmd_<k>.json`), one of `{"kind": "base", "vx", "vy", "wz", "duration"}`, `{"kind": "squat", "level",
  "duration"}`, `{"kind": "eef", "arm", "dpos", "drpy", "frame", "duration", "orient"}`, `{"kind": "hand", "side", "close"}`, or
  `{"done": true}` to stop; ranges and behaviour are in `manual.txt`. A command without `kind` is a `base` command.
- `<out>`: `_result.json` (RoboDojo: `success_rate`, `eval_time`, `score`, `details`), `result.json` (short),
  `result_s<seed>_l<layout>_live.json` (full judge detail, instruction, budget, end reason, any traceback), `commands.jsonl`,
  `judge_log.jsonl`, the video and a key frame after every command.

The prompt is `t09_prep/t09_astra_prompt_template.txt` with `{LIVE}` replaced by the live directory and `{STEP}` by the step script
`live_step_t09.ps1`, which sends one command and prints the new state. `start_eval_t09.ps1 -Runs "label:seed:layout,..." [-Idle 7200]
[-LiveRoot E:\agent_live] [-OutRoot E:\ai4r\agent_eval_out]` does the whole setup, as T22 v1.2 does:

- live directories go to `<LiveRoot>\<id>\live` (outside the repository and the output folder), outputs to `<OutRoot>\<id>`
  (plus `prompt.txt`, `live.out`, `live.err`, `launcher.json`), and a line per run to `<OutRoot>\mapping.csv`;
- the step script and the access check are copied to `<LiveRoot>\tools`;
- the live directories of finished T09 episodes (`live\result.json` written, or no process whose command line carries the run id;
  the launcher chain carries it from the start, so a run that is still starting is left alone) are first copied into their output
  folders (`live_archived`) and removed from the live root, so a later agent cannot read an earlier episode's observations and commands;
- the simulator is started detached through WMI, so it survives a restart of the app that launched it. Such a process does not inherit
  the caller's environment: `AI4R_PYTHON`, `AI4R_ASSETS`, `AI4R_SIMCACHE` and `AI4R_DX12` are passed on explicitly, nothing else is.
  It also outlives the session: it stops on `{"done": true}`, after `--idle-timeout` seconds without a new command, or when killed.

This stands in for the benchmark's own agent API. Unlike T22, there is no MCP tool-server route for T09 yet.

## Episodes, judging, budget

- Each seed gives 3 episodes: layout 0 (baseline: ball 0.40-0.55 m from the table, 20 cm goal), layout 1 (mirror: the ball on the
  other side, so the other hand) and layout 2 (hard: ball farther out sideways and from the table, robot 1.8-2.2 m away, 12 cm goal).
  The robot starts 1.2-2.2 m from the ball with its heading up to 30 deg off; ball and goal colours from red, blue, green, yellow,
  purple. Generation depends only on (seed, layout).
- Success: the ball centre is inside the goal square and the ball rests on the table top (centre 3.5-6 cm above it), the ball is
  still (< 0.02 m/s, < 0.2 rad/s) and the robot is standing, all held for 1 s at the end. A live episode ends on `{"done": true}` or
  after `--idle-timeout` seconds without a new command; the environment then runs 1 s more and judges. Using up the budget does not
  end it: further commands are rejected.
- A fall, the ball leaving the room or a physics anomaly freezes the progress score and makes success impossible; after a fall every
  command is rejected (and still counted).
- Progress score 0-100 from a milestone chain: walked to the ball 5, hand next to it 10, lifted 25, standing with it 15, over the
  table 10, in the goal area 15; success = 100.
- Budget per episode: 200 commands and 3000 action steps (120 s of simulated time at 25 Hz). The configured 3600 s wall clock is not
  enforced by the local harness.
- Scripted solution with 5 mm localisation noise, seeds 0-19 x layouts 0-2, a fresh process each: see "Validation".

## Video

2x2 frames at 25 fps, simulated real time: the head camera (as the agent sees it, but without the sensor noise) | a third-person view |
a close-up from the robot's right side | a panel with the episode instruction, the current command, the last feedback, the posture and
the judge, plus both wrist cameras. Files are named `episode=s<seed>_l<layout>-success=<0|1>-score=<score>.mp4`.

## Isolation

What the environment guarantees in live mode:

- The live directory only ever holds `instruction.txt`, `manual.txt`, `ready.json`, `obs_<k>/` (three images, `obs.json`,
  `state.json`), the agent's `cmd_<k>.json`, and at the end `result.json` = `{"ended": true, "reason": ...}`. No ground truth and
  no score are written there; judging output goes to the separate `--out` folder. (If the simulator crashes, no `result.json` is
  written; `<out>/live.err` has the traceback.)
- Every command is checked before it reaches a controller: unknown kinds or keys, non-numbers, NaN / infinity, negative durations,
  vectors that are not 3 numbers and unknown values of `arm`, `side`, `frame` or `orient` are rejected (0 action steps, no state
  change, but the command counts against the 200). Speeds, levels and closure are clipped to their ranges. A command's `duration` is
  cut to the remaining action-step budget, but the steps that do not come from the duration are not: braking (5 steps) and a pure
  turn's closed loop (up to 25 more), an eef command's interpolation (one step per 1 cm of `dpos` or 0.05 rad of `drpy`) plus 20
  settling steps, a squat's wait for stillness (up to 2.5 s), its rate limits (standing to level 1 takes about 4.8 s whatever the
  duration) and 10 settling steps, and a hand command's fixed 15 steps. So the command that uses up the budget can run past it (several
  seconds of simulated time for a squat, more for a large eef move). A file that is not a JSON object is rejected without being
  counted. Once the budget is used up every command is rejected.

What it cannot guarantee by itself: an agent with a shell on the same machine could read files outside the live directory (the
repository can regenerate any episode from its seed, and `--out` holds the judgement). For runs that count, run the agent under a
separate OS account that can reach the live root only. On Windows, `t09_prep/isolation/setup_agent_account.ps1` (T22's script with
the T09 check; run by an administrator) creates a standard local account that may modify `E:\agent_live` and is denied `E:\ai4r` and
any other folder you list with `-AlsoDeny` (list wherever you unpacked `repo/`, `assets/` and the outputs). `check_agent_access_t09.ps1`,
run as that account with `-Repo`, `-OutRoot`, `-Assets` and `-Profile` pointing at the real folders (and `-LiveRoot` / `-DenyRoot` if
yours are not `E:\agent_live` / `E:\ai4r`), verifies it; a path that does not exist is reported as MISSING, not as a pass. The account can read and write every live directory under the root, so do not run counted
episodes in parallel under one account. Give that agent no desktop-control or browser tools.

## Validation (this package)

- Clean-room run: both zips unpacked into an empty folder, `AI4R_ASSETS` pointing at the unpacked `assets/` and the package's own
  `t09_prep/run_py.ps1` (so `PYTHONPATH` = the unpacked `repo/`; imports were checked to resolve there):
  - `t09_run.py check --seed 0`: the 3 episodes build, start states are legal, `success()` false and score 0 at the start;
  - scripted solution on seed 0, layouts 0-2, one fresh process each: exactly the command logs and results of the official
    calibration runs from the source tree (score 5 / success / success);
  - the sample video's command list replayed with `t09_video.py --no-pre-step`: success, score 100;
  - live mode started with the package's `start_eval_t09.ps1` (detached through WMI, the `AI4R_*` overrides passed through), driven
    with the step script, judged and recorded into the output folder.
- Offline tests on this tree: 53 passed (`tests/test_t09_layout.py`, `test_t09_progress.py`, `test_core.py`, `test_g1_stuck.py`) with
  `PYTHONUTF8=1`, as `run_py.ps1` sets it. Without it, two `test_core.py` tests cannot read the Chinese config files on a Windows
  console with a GBK code page.
- Scripted solution, seeds 0-19 x layouts 0-2 (60 episodes, a fresh process each, 5 mm localisation noise), final code (command
  checks included): 25/60 success (layout 0 / 1 / 2: 11 / 10 / 4 of 20, mean score 79.5 / 63.5 / 40.2), 0 falls, 0 errors
  (the previous controller: 22/60 with 5 falls, all while moving an arm in the crouch).
- Hostile-input test in live mode (`t09_prep/diag/hostile_cmds.py`, run against the unpacked package): garbage text, a JSON array,
  NaN, infinity, a number as a string, a negative duration, an unknown kind, an extra key, a 2-number `dpos`, an unknown arm, an
  unknown frame, a speed of 1000, a 1e9 s duration and a command after the budget. Every case was rejected, clipped or cut as
  described above (this run used a reduced budget, `start_eval_t09.ps1 -Scale 0.1`, i.e. 20 commands / 300 action steps: the 1e9 s
  command was cut to the remaining 285 action steps and ran its 5 braking steps past them, 305 of 300 used at the end; with the
  default budget it is cut to whatever is left of the 3000); no crash; the live directory held only the listed files until
  `result.json` was written at the end.
- App-restart test (`t09_prep/diag/test_job_kill.py`): the starter runs inside a kill-on-close job object that is then killed, as
  when the app that launched it exits. The simulator survives with the WMI launch, and is killed with the previous `Start-Process`
  launch (control, `t09_prep/diag/start_eval_t09_startprocess.ps1`).
- Agent isolation on the build machine: the access check run as the agent account: 13 of 13 checks passed (live root readable and
  writable, `E:\ai4r` and the owner's profile denied, no write into `E:\ai4r`). That was the earlier version of the check; the
  shipped `check_agent_access_t09.ps1` (parameters, MISSING) has not been rerun as that account yet.
- English translation of this package (notes, comments, messages, upstream framework files): checked mechanically against the
  Chinese working copy (Python token streams outside comments and strings identical; YAML / JSON keys and non-text values identical;
  no Chinese characters left) and the offline tests rerun on the translated tree.
- The clean-room runs above were made before the last packaging fixes (command checks, archive step, separate robot zip, comment
  and README changes); the 60-episode calibration was rerun on the final code.
- Sample video rendered from this tree (scripted solution, seed 0 layout 2, success).

## Assets and licences

| Assets | Source | Licence |
|---|---|---|
| `assets/isaac/Isaac/Robots/Unitree/G1/` (robot zip) | Unitree G1, NVIDIA Isaac asset bucket `Isaac/Robots/Unitree/G1/`, as `assets/download.py isaac/unitree_g1` fetches it | BSD-3 (`unitree_ros`, see the [NVIDIA robot asset table](https://docs.isaacsim.omniverse.nvidia.com/latest/assets/usd_assets_robots_humanoid.html#unitree)) |
| `assets/isaac/Isaac/IsaacLab/Policies/Agile/` (robot zip) | Isaac Lab's Agile lower-body policy `agile_locomotion.pt`, NVIDIA Isaac asset bucket | NVIDIA ([upstream terms](https://www.nvidia.com/en-us/agreements/enterprise-software/isaac-sim-additional-software-and-materials-license/)) |
| `assets/t22_look/floor_tile_*.png` | procedural floor-tile texture made for T22 (fixed random seed) | self-made |
| `assets/t22_look/wood_*.png` | made for T22 from the Wood083A material of the SceneSmith example scene House/scene_208 (Hugging Face `nepfaff/scenesmith-example-scenes`), resampled and tiled | see that dataset's page |
| `repo/` | AI4Robot repository with T09 | the repository has no licence file; ask its maintainers |

The G1 BSD-3 copyright notice and licence are included in the robot zip at `assets/isaac/Isaac/Robots/Unitree/G1/LICENSE.unitree_ros.txt`
(the G1 folder is complete as NVIDIA ships it, including the inspire-hand variant T09 does not load). Third-party assets retain their
upstream licences.

## Known limitations

- Large-model agent results so far, seed 0 only: layouts 0 and 1 succeeded (score 100), layout 2 failed (score 5). In that episode the
  hand never reached the ball, and the robot fell while lifting the palm 12 cm with `orient: free` in the deepest crouch, a case the
  controller fixes do not cover yet. These runs used the controller fixes but predate the command checks, which only reject invalid
  commands. Two earlier episodes do not count: one fell before the controller fixes (analysis summarised in `DESIGN.md` 8.1), one was
  cut short when the app that had launched the simulator restarted (since then the detached launch).
- The scripted solution's success with 1-3 cm localisation error (33 / 22 / 14% for 1 / 2 / 3 cm) was measured before the controller
  fixes and has not been re-measured.
- Shared-layer changes against T22 v1.2's tree, for maintainers merging both: `ai4robot/embodiments/__init__.py` registers
  `unitree_g1_loco`; `unitree_g1_walk/robot.py` imports `FallWatch` from the new `unitree_g1_walk/fall.py` instead of `unitree_go2`;
  `unitree_g1_walk/interface.md` differs; the T22 task's default embodiment is `unitree_g1_walk`. The T22 code in this snapshot is not
  T22's release (its tests, `t22_go2_prep/` and its design document are left out); use T22's own package for T22.
- `load_cfg` without an encoding (see Setup). The development scripts in `t09_prep/` always start Kit with DirectX 12 and are
  Windows-only; the `.ps1` helpers default to the build machine's folders (overridable as described above).
- Everything in this package is in English. The task notes, code comments and messages were translated from the Chinese working copy
  for this release (behaviour unchanged: the Python token streams outside comments and strings are identical), and so were the
  upstream framework files (repo README, `docs/`, the other tasks' `TASK.md`, the task registry), which the upstream repository keeps
  in Chinese.