---
title: TacMind-Bench
canonical_url: "https://www.modelscope.cn/datasets/shijiaxin/TacMind-Bench"
md_url: "https://www.modelscope.cn/datasets/shijiaxin/TacMind-Bench.md"
repository: shijiaxin/TacMind-Bench
last_updated: 2026-10-31
license: mit
storage_size: "359 GB"
downloads: 1036
stars: 1
---

# TacMind-Bench

> TacMind-Bench - shijiaxin 在 ModelScope 开源的数据集。Benchmarking and Modeling Tactile Interaction States for Robotic Manipulation

shijiaxin/TacMind-Bench 是 ModelScope 魔搭社区上的数据集，存储大小 359 GB，采用 mit 许可。

- **Repository**: shijiaxin/TacMind-Bench
- **License**: mit
- **Storage size**: 359 GB
- **Downloads**: 1036
- **Stars**: 1
- **Last updated**: 2026-10-31

Source: https://www.modelscope.cn/datasets/shijiaxin/TacMind-Bench

---

<h1 align="center"><img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/tacmind.svg" width="42" height="42" alt=""> TacMind</h1>

<p align="center"><b>Benchmarking and Modeling Tactile Interaction States<br>for Robotic Manipulation</b></p>

<p align="center">
  Jiaxin Shi<sup>1,4</sup> · Zibin Wu<sup>2,4</sup> · Yichi Zhang<sup>3,4</sup> · Jianghua Duan<sup>4</sup> · Siyu Zhu<sup>5</sup> · Weihao Yuan<sup>4,6</sup>
</p>

<p align="center">
  <sup>1</sup> Shanghai Jiaotong University · <sup>2</sup> The Hong Kong University of Science and Technology<br>
  <sup>3</sup> Harbin Institute of Technology · <sup>4</sup> Daimon Robotics<br>
  <sup>5</sup> Fudan University · <sup>6</sup> Nanjing University
</p>

<p align="center">
  <a style="display: inline-block; margin: 0 4px 8px; vertical-align: middle;" href="https://tacmind-bench.github.io/"><img style="display: inline-block; margin: 0; height: 40px; vertical-align: middle;" src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/link-website.svg" height="40" alt="Project website"></a>
  <a style="display: inline-block; margin: 0 4px 8px; vertical-align: middle;" href="https://tacmind-bench.github.io/#leaderboard"><img style="display: inline-block; margin: 0; height: 40px; vertical-align: middle;" src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/link-leaderboard.svg" height="40" alt="Leaderboard"></a>
  <a style="display: inline-block; margin: 0 4px 8px; vertical-align: middle;" href="https://huggingface.co/datasets/JiaxinShi/TacMind-Bench"><img style="display: inline-block; margin: 0; height: 40px; vertical-align: middle;" src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/link-huggingface.svg" height="40" alt="Hugging Face datasets"></a>
  <a style="display: inline-block; margin: 0 4px 8px; vertical-align: middle;" href="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench"><img style="display: inline-block; margin: 0; height: 40px; vertical-align: middle;" src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/link-modelscope.svg" height="40" alt="ModelScope datasets"></a>
</p>

<p align="center"><a href="https://github.com/Tacmind-Bench/TacMind">GitHub</a> · English · <a href="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/file/view/master/README.zh-CN.md">简体中文</a></p>

<p align="center">
  <a href="#benchmark">Benchmark</a> · <a href="#tacmind-0">TacMind-0</a> · <a href="#results">Results</a> · <a href="#usage-guide">Usage guide</a>
</p>

<video controls muted loop playsinline preload="metadata" width="100%" src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/whiteboard.mp4" aria-label="Whiteboard wiping: synchronized external and wrist cameras, left/right tactile RGB, marker motion and deformation."></video>

<p align="center"><b>Whiteboard wiping.</b> Two camera views and both fingertips, synchronized in one episode.<br>Touch is shown as RGB, marker motion and deformation. <a href="https://tacmind-bench.github.io/#demo">Watch the demonstration ↗</a></p>

<a id="benchmark"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-benchmark.svg" width="28" height="28" alt=""> TacMind-Bench

**TacMind-Bench studies tactile interaction across 128 robotic manipulation tasks.**
Built on Isaac Sim and Isaac Lab, the released simulation pipeline uses a Franka arm
with two GelSight Mini sensors. It records external and wrist RGB, tactile RGB,
marker motion, robot states and language instructions together.

The repository provides expert demonstration collection, conversion to LeRobot datasets,
training for seven policies, and shared task-success evaluation. **TacMind-0** is
our policy for modeling tactile interaction states and using them to guide manipulation.

### Task collection

| Suite | Tasks | Capabilities and examples |
| --- | ---: | --- |
| **Precision** | 6 | Fine contact control: insert tubes, keys, connectors, gears, plugs and bulbs |
| **Recog** | 11 | Resolve visual ambiguity through touch: distinguish size, shape and surface texture, including in covered scenes |
| **Dynamic** | 11 | Manipulate moving objects or targets, including insertion into fixtures on a conveyor |
| **Extra** | 100 | Broader contact behaviors: open drawers, turn valves, cut objects and wipe surfaces |

**Standard = Precision + Recog + Dynamic (28 tasks).** The 100 Extra tasks expand the
range of objects and interaction skills. The [interactive task gallery](https://tacmind-bench.github.io/#tasks)
includes synchronized sensor views and additional Daimon demonstrations for Standard tasks.

### Evaluation conditions

Standard tasks support nominal evaluation and **12 controlled perturbations** applied
at evaluation time. Extra tasks support nominal evaluation.

| Perturbation group | Conditions | What changes |
| --- | ---: | --- |
| **Lighting** | 4 | Warm, cool, fluorescent and night lighting |
| **Camera** | 6 | Left/right viewpoints, high/low elevation and two farther viewpoints |
| **Object scale** | 2 | Objects scaled to 0.85× and 0.90× |

Use the same held-out seeds across models and conditions. Results report success rates,
with suite-balanced **InD** (nominal) and **OOD** (perturbed) averages; the complete scoring
procedure and commands are in [Robustness evaluation](#robustness-evaluation).

<a id="tacmind-0"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-model.svg" width="28" height="28" alt=""> TacMind-0

TacMind-0 combines tactile interaction-state modeling with vision, language and robot
state to generate actions. Its tactile input is a history of **left/right Deformation images**
showing marker displacement as arrows.

- **Tac-LeWM** learns tactile interaction representations and action-conditioned predictions of future tactile features. World-model training also uses visual context.
- **VTLA policy** combines these tactile representations with external/wrist images, the language instruction and robot state to generate action sequences.
- **Tac-FRS** predicts future tactile features under candidate actions and uses those predictions to modulate subsequent action generation.

Training starts with Tac-LeWM pretraining on multitask demonstrations. The target-task
pipeline then adapts the tactile model, trains the VTLA policy and trains Tac-FRS.
The [training guide below](#train) includes environments, pretrained dependencies,
commands and checkpoint paths for this complete sequence.

<a id="results"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-results.svg" width="28" height="28" alt=""> Benchmark results

Success rate (%) on the **GelSight Mini Standard tasks**. Higher is better; bold values
mark the best score in each column. The three suite columns show nominal performance.

| Model | Precision InD | Recog InD | Dynamic InD | Avg InD | Avg OOD |
| --- | ---: | ---: | ---: | ---: | ---: |
| ACT+Tactile† | 30.0 | **69.1** | **15.9** | **39.8** | 16.4 |
| DP† | 23.3 | 27.7 | 13.2 | 21.1 | 8.1 |
| OpenPI0.5 | 5.0 | 30.0 | 0.5 | 13.0 | 5.5 |
| OpenDM0.5 | 20.0 | 38.6 | 8.2 | 22.7 | 11.4 |
| N0-VTLA | 25.8 | 45.9 | 5.9 | 25.9 | 7.8 |
| **TacMind-0** | **40.0** | 50.0 | 10.5 | 32.3 | **20.7** |

† **Single-task training:** ACT+Tactile and DP train a separate policy for each task.
The [full leaderboard](https://tacmind-bench.github.io/#leaderboard) also reports lighting,
camera and object-scale scores for each suite.

<a id="usage-guide"></a>

## Usage guide

[Installation](#installation) · [Data](#data) · [Train TacMind-0](#train) · [Evaluation](#policy-evaluation) · [Other policies and configs](#policies) · [Advanced usage](#advanced)

The example below uses Precision 0: install → prepare data → train → evaluate nominal and perturbed conditions. Run all commands from the repository root.

<a id="installation"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-install.svg" width="28" height="28" alt=""> Install the environment and assets

Linux x86_64, Conda and Git LFS are required; collection and evaluation also need
an Isaac-compatible GPU. Run from the repository root.
Environment versions: Ubuntu 22.04, CUDA 12.8, Isaac Lab 2.3.2 and Isaac Sim 5.1.0.

### Base environment

```bash
git clone https://github.com/Tacmind-Bench/TacMind.git
cd TacMind
conda create -n tacmind python=3.11.16 pip
conda activate tacmind
git lfs install
git lfs pull
python -m pip install -e .
```

<a id="training-and-evaluation"></a>
### Learning dependencies

For conversion, ACT/DP training and policy evaluation. Other models need
[their environments](#source-and-environment-setup).
Install system `ffmpeg`, then run in a fresh terminal:

```bash
conda activate tacmind
python -m pip install torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r benchmark/config/learning.txt
python -m pip install --no-deps lerobot==0.4.4
```

<a id="simulation-environment"></a>
### Simulation dependencies

Install the NVIDIA driver and CUDA 12.8 toolkit with `nvcc`.
System tools: `cmake`, `build-essential`, `ffmpeg`, `rsync`, `git-lfs`, `unzip`.
Download the **Isaac Sim 5.1.0 Linux x86_64 standalone ZIP** from the
[official download page](https://docs.isaacsim.omniverse.nvidia.com/5.1.0/installation/download.html)
and save it as `data/downloads/isaac-sim-standalone-5.1.0-linux-x86_64.zip`.
Extract it into a new directory and run its setup script:

```bash
export TACMIND_ISAAC_SIM="$HOME/isaacsim-5.1.0"
mkdir -p "$TACMIND_ISAAC_SIM"
unzip data/downloads/isaac-sim-standalone-5.1.0-linux-x86_64.zip -d "$TACMIND_ISAAC_SIM"
(cd "$TACMIND_ISAAC_SIM" && ./post_install.sh)
```

Keep `TACMIND_ISAAC_SIM` set to that directory when installing Isaac Lab:

```bash
conda activate tacmind
mkdir -p third_party
git clone https://github.com/isaac-sim/IsaacLab.git third_party/IsaacLab
git -C third_party/IsaacLab checkout b0542fe2d45bf91c4e1d9ef6952b9c709c80b4e8
ln -s "$TACMIND_ISAAC_SIM" third_party/IsaacLab/_isaac_sim
./third_party/IsaacLab/isaaclab.sh --install none
```

Restore Conda's Torch after the installer, then install shared dependencies:

```bash
python -m pip install torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r benchmark/config/simulation.txt
```

Source Isaac before building cuRobo to match its Torch/CUDA libraries:

```bash
git clone https://github.com/NVlabs/curobo.git third_party/curobo
git -C third_party/curobo checkout 78fd485fa82d9b9a063fb4985e371814587e666a
python -m pip install 'setuptools-scm>=6.2' wheel ninja
tacmind_root="$PWD"
source third_party/IsaacLab/_isaac_sim/setup_conda_env.sh
cd "$tacmind_root"
python -m pip install --no-build-isolation ./third_party/curobo
```

<a id="run-terminals"></a>
### Run terminals

Use separate terminals; open a fresh shell when switching from simulation to learning.
Isaac loads its bundled libraries; the PyTorch 2.10 installation above applies to the learning environment.

<a id="learning-terminal"></a>
#### Conversion and training

```bash
conda activate tacmind
```

<a id="simulation-terminal"></a>
#### Collection and evaluation

```bash
conda activate tacmind
tacmind_root="$PWD"
source third_party/IsaacLab/_isaac_sim/setup_conda_env.sh
cd "$tacmind_root"
export HEADLESS=1 LIVESTREAM=0
```

Set `CUDA_VISIBLE_DEVICES` to your allocated GPU; evaluation uses logical `cuda:0`.

<a id="assets"></a>
### Simulation assets

Download `tacmind-assets.tar.gz` from [Hugging Face](https://huggingface.co/datasets/JiaxinShi/TacMind-Bench)
or [ModelScope](https://modelscope.cn/datasets/shijiaxin/TacMind-Bench) at `assets/tacmind-assets.tar.gz`. Extract to
`data/downloads/tacmind-assets/`, with `assets/` directly inside. In the simulation terminal:

```bash
tacmind assets import --source data/downloads/tacmind-assets
```

Complete the asset import before starting scenes. Existing files are preserved;
move a file aside before importing its replacement.

<a id="data"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-data.svg" width="28" height="28" alt=""> Get and convert data

Available files in this dataset repository:

- Raw recordings: `releases/gsmini/hdf5/`
- LeRobot data: `releases/gsmini/lerobot/`
- Simulation assets: `assets/`

Some packages are still being uploaded. Third-party assets and components retain their respective licenses.

Run from the repository root. Use the [simulation terminal](#simulation-terminal)
for collection/evaluation and the [learning terminal](#learning-terminal)
for conversion/training. This example uses Precision 0.

<a id="collection"></a>
### 1. Get data

Download and extract a task package from [Hugging Face](https://huggingface.co/datasets/JiaxinShi/TacMind-Bench)
or [ModelScope](https://modelscope.cn/datasets/shijiaxin/TacMind-Bench), keeping `collection.json`
and recordings in their relative locations. To collect 100 successful demonstrations yourself:

```bash
CUDA_VISIBLE_DEVICES=0 python -m benchmark.expert.collection_queue \
  --task precision 0 --gpu 0 --target 100 \
  --output data/expert/gsmini/precision0_100
```

Outputs are HDF5 recordings, tactile previews and `summary.json`; convert when its
`status="complete"`. `--gpu` uses a physical index or UUID from `CUDA_VISIBLE_DEVICES`,
as shown by `nvidia-smi` in the current container. Use a new directory for each task
or changed collection setting. To resume, stop the collector and its workers, then
rerun the command. The default limit is 500 attempts; increase it with `--max-attempts`.
Read failure reasons in the summary and worker logs.

<a id="tasks"></a>

| Collection suite | Conversion suite | IDs |
| --- | --- | --- |
| `precision` | `tac_precision` | 0–5 |
| `recog` | `tac_recog` | 0–10 |
| `dynamic` | `tac_dynamic` | 0–10 |
| `extra` | `tac_extra` | 0–99 |

List task names and IDs with `tacmind catalog --suite precision`.

<a id="dataset-conversion"></a>
### 2. Convert data

```bash
mkdir -p tmp
cp examples/learning/convert_precision0.json tmp/convert_precision0.json
```

Set `collection_manifest` to the downloaded `collection.json` or your collection's `summary.json`:

```bash
tacmind learning convert --config tmp/convert_precision0.json
```

Output: `data/lerobot/gsmini_precision0`, with vision, tactile RGB, marker images and Deformation images.
The directory must be absent or empty. Config paths resolve from the repository root;
recording paths resolve from the manifest. Move the entire collection directory together.
For other data, edit these fields and match the training config's `dataset.root` / `dataset.repo_id`:

| Field | Value |
| --- | --- |
| `suite` / `task_id` | Conversion suite and task ID from the table above |
| `episodes` | Positive successful demonstration count, no greater than the available count; selected in manifest order |
| `dataset_root` / `repo_id` | New dataset directory and identifier |
| `camera_names` | The template keeps `agentview`, `wristview`, `gsmini_left`, `gsmini_right` |
| `include_markers` / `include_deformation` | The template enables both; TacMind-0 requires `include_deformation: true` |

<a id="train"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-train.svg" width="28" height="28" alt=""> Train TacMind-0

Inputs must include `agentview` and `wristview` RGB, joint states, language instructions and
`observation.images.deformation_left/right`, the lossless FOTS displacement-arrow images.
Keep the generated `tactile_input.json` with each training bundle.

<a id="tacmind0-setup"></a>
### 1. Prepare the environment and pretrained weights

Complete [TacMind installation](#installation). Run from the repository root.
In a fresh terminal without Isaac, prepare the Python 3.12 training environment:

```bash
conda create -n tacmind0-python python=3.12 pip
conda run -n tacmind0-python python -m venv policy/TacMind-0/.venv
policy/TacMind-0/.venv/bin/python -m pip install -r policy/TacMind-0/requirements.txt
policy/TacMind-0/.venv/bin/python -m pip install --no-deps -e policy/TacMind-0
```

Download these pretrained weights, retaining their configuration and processor/tokenizer files:

| Resource | Source and local location | Use |
| --- | --- | --- |
| OpenDM 0.5 | [Dexmal/DM05](https://huggingface.co/Dexmal/DM05) → `data/weights/DM05/` | `pretrained_path` |
| DINOv2 Small | [facebook/dinov2-with-registers-small](https://huggingface.co/facebook/dinov2-with-registers-small) → `data/weights/dinov2-with-registers-small/` | Tac-LeWM `backbone_dir` |
| DINOv3 ViT-B/16 | [facebook/dinov3-vitb16-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m) → `data/weights/dinov3-vitb16/` | Tac-LeWM `vision_backbone_dir` |

<a id="taclewm-pretraining"></a>
### 2. Train Tac-LeWM

Use a GSmini multitask LeRobot dataset with vision, FOTS Deformation images and absolute joint actions.

If you already have Tac-LeWM weights pretrained on Deformation images and matching `resources/`, continue with step 3.

Install the pretraining environment and copy its configuration:

```bash
conda run -n tacmind0-python python -m venv policy/TacMind-0/.venv-pretraining
policy/TacMind-0/.venv-pretraining/bin/python -m pip install -r policy/TacMind-0/requirements-pretraining.txt
mkdir -p tmp
cp policy/configs/taclewm_pretrain.json tmp/pretrain.json
```

Use the following local paths in `tmp/pretrain.json`, replacing `dataset_path` with your multitask dataset. Keep the remaining recipe parameters from the template:

```json
{
  "runtime_python": "policy/TacMind-0/.venv-pretraining/bin/python",
  "dataset_path": "/path/to/gsmini_multitask_lerobot",
  "output_dir": "data/runs/taclewm_pretrain",
  "backbone_dir": "data/weights/dinov2-with-registers-small",
  "vision_backbone_dir": "data/weights/dinov3-vitb16",
  "devices": 3
}
```

`devices` must match the GPU count selected by `CUDA_VISIBLE_DEVICES`; this example uses 3 GPUs. The default trains for 20 epochs and writes to
`data/runs/taclewm_pretrain`; use a new `output_dir` for each run. In the TacMind learning terminal:

```bash
CUDA_VISIBLE_DEVICES=0,1,2 python -m policy.taclewm --config tmp/pretrain.json
```

Keep the generated `checkpoints/weights_epoch_N.pt`, its adjacent `tactile_input.json`, and the complete `resources/` directory together for TacMind-0 training.
With the default config, use `checkpoints/weights_epoch_20.pt`. This entry does not support optimizer resume.

<a id="tacmind0-policy-training"></a>
### 3. Train the target-task policy and Tac-FRS

The example uses the Precision 0 dataset converted above. Start in the TacMind learning
terminal (`conda activate tacmind`); the launcher starts the model's Python 3.12 process.

```bash
mkdir -p tmp
cp policy/configs/tacmind0.json tmp/tacmind0_precision0.json
```

The template uses these paths. Update them if your data or weights are elsewhere:

| Field | Precision 0 example |
| --- | --- |
| `dataset.root` / `dataset.repo_id` | `data/lerobot/gsmini_precision0` / `tacmind/gsmini_precision0` |
| `runtime_python` | `policy/TacMind-0/.venv/bin/python` |
| `pretrained_path` | `data/weights/DM05` |
| `model.world_weights` | `data/runs/taclewm_pretrain/checkpoints/weights_epoch_20.pt` |
| `model.resources` | `data/runs/taclewm_pretrain/resources` |
| `output_dir` | `data/runs/precision0_tacmind0` (new directory) |

```bash
CUDA_VISIBLE_DEVICES=0 tacmind learning train --config tmp/tacmind0_precision0.json
```

The default runs all three stages in order. Outputs are relative to `output_dir`:

| Stage | Default steps | Output |
| --- | ---: | --- |
| Adapt the pretrained Tac-LeWM to the target task | 10,000 | `tactile/checkpoint-10000/` |
| Train the VTLA action policy | 10,000 | `policy/`, with intermediate `policy/checkpoint-<step>/` |
| Train Tac-FRS | 10,000 | `frs/checkpoint-10000/` |

Read `stages.json` for the completed checkpoint paths. Use the policy or Tac-FRS checkpoint
for evaluation; tactile world-model weights alone are not an action policy.
This unified training entry uses one process and does not support optimizer resume.
The evaluation commands below use the final Tac-FRS checkpoint.

<a id="policy-evaluation"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-evaluate.svg" width="28" height="28" alt=""> Evaluate policies

<a id="tacmind0-inference"></a>
### TacMind-0 inference environment

TensorRT + CUDA Graph inference uses a separate PyTorch 2.10 environment and requires
a CUDA 13.0-compatible driver. In a fresh terminal without Isaac:

```bash
conda run -n tacmind0-python python -m venv policy/TacMind-0/.venv-inference
policy/TacMind-0/.venv-inference/bin/python -m pip install -r policy/TacMind-0/requirements-tensorrt.txt
policy/TacMind-0/.venv-inference/bin/python -m pip install --no-deps -e policy/TacMind-0
```

### Nominal evaluation

Switch to the simulation terminal configured above; the evaluator starts the specified inference process.

```bash
CUDA_VISIBLE_DEVICES=0 tacmind learning evaluate \
  --checkpoint data/runs/precision0_tacmind0/frs/checkpoint-10000 \
  --policy-python "$PWD/policy/TacMind-0/.venv-inference/bin/python" \
  --episodes 20 --seed-start 10000 \
  --output-dir tmp/eval/precision0_tacmind0/nominal --headless --device cuda:0
```

For ACT/DP, omit `--policy-python`; for other separate model environments, set it to their Python executable. Change the checkpoint and output directory accordingly.

Checkpoint paths for other models are listed below.
Evaluation reads the checkpoint's task and settings; use `--policy-python` if its interpreter moved.
Choose seeds held out from training and normalization; compare models with the same data,
seeds and conditions. `--episodes` must be positive; all seeds must be in `[0, 2**32)`.
Task limits are fixed at the longest reference episode plus 100 steps, at 20 Hz.

Read `success_rate` in `results.json` when `score_complete=true`. Videos go to `videos/`
by default; disable with `--video-mode none`. The output directory must be absent or empty.
After interruption, rerun the same checkpoint and seeds in a new directory.

<a id="checkpoint-layouts"></a>
### Choose the checkpoint

| Policy | Checkpoint path |
| --- | --- |
| ACT / DP | `checkpoints/<step>/pretrained_model/` |
| OpenPI / N0-VTLA | `checkpoints/tacmind/run/<step>/` |
| OpenDM | `checkpoints/checkpoint-<step>/`, or final `checkpoints/` |
| ACT Tactile | `checkpoints/best/`, or `checkpoints/<six-digit-step>/` |
| TacMind-0 | Policy/FRS path in `stages.json`; the default final bundle is `frs/checkpoint-10000/` |

Keep weights, configuration and normalization files when copying checkpoints;
copy symlink targets. TacMind-0 uses the separate inference environment above.

<a id="robustness-evaluation"></a>
### Robustness evaluation

Run all 12 perturbations for the same TacMind-0 checkpoint in a separate output directory:

```bash
CUDA_VISIBLE_DEVICES=0 tacmind learning evaluate \
  --checkpoint data/runs/precision0_tacmind0/frs/checkpoint-10000 \
  --policy-python "$PWD/policy/TacMind-0/.venv-inference/bin/python" \
  --perturb --episodes 20 --seed-start 10000 \
  --output-dir tmp/eval/precision0_tacmind0/perturbations --headless --device cuda:0
```

`--perturb lighting`, `camera` or `scale` selects 4, 6 or 2 conditions.
Supported suites: Precision, Recog, Dynamic; Extra supports nominal evaluation only.
Each condition has a `results.json`; the root report is an index.

Aggregation requires all 13 complete conditions per task. **Avg InD** averages nominal
scores within Precision (6), Recog (11) and Dynamic (11), then averages the three categories.
**Avg OOD** averages conditions within each task's perturbation group, then tasks within
each category, then the nine category × group scores. Report percentages; missing conditions
leave the aggregate incomplete. Keep manifests, configs, source/environment versions,
complete checkpoints and evaluation commands with results.

<a id="policies"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-policies.svg" width="28" height="28" alt=""> Other policies and configurations

<a id="supported-policies"></a>
### Supported policies

| Policy | Configuration |
| --- | --- |
| ACT | [act/config.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/act/config.json) |
| Diffusion Policy | [diffusion/config.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/diffusion/config.json) |
| OpenPI (π0.5) | [configs/openpi.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/configs/openpi.json) |
| OpenDM (DM05) | [configs/opendm.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/configs/opendm.json) |
| N0-VTLA | [configs/n0-vtla.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/configs/n0-vtla.json) |
| ACT Tactile | [configs/act_tactile.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/configs/act_tactile.json) |
| TacMind-0 | [configs/tacmind0.json](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/configs/tacmind0.json) |

<a id="source-and-environment-setup"></a>
### Source and environment setup

ACT/DP use the [learning environment](#training-and-evaluation).
TacMind-0 uses the separate training and inference environments prepared above.
All policy sources are included under `policy/`. Use separate environments for
OpenPI, OpenDM and N0-VTLA. Follow the installation steps below in their existing
source directories; skip cloning the repositories again.

| Model environment | Installation reference |
| --- | --- |
| OpenPI | [README](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/openpi/README.md), including PyTorch support and Transformers replacement |
| OpenDM | [README](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/opendm/README.md) |
| N0-VTLA | [Installation guide](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/n0-vtla/docs/INSTALL.md), including model weights, Transformers replacement and DINOv2 cache |
| ACT Tactile | Training environment with Torch, Torchvision, NumPy, Pillow and safetensors; cache the visual ResNet18 ImageNet weights or allow the initial download |

ALOHA CAD and LIBERO scene assets are optional; see [asset setup](https://github.com/Tacmind-Bench/TacMind/blob/main/policy/optional-assets.md)
for upstream hardware and simulator examples.

<a id="training-configuration"></a>
### Training configuration

In a copy of the selected model configuration, set these fields before running the training command at the end of this section:

| Field | Value |
| --- | --- |
| `dataset.root` / `dataset.repo_id` | Dataset directory and identifier |
| `dataset.episodes` | Optional list of training episode indices; omit to use all episodes |
| `output_dir` | New training output directory |
| `runtime_python` | Python executable in the model environment; omit for ACT/DP |
| `pretrained_path` | OpenPI/N0-VTLA PyTorch directory containing `model.safetensors`, or DM05 directory for OpenDM/TacMind-0; omit for ACT/DP and ACT Tactile |

Replace placeholder paths; relative paths resolve from the repository root.
ACT/DP normalization uses the whole dataset even when `dataset.episodes` selects a subset.

#### Other model settings

The unified training entry for OpenPI, OpenDM, N0-VTLA, ACT Tactile and TacMind-0 supports one process.
ACT Tactile and N0-VTLA also require `gsmini_left` and `gsmini_right` tactile RGB.
ACT Tactile uses an 80/20 training/validation split. Set `model.tactile_ckpt` to initialize its
tactile encoder; otherwise it trains from scratch.

#### Reuse prepared data

These models prepare episodes under `dataset.root/policy_exports/<policy_type>` by default.
Set `prepared_root` to choose another directory, or run `tacmind learning prepare --config tmp/policy.json`
to prepare separately. Reuse requires the same policy type, dataset, episode selection and prompt.
When these change, choose a new `prepared_root` and prepare again.
To reuse an export on another host, copy the entire prepared directory and set `prepared_root`;
omit `dataset` and `prompt` to use the saved settings without the original dataset.

<a id="baseline-training"></a>
### Start training and resume

Copy a model configuration from the table above, using DP here:

```bash
mkdir -p tmp
cp policy/diffusion/config.json tmp/policy.json
```

Set the dataset and output paths using the fields above, then run:

```bash
tacmind learning train --config tmp/policy.json
```

The default DP configuration trains for 50,000 steps and saves to `data/runs/gsmini_precision0_dp`.

<a id="training-and-checkpoint-resume"></a>
To resume ACT/DP, keep the complete checkpoint, including `pretrained_model/` and `training_state/`:

```bash
tacmind learning train \
  --config data/runs/gsmini_precision0_dp/checkpoints/last/pretrained_model/train_config.json \
  --resume
```

<a id="advanced"></a>
## <img src="https://modelscope.cn/datasets/shijiaxin/TacMind-Bench/resolve/master/docs/media/icon-advanced.svg" width="28" height="28" alt=""> Advanced usage

Use the training or inference environment prepared above.
Run all commands from the TacMind repository root.

<a id="individual-stages"></a>
### Individual training stages

Activate the training environment, then use the stage entry below with `--help`:

```bash
source policy/TacMind-0/.venv/bin/activate
```

| Stage | Native entry |
| --- | --- |
| Tac-LeWM fine-tuning | `python -m tacmind0.tactile.finetune --help` |
| TacMind-0 policy training | `python -m tacmind0 --task train --help` |
| Tac-FRS | `python -m tacmind0.tac_frs.training --help` |

For fine-tuning, pass pretrained weights with `--weights` and matching configurations with `--resources`.
Policy training needs the base policy directory, fine-tuned `encoder.pt` and `backbone_config.json`,
episode JSONL files and images. Tac-FRS requires the complete fine-tuned world model, not just
its encoder.

### Policy service

Activate the inference environment and provide a trained policy or Tac-FRS bundle:

```bash
source policy/TacMind-0/.venv-inference/bin/activate
python -m tacmind0 --task inference \
  --model-config.model-name-or-path /path/to/policy_or_frs_bundle
```

The default port is `7897`; change it with `--inference-config.port`.

```python
from tacmind0.policy.client import TacMind0Client

client = TacMind0Client("http://127.0.0.1:7897")
client.reset(lane=0)
# Supply both Deformation images every simulation step, with consecutive frame indices starting at 0.
client.observe(left_deformation, right_deformation, frame_index=0, lane=0)
actions = client.infer(agentview, wrist, state, prompt, lane=0)
```

Images are RGB `uint8` arrays; `state` contains 8 joint/gripper values. Returned physical actions
have shape `(H, 8)`. Call `observe()` on every step, including while executing an existing action chunk.
Use separate lanes for parallel environments. For Tac-FRS, use `tacmind0.tac_frs.client.TacFRSClient`,
call `reset()` at each episode start and `close(lane=0)` when finished using the lane.

<a id="multitask"></a>
### Multitask policy training

Use a multitask dataset in the [pretraining format](#taclewm-pretraining), pretrained Tac-LeWM
and OpenDM 0.5 weights. Create a config with local paths:

```json
{
  "dataset": "/path/to/lerobot",
  "world_weights": "/path/to/taclewm/checkpoints/weights_epoch_20.pt",
  "world_resources": "/path/to/taclewm/resources",
  "base": "/path/to/dm05",
  "output": "/path/to/new-output"
}
```

```bash
source policy/TacMind-0/.venv/bin/activate
python -m policy.tacmind0.train_multitask_lerobot --config /path/to/direct.json prepare
torchrun --standalone --nproc_per_node=4 -m policy.tacmind0.train_multitask_lerobot \
  --config /path/to/direct.json train
```

The default uses global batch 32 and 100,000 steps, saving to `output/policy/checkpoint-N`
every 10,000 steps. Supported process counts are 1, 2, 4, 8, 16 and 32.
`process count × per_device_batch_size` must divide 32 exactly.
After interruption, rerun `train`
with the same config without repeating `prepare`. These checkpoints cannot be used
directly with unified evaluation; use [staged training](#tacmind0-policy-training) for single-task evaluation.
