---
title: RoboFAC-dataset
canonical_url: "https://www.modelscope.cn/datasets/MinghaoYE/RoboFAC-dataset"
md_url: "https://www.modelscope.cn/datasets/MinghaoYE/RoboFAC-dataset.md"
repository: MinghaoYE/RoboFAC-dataset
last_updated: 2025-08-21
license: "MIT License"
storage_size: "14 GB"
downloads: 52084
stars: 1
---

# RoboFAC-dataset

> RoboFAC-dataset - MinghaoYE 在 ModelScope 开源的数据集。This is the official repository of RoboFAC dataset from the paper: "RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction" (https://arxiv.org/abs/2505.12224)

MinghaoYE/RoboFAC-dataset 是 ModelScope 魔搭社区上的数据集，存储大小 14 GB，采用 MIT License 许可。

- **Repository**: MinghaoYE/RoboFAC-dataset
- **License**: MIT License
- **Storage size**: 14 GB
- **Downloads**: 52084
- **Stars**: 1
- **Last updated**: 2025-08-21

Source: https://www.modelscope.cn/datasets/MinghaoYE/RoboFAC-dataset

---

# Dataset Card for RoboFAC
[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://mint-sjtu.github.io/RoboFAC.io/)
[![Github](https://img.shields.io/badge/Project-Github-light)](https://github.com/MINT-SJTU/RoboFAC)
[![Paper](https://img.shields.io/badge/Paper-PDF-red)](https://arxiv.org/abs/2505.12224)
[![Dataset](https://img.shields.io/badge/Dataset-Huggingface-green)](https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset)
[![Model](https://img.shields.io/badge/Model-Huggingface-yellow)](https://huggingface.co/MINT-SJTU/RoboFAC-7B)

RoboFAC is a multimodal VQA dataset for robotic failure analysis and correction, comprising over **10,000** robot manipulation videos and **78,623** question-answer pairs. The dataset supports both **VQA** and **failure reasoning** tasks across simulated and real-world environments.

## Files
```
# Make sure git-lfs is installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset
```
## Dataset Details

### Dataset Description

RoboFAC is designed to benchmark multimodal reasoning and error correction in robot manipulation. It includes:

- **9,440** erroneous robot manipulation trajectories
- **78,623** QA pairs
- **16** diverse manipulation tasks
- **53** unique scenes
- **1282** successful robot trajectories (as reference)

The dataset contains **simulated** and **real-world** demonstrations and is organized to support both **training** and **evaluation** of vision-language and robotic reasoning models.

- **Curated by:** MINT Group @ SJTU

- **Language(s) (NLP):** English
- **License:** MIT License

### Tasks and Descriptions

| Task Name         | Description                                                                 |
|-------------------|-----------------------------------------------------------------------------|
| SpinStack         | Pick up the cube on the spinning disc and stack it on another cube on the disc. |
| SpinPullStack     | Pull out the cube on the spinning disc and stack it on another cube on the disc. |
| MicrowaveTask     | Put the spoon on the table into the cup. Open the door of microwave, put the cup into the microwave and close the door. |
| SafeTask          | Put the gold bar into the safe, close the door of the safe and rotate the cross knob on the door to lock it. |
| ToolsTask         | Choose the correct (L-shaped) tools, grasp it to pull the correct (2-pins) charger and plug it. |
| UprightStask      | Upright the peg and stack it on the cube.                                     |
| PegInsetionSide   | Insert the peg into the hole on the side of the block.                        |
| PullCubeTool      | Grasp the L-shaped tool and pull the cube by it.                              |
| PlugCharger       | Grasp the charger and plug it into the receptacle.                            |
| InsertCylinder    | Upright the cylinder and insert it into the middle hole on the shelf.         |
| PlaceCube         | Pick up the cube and place it into the box.                                   |
| LiftPegUpright    | Lift the peg and upright it.                                                  |
| PickCube          | Pick the cube to the target position.                                        |
| PullCube          | Pull the cube to the red and white target.                                   |
| PushCube          | Push the cube to the red and white target.                                   |
| StackCube         | Pick up the cube and stack it on another cube.                               |

### Dataset Sources

- **Repository:** [https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset](https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset)
- **Paper:** [https://arxiv.org/abs/2505.12224](https://arxiv.org/abs/2505.12224)


## Uses


### Direct Use

RoboFAC is intended for:

- Robotic failure detection and correction
- Video-based VQA for robot actions
- Evaluating robotic vision-language models

## Dataset Structure

Each item in `training_qa.json` contains:

```json
{
  "id": "uuid",
  "video": "task/scenario/video_id.mp4",
  "conversations": [
    {
      "from": "human",
      "value": "<video>\\n[Prompt Question]"
    },
    {
      "from": "assistant",
      "value": "[Answer from model or annotator]"
    }
  ]
}