---
title: OmniR2V
canonical_url: "https://www.modelscope.cn/datasets/TencentOVBU/OmniR2V"
md_url: "https://www.modelscope.cn/datasets/TencentOVBU/OmniR2V.md"
repository: TencentOVBU/OmniR2V
last_updated: 2026-10-21
license: other
downloads: 4955
stars: 9
---

# OmniR2V

> OmniR2V - TencentOVBU 在 ModelScope 开源的数据集。Large-scale training data for omni reference-to-video generation

TencentOVBU/OmniR2V 是 ModelScope 魔搭社区上的数据集，采用 other 许可。

- **Repository**: TencentOVBU/OmniR2V
- **License**: other
- **Downloads**: 4955
- **Stars**: 9
- **Last updated**: 2026-10-21

Source: https://www.modelscope.cn/datasets/TencentOVBU/OmniR2V

---

<div align="center">

<h1 align="center">Omni-R2V Dataset</h1>

<p align="center"><strong>Large-scale training data for omni reference-to-video generation</strong></p>

[![Project Page](https://img.shields.io/badge/🌐_Project_Page-Visit-6366f1?style=for-the-badge)](https://wxliii.github.io/OmniVBench/)
[![Paper](https://img.shields.io/badge/arXiv-2609.22069-B31B1B?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2609.22069)
[![GitHub](https://img.shields.io/badge/GitHub-OmniVBench-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/wxliii/OmniVBench)
[![Benchmark](https://img.shields.io/badge/🤗_Benchmark-OmniVBench-ffb000?style=for-the-badge)](https://huggingface.co/datasets/wxli318/OmniVBench)

![Samples](https://img.shields.io/badge/Samples-339%2C570-2563eb?style=flat-square)
![Reference Families](https://img.shields.io/badge/Reference_Families-7-7c3aed?style=flat-square)
![Languages](https://img.shields.io/badge/Instructions-EN_%2F_ZH-059669?style=flat-square)

[Overview](#overview) · [Task coverage](#task-coverage) · [Quick start](#quick-start) · [Data format](#data-format) · [Citation](#citation)

</div>

<p align="center">
  <img src="figures/show_case.jpg" alt="Omni-R2V examples across seven reference families" width="100%">
</p>
<p align="center"><sub>From individual reference factors to multi-content and cross-aspect combinations.</sub></p>

## Overview

For evaluation, download the companion [OmniVBench benchmark](https://huggingface.co/datasets/wxli318/OmniVBench), which provides generation instructions and reference media for 813 evaluation cases.

**Omni-R2V** provides **339,570 processed training samples** across **seven reference families and 13 subtasks**. Given one or more image or video references, a model follows an instruction to generate a target video while preserving the specified reference factors.

| | What the dataset provides |
|:--|:--|
| 🧩 **Broad reference coverage** | Content, motion, style, structure, narrative, multi-content, and cross-aspect settings in one dataset. |
| 🖼️ **Reusable reference collections** | Multiple candidate references are retained where available, supporting single-reference selection, multi-reference conditioning, and alternative training pairs. |
| 💬 **Bilingual instructions** | English and Chinese prompts describe the requested video and identify the role of each reference. |
| 📦 **Self-contained samples** | References and the target video are stored together, with relative paths in the annotations. |

Pairs are built through task-specific cross-pair matching, inverse construction, and complementary composition. Instructions identify the supplied references and the factors they should constrain.

### Multiple references, flexible training pairs

**We retain multiple candidate references where available, preserving the reference collection associated with each target video.** Organizing these candidates alongside the target and reference-aware instructions makes the collected data reusable across different conditioning setups. The released sample count describes the stored annotation records; additional training pairs can be constructed from suitable reference candidates.

| Training setup | How to use the retained references |
|:--|:--|
| **Single-reference training** | Select one suitable candidate per sample, either once during preprocessing or dynamically during training. |
| **Multiple reference–target pairs** | For interchangeable candidates, pair each reference with the same target video to construct separate training examples. |
| **Multi-reference training** | Use the available references together, or sample compatible subsets to vary the number of conditioning inputs. |

When selecting or splitting references, update the instruction and its `<Figure N>` / `<Video N>` tokens to match the retained inputs. Candidates describing the same subject may be used as alternatives; references that supply complementary subjects or factors in multi-content and cross-aspect tasks should be retained together when required by the instruction. Derived pairs share a target video and should stay in the same data split.

## Task coverage

| Family | Sub-task | Samples | What the reference constrains |
|---|---|---:|---|
| Content | `content` | 18,615 | Subject identity (person) |
| Multi-content | `multi_content` | 19,497 | Several subjects (person / object / environment) |
| Motion | `action` | 29,929 | Subject motion |
| Motion | `camera_motion` | 27,471 | Camera movement |
| Style | `style` | 8,842 | Visual style |
| Structure | `lineart` | 19,243 | Single-shot geometry (line art / edges) |
| Structure | `greybox` | 9,880 | Single-shot geometry (greybox layout) |
| Structure | `rough_storyboard` | 9,714 | Single-shot geometry (rough storyboard) |
| Narrative | `preceding_shot` | 67,882 | Cross-shot continuity (continue a previous shot) |
| Narrative | `multi_panel_storyboard` | 99,999 | Cross-shot sequence (multi-panel storyboard) |
| Cross-aspect | `content_lineart` | 9,682 | Subject + line art |
| Cross-aspect | `content_storyboard` | 9,647 | Subject + rough storyboard |
| Cross-aspect | `content_style` | 9,169 | Subject + style |

<p align="center">
  <img src="figures/statistic.jpg" alt="Sample distribution across Omni-R2V reference families" width="100%">
</p>

## Quick start

Annotations are UTF-8 JSONL files: **one record per sample**. All media paths are relative to the dataset root. The loader below uses only the Python standard library.

```python
import json
from pathlib import Path

root = Path("./OmniR2V")


def iter_samples(annotation):
    with (root / "meta" / annotation).open(encoding="utf-8") as handle:
        for line in handle:
            if line.strip():
                yield json.loads(line)


sample = next(iter_samples("content/content.jsonl"))
references = [
    {"role": group["role"], "modality": asset["modality"],
     "path": root / asset["rel_path"]}
    for group in sample["inputs"]
    for asset in group["assets"]
]
target = root / sample["target"]

print(sample["prompt_en"])
print(references)
print(target)
```

## Data format

### Directory structure

```text
OmniR2V/
├── README.md
├── figures/
├── meta/                              # 13 annotation files
│   ├── content/content.jsonl
│   ├── motion/
│   │   ├── action.jsonl
│   │   └── camera_motion.jsonl
│   ├── style/style.jsonl
│   ├── structure/
│   │   ├── lineart.jsonl
│   │   ├── greybox.jsonl
│   │   └── rough_storyboard.jsonl
│   ├── narrative/
│   │   ├── preceding_shot.jsonl
│   │   └── multi_panel_storyboard.jsonl
│   ├── multi_content/multi_content.jsonl
│   └── cross_aspect/
│       ├── content_lineart.jsonl
│       ├── content_storyboard.jsonl
│       └── content_style.jsonl
└── assets_tar/                        # Media, packed as tar volumes
    ├── action_0.tar … action_3.tar
    ├── camera_motion_0.tar … camera_motion_11.tar
    └── …                              # 130 volumes in total
```

Each volume restores the original layout when extracted:

```text
assets/{subtask}/{shard}/{seq}/        # One directory per sample
├── subject_00.jpg                     # Example reference filenames
├── subject_00_1.jpg
└── target.mp4
```

### Media volumes

Media files are packed into **130 tar volumes** of roughly 18 GB each, one set per sub-task. Packing keeps the repository at a few hundred files instead of ~848K individual media files, which makes downloads and resuming far more reliable. The archives are uncompressed. Allow additional disk space for the extracted media.

Download from ModelScope:

```bash
pip install -U modelscope-hub

# Full dataset: annotations, figures, and all media volumes
ms-hub download wli318/OmniR2V --repo-type dataset --local-dir ./OmniR2V

# Annotations and media for a single subtask
ms-hub download wli318/OmniR2V --repo-type dataset --local-dir ./OmniR2V \
  --include "README.md" "meta/motion/action.jsonl" "assets_tar/action_*"
```

Extract from the dataset root so that the relative paths in `meta/` resolve:

```bash
cd ./OmniR2V
for t in assets_tar/*.tar; do tar -xf "$t"; done
```

Volumes can be extracted in any order and deleted afterwards to reclaim disk space.

### Annotation fields

| Field | Description |
|:--|:--|
| `sample_id` | Unique identifier in the form `r2v_{subtask}_{seq:06d}`. |
| `aspect` / `subtask` | Reference family and subtask. |
| `prompt_en` / `prompt_cn` | English and Chinese instructions. |
| `inputs` | Ordered reference groups, each with a `role` and an `assets` list. |
| `inputs[].assets[].modality` | Reference media type: `image` or `video`. |
| `inputs[].assets[].rel_path` | Reference path relative to the dataset root. |
| `target` | Target video path relative to the dataset root. |

<details>
<summary><strong>View an example annotation</strong></summary>

The prompt text below is abbreviated; the two reference images correspond to the first content sample.

```json
{
  "sample_id": "r2v_content_000000",
  "aspect": "content",
  "subtask": "content",
  "prompt_en": "<Subject 1> is the boy in <Figure 1> and <Figure 2> ...",
  "prompt_cn": "<Subject 1> 是 <Figure 1> 和 <Figure 2> 中的少年……",
  "inputs": [
    {
      "role": "subject",
      "assets": [
        {
          "modality": "image",
          "rel_path": "assets/content/00/000000/subject_00.jpg"
        },
        {
          "modality": "image",
          "rel_path": "assets/content/00/000000/subject_00_1.jpg"
        }
      ]
    }
  ],
  "target": "assets/content/00/000000/target.mp4"
}
```

</details>

**Reading reference tokens.** `<Figure N>` identifies the N-th image and `<Video N>` the N-th video, counted separately as assets are traversed in `inputs` order. `<Subject N>` is a semantic identifier in the prompt; it does not map one-to-one to an input group.

## Use & license

The dataset supports research on reference-conditioned video generation, disentangling factors across heterogeneous references, and multi-reference composition.

**Research use only.** Commercial use and redistribution of the media are not covered. Annotations and instructions are provided by the authors under the same research-only terms. The detailed `TERMS_OF_USE.md` document has not yet been included in this directory.

## Citation

```bibtex
@article{omnivbench2026,
  title   = {OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation},
  author  = {Li, Wenxue and Guan, Peiyan and Jiang, Haoyang and Cai, Junxian and
             Liu, Hualuo and Zhang, Chunjie and Guan, Chong and Huang, Kai and
             Li, Songlian and
             Wu, Taiyi and Yu, Yongjian and Zhao, Xiaotong and Zhao, Alan and
             Liu, Eric and Chen, Xi and Liu, Yu and Zhu, Lei},
  journal = {arXiv preprint arXiv:2609.22069},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.22069}
}
```