---
title: OmniCompass
canonical_url: "https://www.modelscope.cn/datasets/liuxuyang16/OmniCompass"
md_url: "https://www.modelscope.cn/datasets/liuxuyang16/OmniCompass.md"
repository: liuxuyang16/OmniCompass
last_updated: 2026-10-14
license: cc-by-nc-sa-4.0
storage_size: "117 GB"
downloads: 435
stars: 0
---

# OmniCompass

> OmniCompass - liuxuyang16 在 ModelScope 开源的数据集。OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

liuxuyang16/OmniCompass 是 ModelScope 魔搭社区上的数据集，存储大小 117 GB，采用 cc-by-nc-sa-4.0 许可。

- **Repository**: liuxuyang16/OmniCompass
- **License**: cc-by-nc-sa-4.0
- **Storage size**: 117 GB
- **Downloads**: 435
- **Stars**: 0
- **Last updated**: 2026-10-14

Source: https://www.modelscope.cn/datasets/liuxuyang16/OmniCompass

---

<div align="center">
<h1>OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding</h1>

<p align="center">
  <strong>English</strong> | <a href="https://huggingface.co/datasets/VCLab-PolyU/OmniCompass/blob/main/README_zh-CN.md">中文</a>
</p>

<p align="center">
Xuyang Liu<sup>1,2</sup>,
Shihao Wang<sup>1</sup>,
Xinyu Wei<sup>1,2</sup>,
Minghan Li<sup>3</sup>,
Zeqing Wang<sup>1,2</sup>,
<br>
Zuhao Yang<sup>4</sup>,
Zichen Wen<sup>5</sup>,
Lei Zhang<sup>1,2</sup>
<br><br>
<sup>1</sup> The Hong Kong Polytechnic University,
<sup>2</sup> OPPO Research Institute
<sup>3</sup> Colorado School of Mines,
<br>
<sup>4</sup> Nanyang Technological University
<sup>5</sup> Shanghai Jiao Tong University
</p>

<p align="center">
  <a href="https://arxiv.org/pdf/2610.18829"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b.svg" alt="Paper"></a>
  <a href="https://github.com/PolyU-VCLab/OmniCompass"><img src="https://img.shields.io/badge/Code-GitHub-D3D3D3.svg" alt="GitHub"></a>
  <a href="https://huggingface.co/datasets/VCLab-PolyU/OmniCompass"><img src="https://img.shields.io/badge/Dataset-HuggingFace-FCC624.svg" alt="Hugging Face"></a>
  <a href="https://modelscope.cn/datasets/liuxuyang16/OmniCompass"><img src="https://img.shields.io/badge/Dataset-ModelScope-624AFF.svg" alt="ModelScope"></a>
  <a href="https://omnicompass.netlify.app/"><img src="https://img.shields.io/badge/Project-Homepage-8F1329.svg" alt="Homepage"></a>
</p>

</div>

<p align="center">
  <img src="assets/index.png" width="1000" alt="Overall model performance on OmniCompass-VL">
</p>

OmniCompass is a holistic benchmark for omni-modal video grounding and understanding, comprising **820 videos** and built with approximately **1,800 human annotation hours**. It covers **2 target specifications** (Text and Text+Image), **2 complementary evaluation tracks** (OmniCompass-VL and OmniCompass-AV), **3 core tasks** (temporal grounding, spatial grounding, and question answering), and **3 video duration categories** (short, medium, and long). We evaluate **50 MLLMs** on both task performance and cross-specification consistency.

## 📝 Content

- [News](#🔥-news)
- [Overview](#✨-overview)
- [Dataset Statistics](#📈-dataset-statistics)
- [Download and Use](#📥-download-and-use)
- [Experiments Results](#📊-experiments-results)
- [License](#license)
- [Citation](#📌-citation)
- [Contact](#📩-contact)

## 🔥 News

* **`2026.10.06`** 🤗🤗 We release OmniCompass, a holistic benchmark for omni-modal video grounding and understanding.

## ✨ Overview

<p align="center">
  <img src="assets/overview.png" width="1000" alt="Overview of OmniCompass">
</p>

> **TLDR:** OmniCompass spans four dimensions: target specification, video evidence,
> task coverage, and video duration. Paired Text/Text+Image specifications are
> evaluated on the same targets. OmniCompass-VL contains 820 videos and 2,460
> instances, while the 240-video OmniCompass-AV subset contains 720 instances
> with native audio. Both tracks cover temporal grounding, spatial grounding,
> and question answering.

OmniCompass-VL evaluates general VLMs, while OmniCompass-AV evaluates
OmniLLMs with native audio-video understanding capabilities.

## 📈 Dataset Statistics

<p align="center">
  <img src="assets/dataset_statistics.png" width="1000" alt="OmniCompass video domains, durations, QA types, and grounding granularity">
</p>

OmniCompass contains **820 videos across 7 domains and 52 subcategories**,
totaling **80.6 hours**. Video durations range from 30 seconds to 88.4 minutes.
The figure summarizes domain coverage, video duration, eight VL and five AV
QA types, and the temporal and spatial granularity of grounding annotations.

Each instance has paired Text and Text+Image specifications.
The 240 AV videos are a subset of VL and are stored once.

## 📥 Download and Use

### For Coding Agent Users

As for Coding Agent users, such as Codex and Claude Code, we suggest installing
the [omnicompass-dataset skill](skills/omnicompass-dataset/SKILL.md) to prepare
the data. Use the following prompt:

Without installation, ask your agent to read
`skills/omnicompass-dataset/SKILL.md` from this dataset repository and follow it.

> Please use the omnicompass-dataset skill to prepare OmniCompass-AV for TG, SG,
> and QA under both Text and Text+Image in `./data/OmniCompass`. Check data
> availability and existing local files, then list the download scope and
> estimated size. Wait for my confirmation before downloading media.
> Prepare the data only; do not start evaluation.

For inference and scoring, use the separate
[omnicompass-eval skill](https://github.com/xuyang-liu16/OmniCompass/tree/main/skills/omnicompass-eval)
in the code repository.

### Download with the CLI

Use the [evaluation code](https://github.com/xuyang-liu16/OmniCompass)
and download the dataset into its working directory:

Use Python 3.12+ for the evaluation dependencies.

```bash
git clone https://github.com/xuyang-liu16/OmniCompass.git
cd OmniCompass
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements-api.txt huggingface_hub
hf auth login
export REVISION=$(python -c "from huggingface_hub import HfApi; print(HfApi().dataset_info('VCLab-PolyU/OmniCompass').sha)")
hf download VCLab-PolyU/OmniCompass --repo-type dataset \
  --revision "$REVISION" --local-dir ./downloads/OmniCompass
```

Keep the same `REVISION` when resuming. The full download is approximately
126 GB; retaining both archives and extracted media requires approximately
252 GB, plus space for frame caches and any model weights.

The dataset download has the following structure:

```text
OmniCompass/
|-- README.md                     # Dataset introduction and usage
|-- README_zh-CN.md               # Chinese dataset introduction and usage
|-- assets/                       # Overview, statistics, and result figures
|-- skills/
|   `-- omnicompass-dataset/       # Dataset download and preparation skill
|       `-- SKILL.md
|-- videos.jsonl                  # Video IDs, paths, duration, categories, and sources
|-- questions.jsonl               # TG, SG, QA; paired Text/Text+Image and ground truth
|-- schema.json                   # Annotation field definitions
|-- taxonomy.json                 # Video categories and QA types
|-- release.json                  # Release version and metadata checksums
|-- archives.json                 # Media archive paths, sizes, and checksums
|-- media_manifest.jsonl          # Individual media paths, sizes, and checksums
`-- media/
    |-- videos-000.tar            # Original MP4 videos with existing audio tracks
    |-- ...
    |-- videos-030.tar            # 31 video shards covering all 820 videos
    `-- references-000.tar        # Task-specific reference images for Text+Image
```

### Prepare for Evaluation

Run these commands from the cloned code repository:

```bash
python code/release_tools.py validate --data ./downloads/OmniCompass
python code/release_tools.py extract   --data ./downloads/OmniCompass --out ./data/OmniCompass
python code/release_tools.py verify --data ./data/OmniCompass
export DATA_ROOT=./data/OmniCompass
```

Use a new output directory for extraction. Pass `--data "$DATA_ROOT"` to
`bash run.sh` using the model configuration in the
[Evaluation Pipeline](https://github.com/xuyang-liu16/OmniCompass#-evaluation-pipeline).
The runner reads video and reference-image paths from the annotations.
For a subset, use the same repeated `--video-id` flags during extraction,
verification, and evaluation; the extracted annotations still cover the full dataset.

Set `--nframe 32` for 32 uniformly sampled frames, or replace it with `--fps 1`
for one frame per second. The runner extracts and caches frames automatically.
Videos retain their original encoding; the dataset does not impose a frame budget.
By default, VL evaluation does not use audio, while AV evaluation uses audio.
Install FFmpeg for audio-enabled evaluation. The runner extracts and caches
the complete audio locally, without modifying the videos.
Use `--audio on` or `--audio off` for explicit modality experiments; selections with audio enabled
must contain audio in every video. Text+Image additionally supplies the
task-specific reference image.

## 📊 Experiments Results

**Evaluation results of different MLLMs on OmniCompass-VL.**

<p align="center">
  <img src="assets/results_vl.png" width="1000" alt="Evaluation results of different MLLMs on OmniCompass-VL">
</p>

**Evaluation results of different MLLMs on OmniCompass-AV.**

<p align="center">
  <img src="assets/results_av.png" width="1000" alt="Evaluation results of different MLLMs on OmniCompass-AV">
</p>

## License

OmniCompass's original annotations are licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).
Third-party videos, audio, and images retain their original copyrights and applicable licenses.

## 📌 Citation

Please consider citing our works in your paper, if our findings help your research.

```bibtex
@misc{liu2026omnicompass,
  title = {{OmniCompass}: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding},
  author = {Liu, Xuyang and Wang, Shihao and Wei, Xinyu and Li, Minghan and Wang, Zeqing and Yang, Zuhao and Wen, Zichen and Zhang, Lei},
  year = {2026},
  eprint = {2610.18829},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/pdf/2610.18829}
}
```

## 📩 Contact

For any question about our paper or code, please email `xu-yang.liu@connect.polyu.hk`.