---
title: MVBench
canonical_url: "https://www.modelscope.cn/datasets/modelscope/MVBench"
md_url: "https://www.modelscope.cn/datasets/modelscope/MVBench.md"
repository: modelscope/MVBench
chinese_name: MVBench
last_updated: 2024-11-21
license: "Apache License 2.0"
storage_size: "16 GB"
domain:
  - modalities
  - task_categories
tasks:
  - Text
  - video-classification
downloads: 59780
stars: 4
---

# MVBench

> MVBench - modelscope 在 ModelScope 开源的数据集。Repository: MVBench Paper: 2311.17005 Point of Contact: mailto:kunchang li

modelscope/MVBench 是 ModelScope 魔搭社区上的Text、video-classification数据集，涉及 modalities、task_categories 领域，存储大小 16 GB，采用 Apache License 2.0 许可。

- **Repository**: modelscope/MVBench
- **License**: Apache License 2.0
- **Tasks**: Text, video-classification
- **Domain**: modalities, task_categories
- **Storage size**: 16 GB
- **Downloads**: 59780
- **Stars**: 4
- **Last updated**: 2024-11-21

Source: https://www.modelscope.cn/datasets/modelscope/MVBench

---

# MVBench

## Dataset Description

- **Repository:** [MVBench](https://github.com/OpenGVLab/Ask-Anything/blob/main/video_chat2/mvbench.ipynb)
- **Paper:** [2311.17005](https://arxiv.org/abs/2311.17005)
- **Point of Contact:** mailto:[kunchang li](likunchang@pjlab.org.cn)


## <span style="color: red;">Important Update</span>
[18/10/2024] Due to NTU RGB+D License, 320 videos from NTU RGB+D need to be downloaded manually. Please visit [ROSE Lab](https://rose1.ntu.edu.sg/dataset/actionRecognition/) to access the data. We also provide a [list of the 320 videos](https://huggingface.co/datasets/OpenGVLab/MVBench/blob/main/video/MVBench_videos_ntu.txt) used in MVBench for your reference.


![images](./assert/generation.png)

We introduce a novel static-to-dynamic method for defining temporal-related tasks. By converting static tasks into dynamic ones, we facilitate systematic generation of video tasks necessitating a wide range of temporal abilities, from perception to cognition. Guided by task definitions, we then **automatically transform public video annotations into multiple-choice QA** for task evaluation. This unique paradigm enables efficient creation of MVBench with minimal manual intervention while ensuring evaluation fairness through ground-truth video annotations and avoiding biased LLM scoring. The **20** temporal task examples are as follows.

![images](./assert/task_example.png)

## Evaluation

An evaluation example is provided in [mvbench.ipynb](https://github.com/OpenGVLab/Ask-Anything/blob/main/video_chat2/mvbench.ipynb). Please follow the pipeline to prepare the evaluation code for various MLLMs.

- **Preprocess**: We preserve the raw video (high resolution, long duration, etc.) along with corresponding annotations (start, end, subtitles, etc.) for future exploration; hence, the decoding of some raw videos like Perception Test may be slow.
- **Prompt**: We explore effective system prompts to encourage better temporal reasoning in MLLM, as well as efficient answer prompts for option extraction.

## Leadrboard

While an [Online leaderboard]() is under construction, the current standings are as follows:

![images](./assert/leaderboard.png)