---
title: MileBench
canonical_url: "https://www.modelscope.cn/datasets/FreedomIntelligence/MileBench"
md_url: "https://www.modelscope.cn/datasets/FreedomIntelligence/MileBench.md"
repository: FreedomIntelligence/MileBench
last_updated: 2025-02-06
license: "Apache License 2.0"
storage_size: "16 GB"
downloads: 4628
stars: 0
---

# MileBench

> MileBench - FreedomIntelligence 在 ModelScope 开源的数据集。We introduce MileBench, a pioneering benchmark designed to test the MultImodal Long-contExt capabilities of MLLMs. This benchmark comprises not only multimodal long contexts, but also multiple tasks…

FreedomIntelligence/MileBench 是 ModelScope 魔搭社区上的数据集，存储大小 16 GB，采用 Apache License 2.0 许可。

- **Repository**: FreedomIntelligence/MileBench
- **License**: Apache License 2.0
- **Storage size**: 16 GB
- **Downloads**: 4628
- **Stars**: 0
- **Last updated**: 2025-02-06

Source: https://www.modelscope.cn/datasets/FreedomIntelligence/MileBench

---

# MileBench

## Introduction

We introduce MileBench, a pioneering benchmark designed to test the **M**ult**I**modal **L**ong-cont**E**xt capabilities of MLLMs. 
This benchmark comprises not only multimodal long contexts, but also multiple tasks requiring both comprehension and generation. 
We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs’ long-context adaptation capacity and their ability to completetasks in long-context scenarios

 <img src="./images/MileBench.png" width="600" alt="MileBench" align="center" />

To construct our evaluation sets, we gather 6,440 multimodal long-context samples from 21 pre-existing or self-constructed datasets, 
with an average of 15.2 images and 422.3 words each, as depicted in the figure, and we categorize them into their respective subsets.
<center class="half">
  <img src="./images/stat2.png" width="300" alt="stat2"/><img src="./images/stat1.png" width="300" alt="stat1"/>
</center>

## How to use?

Please download MileBench_part*.tar.gz and unzip them using the following command. 

```bash
for file in MileBench_part*.tar.gz
do
  tar -xzvf "$file"
done
```

Then please refer to [Code for MileBench](https://github.com/MileBench/MileBench?tab=readme-ov-file#-dataset-preparation) to evaluate.

## Links

- **Homepage:** [MileBench Homepage](https://milebench.github.io/)
- **Repository:** [MileBench GitHub](https://github.com/MileBench/MileBench)
- **Paper:** [Arxiv](https://arxiv.org/abs/2404.18532)
- **Point of Contact:** [Dingjie Song](mailto:bbsngg@outlook.com)

## Citation

If you find this project useful in your research, please consider citing:
```BibTeX
@article{song2024milebench,
  title={MileBench: Benchmarking MLLMs in Long Context},
  author={Song, Dingjie and Chen, Shunian and Chen, Guiming Hardy and Yu, Fei and Wan, Xiang and Wang, Benyou},
  journal={arXiv preprint arXiv:2404.18532},
  year={2024}
}
```