---
title: VisitBench
canonical_url: "https://www.modelscope.cn/datasets/lmms-lab/VisitBench"
md_url: "https://www.modelscope.cn/datasets/lmms-lab/VisitBench.md"
repository: lmms-lab/VisitBench
last_updated: 2024-11-21
license: "Apache License 2.0"
storage_size: "776 MB"
downloads: 2591
stars: 0
---

# VisitBench

> VisitBench - lmms-lab 在 ModelScope 开源的数据集。Dataset Card for "VisitBench"

lmms-lab/VisitBench 是 ModelScope 魔搭社区上的数据集，存储大小 776 MB，采用 Apache License 2.0 许可。

- **Repository**: lmms-lab/VisitBench
- **License**: Apache License 2.0
- **Storage size**: 776 MB
- **Downloads**: 2591
- **Stars**: 0
- **Last updated**: 2024-11-21

Source: https://www.modelscope.cn/datasets/lmms-lab/VisitBench

---

# Dataset Card for "VisitBench"

<p align="center" width="100%">
<img src="https://i.postimg.cc/g0QRgMVv/WX20240228-113337-2x.png"  width="100%" height="80%">
</p>

# Large-scale Multi-modality Models Evaluation Suite

> Accelerating the development of large-scale multi-modality models (LMMs) with `lmms-eval`

🏠 [Homepage](https://lmms-lab.github.io/) | 📚 [Documentation](docs/README.md) | 🤗 [Huggingface Datasets](https://huggingface.co/lmms-lab)

# This Dataset

This is a formatted version of [VistBench](https://visit-bench.github.io/). It is used in our `lmms-eval` pipeline to allow for one-click evaluations of large multi-modality models.

```
@article{bitton2023visit,
  title={Visit-bench: A benchmark for vision-language instruction following inspired by real-world use},
  author={Bitton, Yonatan and Bansal, Hritik and Hessel, Jack and Shao, Rulin and Zhu, Wanrong and Awadalla, Anas and Gardner, Josh and Taori, Rohan and Schimdt, Ludwig},
  journal={arXiv preprint arXiv:2308.06595},
  year={2023}
}
```

Including visit_bench_single.csv and visit_bench_multi.csv, in total 1.2k items. 

Some of them are with `reference_output`, directly copied from [here](https://docs.google.com/spreadsheets/d/1hi8rGXf2WYufkFvGJ2MZ92JNChliM1QEJwZxNboUFlE/edit#gid=696111549).

For each split, please follow the steps here to submit to VisitBench.

## Leaderboard

The link to our public leaderboard is present [here](https://visit-bench.github.io/).

## How to add new models to the Leaderboard?

1. You can access the single-image and multiple-image datasets above.
2. For every instance (row) in the dataset csv, you would have your model's predictions.
3. Create a `predictions.csv` with 4 mandatory columns `instruction`, `instruction_category`, `image` (single-image case) / `images` (multi-image case), `<model name> prediction`. Here, `<model name>`should be your model name with version if multiple-versions are available. 
4. Send a `prediction.csv` to us on `yonatanbitton1@gmail.com`. 
5. We will use our internal prompting sandbox with reference-free GPT-4 as an evaluator.
6. We will add your model to the leaderboard once we receive all the pairwise judgments from the sandbox.  
7. You will receive a confirmation email as soon as your model has been added to the leaderboard.
8. Estimated time from Step 4-7 would be 1-2 weeks, however, we will try to work on your prediction files as soon as they are sent.

Please include in your email 1) a name for your model, 2) your team name (including your affiliation), and optionally, 3) a github repo or paper link.

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)