---
title: Mantis-Instruct
canonical_url: "https://www.modelscope.cn/datasets/swift/Mantis-Instruct"
md_url: "https://www.modelscope.cn/datasets/swift/Mantis-Instruct.md"
repository: swift/Mantis-Instruct
last_updated: 2024-12-25
license: "Apache License 2.0"
storage_size: "176 GB"
downloads: 12395
stars: 2
---

# Mantis-Instruct

> Mantis-Instruct - swift 在 ModelScope 开源的数据集。Paper | Website | Github | Models | Demo

swift/Mantis-Instruct 是 ModelScope 魔搭社区上的数据集，存储大小 176 GB，采用 Apache License 2.0 许可。

- **Repository**: swift/Mantis-Instruct
- **License**: Apache License 2.0
- **Storage size**: 176 GB
- **Downloads**: 12395
- **Stars**: 2
- **Last updated**: 2024-12-25

Source: https://www.modelscope.cn/datasets/swift/Mantis-Instruct

---

# Mantis-Instruct

[Paper](https://arxiv.org/abs/2405.01483) | [Website](https://tiger-ai-lab.github.io/Mantis/) | [Github](https://github.com/TIGER-AI-Lab/Mantis) | [Models](https://huggingface.co/collections/TIGER-Lab/mantis-6619b0834594c878cdb1d6e4) | [Demo](https://huggingface.co/spaces/TIGER-Lab/Mantis)

## Summaries

Mantis-Instruct is a fully text-image interleaved multimodal instruction tuning dataset, 
containing 721K examples from 14 subsets and covering multi-image skills including co-reference, reasoning, comparing, temporal understanding. 
**It's been used to train Mantis Model families**

- Mantis-Instruct has a total of **721K instances**, consisting of **14 subsets** to cover all the multi-image skills.
- Among the 14 subsets, 10 subsets are from the existing datasets. For example, NLVR2, IconQA, etc for reasoning skill; DreamSim, Birds-to-Words, etc for comparison skill; NExT-QA, STAR, for temporal understanding
- We additionally curate four new datasets LLaVA-665k-multi, LRV-multi to cover coref skill and Contrast-Caption, Multi-VQA to broaden reasoning skill, where Multi-VQA is generated by prompting GPT-4.

![Mantis-Instruct Statistics](https://github.com/TIGER-AI-Lab/Mantis/blob/gh-pages/images/miqa_stat.png?raw=true)
## Loading dataset

- to load the dataset without automatically downloading and process the images
```python
import datasets
dataset = datasets.load_dataset("TIGER-Lab/Mantis-Instruct", "multi_vqa") # revision is 'main' by default
# dataset['train'][0]['images']: image paths relative to the text file, change it to the valid path on your local machine.
```
In this case, you need to manually download the image zips from the [`revision`](https://huggingface.co/datasets/TIGER-Lab/Mantis-Instruct/tree/script) branch of this repo for each subset, and set the prepend the directory of the images.

- to load the dataset that automatically downloads and process the images (**Please run the following codes with datasets==2.18.0** )
```python
import datasets
dataset = datasets.load_dataset("TIGER-Lab/Mantis-Instruct", "multi_vqa", revision="script")
# dataset['train'][0]['images']: processed absolution valid path of the downloaded images on your local machine
```

- to load all the subsets of the images
```python
from datasets import get_dataset_config_names, load_dataset
config_dataset = {}
for config_name in get_dataset_config_names():
    config_dataset[config_name] = load_dataset("TIGER-Lab/Mantis-Instruct", config_name)
```

- to load all the subsets of the images, with automatically downloading
```python
from datasets import get_dataset_config_names, load_dataset
config_dataset = {}
for config_name in get_dataset_config_names():
    config_dataset[config_name] = load_dataset("TIGER-Lab/Mantis-Instruct", config_name, revision="script")
```
## Citation
```
@article{Jiang2024MANTISIM,
  title={MANTIS: Interleaved Multi-Image Instruction Tuning},
  author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
  journal={Transactions on Machine Learning Research},
  year={2024},
  volume={2024},
  url={https://openreview.net/forum?id=skLtdUVaJa}
}
```
