---
title: colorswap
canonical_url: "https://www.modelscope.cn/datasets/stanfordnlp/colorswap"
md_url: "https://www.modelscope.cn/datasets/stanfordnlp/colorswap.md"
repository: stanfordnlp/colorswap
last_updated: 2025-10-12
license: "[mit]"
storage_size: "2.6 GB"
downloads: 375
stars: 0
---

# colorswap

> colorswap - stanfordnlp 在 ModelScope 开源的数据集。ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation

stanfordnlp/colorswap 是 ModelScope 魔搭社区上的数据集，存储大小 2.6 GB，采用 [mit] 许可。

- **Repository**: stanfordnlp/colorswap
- **License**: [mit]
- **Storage size**: 2.6 GB
- **Downloads**: 375
- **Stars**: 0
- **Last updated**: 2025-10-12

Source: https://www.modelscope.cn/datasets/stanfordnlp/colorswap

---

# ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation

## Dataset Description

ColorSwap is a dataset designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a caption-image pair, along with a "color-swapped" pair. Crucially, the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. 

Paper: Coming soon!

## Usage

You can download the dataset directly from the Hugging Face API with the following code:

```python
from datasets import load_dataset

dataset = load_dataset("stanfordnlp/colorswap", use_auth_token=True)
```

Please make sure to install the `datasets` library and use the `use_auth_token` parameter to authenticate with the Hugging Face API.

An example of the dataset is as follows:

```python
[
    {
        'id': 0, 
        'image_1': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x14D908B20>, 
        'image_2': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x14D9DCE20>, 
        'caption_1': 'someone holding a yellow umbrella wearing a white dress', 
        'caption_2': 'someone holding a white umbrella wearing a yellow dress', 
        'image_source': 'midjourney', 
        'caption_source': 'human'
    }
    ...
]
```

## Evaluations

[This Google Colab](https://colab.research.google.com/drive/1EWPsSklfq49WiX2nUyOTmKZftU0AC4YL?usp=sharing) showcases our ITM model evaluations.

Please refer to our Github repository for the VLM evaluations: [ColorSwap](https://github.com/Top34051/colorswap).

## Citation

If you find our work useful, please cite the following paper:

```
@article{burapacheep2024colorswap,
    author    = {Jirayu Burapacheep and Ishan Gaur and Agam Bhatia and Tristan Thrush},
    title     = {ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation},
    journal   = {arXiv},
    year      = {2024},
}
```