---
title: Img-Diff
canonical_url: "https://www.modelscope.cn/datasets/Data-Juicer/Img-Diff"
md_url: "https://www.modelscope.cn/datasets/Data-Juicer/Img-Diff.md"
repository: Data-Juicer/Img-Diff
last_updated: 2024-12-20
license: "Apache License 2.0"
storage_size: "4.6 GB"
domain:
  - viewer
tasks:
  - false
downloads: 314
stars: 1
---

# Img-Diff

> Img-Diff - Data-Juicer 在 ModelScope 开源的数据集。Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

Data-Juicer/Img-Diff 是 ModelScope 魔搭社区上的false数据集，涉及 viewer 领域，存储大小 4.6 GB，采用 Apache License 2.0 许可。

- **Repository**: Data-Juicer/Img-Diff
- **License**: Apache License 2.0
- **Tasks**: false
- **Domain**: viewer
- **Storage size**: 4.6 GB
- **Downloads**: 314
- **Stars**: 1
- **Last updated**: 2024-12-20

Source: https://www.modelscope.cn/datasets/Data-Juicer/Img-Diff

---

# Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

我们发布了 **Img-Diff** 数据集，一份高质量的专注于描述图像之间的差异区域的合成数据集，以用于多模态大模型（MLLMs）的训练。更多信息请移步[论文](https://arxiv.org/abs/2408.04594)与[代码](https://github.com/modelscope/data-juicer/tree/ImgDiff).



> **Abstract:** High-performance Multimodal Large Language Models (MLLMs) rely heavily on data quality. This study introduces a novel dataset named Img-Diff, designed to enhance fine-grained image recognition in MLLMs by leveraging insights from contrastive learning and image difference captioning. By analyzing object differences between similar images, we challenge models to identify both matching and distinct components. We utilize the Stable-Diffusion-XL model and advanced image editing techniques to create pairs of similar images that highlight object replacements. Our methodology includes a Difference Area Generator for object differences identifying, followed by a Difference Captions Generator for detailed difference descriptions. The result is a relatively small but high-quality dataset of "object replacement" samples. We use the the proposed dataset to finetune state-of-the-art (SOTA) MLLMs such as MGM-7B, yielding comprehensive improvements of performance scores over SOTA models that trained with larger-scale datasets, in numerous image difference and Visual Question Answering tasks. For instance, our trained models notably surpass the SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate alternative methods for generating image difference data through "object removal" and conduct a thorough evaluation to confirm the dataset's diversity, quality, and robustness, presenting several insights on the synthesis of such a contrastive dataset. We release our codes and dataset, to encourage further research and advance the field of multimodal data synthesis and enhancement of MLLMs' fundamental capabilities for image understanding.


![image_pairs_examples](https://image-qr.pages.dev/file/89891a614183f79792043.jpg)
**Picture1**: Examples of Img-Diff Image Pairs: The top row shows "object replacement" image pairs, while the bottom row shows "object removal" image pairs.

![object_replacement_overview](https://img.alicdn.com/imgextra/i1/O1CN01Ut5eAM1TseaW5g17W_!!6000000002438-2-tps-3970-1778.png)
**Picture2**: Illustration of the generation process for “object replacement” data within Img-Diff.


## 文件描述：
- img_diff_object_replacement.json为“物体替换”类图像对的标注信息；
- img_diff_object_removal.json为“物体抹除”类图像对的标注信息；
- object_replacement.zip为“物体替换”类图像对的图像压缩包；
- object_removal.zip为“物体抹除”类图像对的图像压缩包；
- object_replacement_no_bbox.zip为“物体替换”类图像对（图像中不画红框）的图像压缩包，提供给不需要高亮差异区域的任务使用。


## 引用：
如果您觉得我们的工作对您的研究有帮助，请考虑引用我们的论文：
```
@misc{jiao2024imgdiffcontrastivedatasynthesis,
      title={Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models}, 
      author={Qirui Jiao and Daoyuan Chen and Yilun Huang and Bolin Ding and Yaliang Li and Ying Shen},
      year={2024},
      eprint={2408.04594},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.04594}, 
}
```
感谢您对本项目的关注与喜爱！