---
title: MM-RLHF
canonical_url: "https://www.modelscope.cn/datasets/yifanzhang/MM-RLHF"
md_url: "https://www.modelscope.cn/datasets/yifanzhang/MM-RLHF.md"
repository: yifanzhang/MM-RLHF
last_updated: 2025-03-04
license: "Apache License 2.0"
storage_size: "28 GB"
downloads: 30098
stars: 0
---

# MM-RLHF

> MM-RLHF - yifanzhang 在 ModelScope 开源的数据集。https://mm-rlhf.github.io/，目前最大的纯人工标注多模态human preference数据集

yifanzhang/MM-RLHF 是 ModelScope 魔搭社区上的数据集，存储大小 28 GB，采用 Apache License 2.0 许可。

- **Repository**: yifanzhang/MM-RLHF
- **License**: Apache License 2.0
- **Storage size**: 28 GB
- **Downloads**: 30098
- **Stars**: 0
- **Last updated**: 2025-03-04

Source: https://www.modelscope.cn/datasets/yifanzhang/MM-RLHF

---

---


![image/png](https://cdn-uploads.huggingface.co/production/uploads/623d8ca4c29adf5ef6175615/ZxbX8GLRok8cUlwj4q_xo.png)


<font size=3><div align='center' >  
[[📖 arXiv Paper](https://arxiv.org/abs/2502.10391)] 
[[📊 Training Code](https://github.com/Kwai-YuanQi/MM-RLHF)] 
[[📝 Homepage](https://mm-rlhf.github.io/)] 
[[🏆 Reward Model](https://huggingface.co/yifanzhang114/MM-RLHF-Reward-7B-llava-ov-qwen)] 

[[🔮 MM-RewardBench](https://huggingface.co/datasets/yifanzhang114/MM-RLHF-RewardBench)] 
[[🔮 MM-SafetyBench](https://github.com/yfzhang114/mmrlhf-eval)] 
[[📈 Evaluation Suite](https://github.com/yfzhang114/mmrlhf-eval)] 
</div></font>


# The Next Step Forward in Multimodal LLM Alignment

**[2025/02/10]** 🔥 We are proud to open-source **MM-RLHF**, a comprehensive project for aligning Multimodal Large Language Models (MLLMs) with human preferences. This release includes:

- A **high-quality MLLM alignment dataset**.
- A **strong Critique-Based MLLM reward model** and its training algorithm.
- A **novel alignment algorithm MM-DPO**.
- **Two new benchmarks**.

Our dataset and algorithms enable consistent performance improvements across **10 dimensions** and **27 benchmarks** for open-source MLLMs.
<p align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/623d8ca4c29adf5ef6175615/8nVZQd8bfB6NJIixCv6_X.png" width="80%" />
</p>

# Dataset Prepration

<p align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/623d8ca4c29adf5ef6175615/IMKSIpdLlv2GxHcwrKETe.png" width="90%" />
</p>


Download [📊 MM-RLHF Data](https://huggingface.co/datasets/yifanzhang114/MM-RLHF), unzip the image and video datasets, and the final structure should look like this:

```
MM-RLHF
-- | long
-- | mcq
-- | safety
-- | short
-- | data.jsonl
-- | dpo_pairs.jsonl
```

Here, `data.jsonl` contains all labeled information, and `dpo_pairs.jsonl` contains pairs of different ranks used for subsequent DPO and reward model training. The other folders contain the image and video frames.


## Citation

If you find it useful for your research and applications, please cite related papers/blogs using this BibTeX:
```bibtex
@article{zhang2025mm,
  title={MM-RLHF: The Next Step Forward in Multimodal LLM Alignment},
  author={Zhang, Yi-Fan and Yu, Tao and Tian, Haochen and Fu, Chaoyou and Li, Peiyan and Zeng, Jianshu and Xie, Wulin and Shi, Yang and Zhang, Huanyu and Wu, Junkang and others},
  journal={arXiv preprint arXiv:2502.10391},
  year={2025}
}
```