---
title: orpo-dpo-mix-40k
canonical_url: "https://www.modelscope.cn/datasets/AI-ModelScope/orpo-dpo-mix-40k"
md_url: "https://www.modelscope.cn/datasets/AI-ModelScope/orpo-dpo-mix-40k.md"
repository: AI-ModelScope/orpo-dpo-mix-40k
last_updated: 2024-10-18
license: apache-2.0
storage_size: "121 MB"
domain:
  - dataset_info
  - language
  - pretty_name
  - task_categories
tasks:
  - dataset_size
  - en
  - ORPO-DPO-mix-40k
  - text-generation
downloads: 11401
stars: 3
---

# orpo-dpo-mix-40k

> orpo-dpo-mix-40k - AI-ModelScope 在 ModelScope 开源的数据集。ORPO-DPO-mix-40k v1.2

AI-ModelScope/orpo-dpo-mix-40k 是 ModelScope 魔搭社区上的dataset_size、en、ORPO-DPO-mix-40k数据集，涉及 dataset_info、language、pretty_name 领域，存储大小 121 MB，采用 apache-2.0 许可。

- **Repository**: AI-ModelScope/orpo-dpo-mix-40k
- **License**: apache-2.0
- **Tasks**: dataset_size, en, ORPO-DPO-mix-40k, text-generation
- **Domain**: dataset_info, language, pretty_name, task_categories
- **Storage size**: 121 MB
- **Downloads**: 11401
- **Stars**: 3
- **Last updated**: 2024-10-18

Source: https://www.modelscope.cn/datasets/AI-ModelScope/orpo-dpo-mix-40k

---

# ORPO-DPO-mix-40k v1.2

![image/webp](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/s3uIwTgVl1sTm5_AX3rXH.webp)

This dataset is designed for [ORPO](https://huggingface.co/docs/trl/main/en/orpo_trainer#expected-dataset-format) or [DPO](https://huggingface.co/docs/trl/main/en/dpo_trainer#expected-dataset-format) training.
See [Fine-tune Llama 3 with ORPO](https://huggingface.co/blog/mlabonne/orpo-llama-3) for more information about how to use it.
It is a combination of the following high-quality DPO datasets:

- [`argilla/Capybara-Preferences`](https://huggingface.co/datasets/argilla/Capybara-Preferences): highly scored chosen answers >=5 (7,424 samples)
- [`argilla/distilabel-intel-orca-dpo-pairs`](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs): highly scored chosen answers >=9, not in GSM8K (2,299 samples)
- [`argilla/ultrafeedback-binarized-preferences-cleaned`](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned): highly scored chosen answers >=5 (22,799 samples)
- [`argilla/distilabel-math-preference-dpo`](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo): highly scored chosen answers >=9 (2,181 samples)
- [`unalignment/toxic-dpo-v0.2`](https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2) (541 samples)
- [`M4-ai/prm_dpo_pairs_cleaned`](https://huggingface.co/datasets/M4-ai/prm_dpo_pairs_cleaned) (7,958 samples)
- [`jondurbin/truthy-dpo-v0.1`](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1) (1,016 samples)

Rule-based filtering was applied to remove gptisms in the chosen answers (2,206 samples).

Thanks to [argilla](https://huggingface.co/argilla), [unalignment](https://huggingface.co/unalignment), [M4-ai](https://huggingface.co/M4-ai), and [jondurbin](https://huggingface.co/jondurbin) for providing the source datasets.

## 🔎 Usage

v1.2 adds a `question` column to ensure compatibility with both DPO and ORPO formats in Axolotl.

Here's an example as an ORPO dataset in Axolotl:

```yaml
rl: orpo
orpo_alpha: 0.1
chat_template: chatml
datasets:
  - path: mlabonne/orpo-dpo-mix-40k
    type: chat_template.argilla
    chat_template: chatml
```

For DPO, I recommend using [mlabonne/orpo-dpo-mix-40k-flat](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k-flat) instead.

## Toxicity

Note that ORPO-DPO-mix-40k contains a dataset (`toxic-dpo-v0.2`) designed to prompt the model to answer illegal questions. You can remove it as follows:

```python
dataset = load_dataset('mlabonne/orpo-dpo-mix-40k', split='train')
dataset = dataset.filter(
    lambda r: r["source"] != "toxic-dpo-v0.2"
)
```

## History

I'm saving previous versions of this dataset in different branches.

- [v1.0](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k/tree/v1.0)
