---
title: ultrafeedback-binarized-preferences-cleaned-kto
canonical_url: "https://www.modelscope.cn/datasets/AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto"
md_url: "https://www.modelscope.cn/datasets/AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto.md"
repository: AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto
last_updated: 2024-08-23
license: mit
storage_size: "216 MB"
domain:
  - configs
  - size_categories
  - dataset_info
  - language
  - task_categories
  - pretty_name
tasks:
  - "{config_name=default, data_files=[{split=train, path=data/train-*}]}"
  - "10K<n<100K"
  - dataset_size
  - en
  - text-generation
  - "UltraFeedback Binarized Preferences Cleaned KTO"
downloads: 6230
stars: 1
---

# ultrafeedback-binarized-preferences-cleaned-kto

> ultrafeedback-binarized-preferences-cleaned-kto - AI-ModelScope 在 ModelScope 开源的数据集。UltraFeedback - Binarized using the Average of Preference Ratings (Cleaned) KTO

AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto 是 ModelScope 魔搭社区上的{config_name=default, data_files=[{split=train, path=data/train-*}]}、10K<n<100K、dataset_size数据集，涉及 configs、size_categories、dataset_info 领域，存储大小 216 MB，采用 mit 许可。

- **Repository**: AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto
- **License**: mit
- **Tasks**: {config_name=default, data_files=[{split=train, path=data/train-*}]}, 10K<n<100K, dataset_size, en, text-generation, UltraFeedback Binarized Preferences Cleaned KTO
- **Domain**: configs, size_categories, dataset_info, language, task_categories, pretty_name
- **Storage size**: 216 MB
- **Downloads**: 6230
- **Stars**: 1
- **Last updated**: 2024-08-23

Source: https://www.modelscope.cn/datasets/AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto

---

# UltraFeedback - Binarized using the Average of Preference Ratings (Cleaned) KTO

> A KTO signal transformed version of the highly loved [UltraFeedback Binarized Preferences Cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned), the preferred dataset by Argilla to use from now on when fine-tuning on UltraFeedback

This dataset represents a new iteration on top of [`argilla/ultrafeedback-binarized-preferences`](https://huggingface.co/argilla/ultrafeedback-binarized-preferences),
and is the **recommended and preferred dataset by Argilla to use from now on when fine-tuning on UltraFeedback**.

Read more about Argilla's approach towards UltraFeedback binarization at [`argilla/ultrafeedback-binarized-preferences/README.md`](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences/blob/main/README.md).

## Why KTO?

The [KTO paper](https://arxiv.org/abs/2402.01306) states:

- KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal.
- KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.
- When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.

## Reproduce KTO Transformation

Orginal [UltraFeedback binarized prefrence cleaned DPO dataset](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)

<a target="_blank" href="https://colab.research.google.com/drive/10MwyxzcQogwO8e1ZcVu7aGTQvjXWpFuD?usp=sharing">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
