---
title: Qwen3-8B-KTO-ultrafeedback
canonical_url: "https://www.modelscope.cn/models/theGAI/Qwen3-8B-KTO-ultrafeedback"
md_url: "https://www.modelscope.cn/models/theGAI/Qwen3-8B-KTO-ultrafeedback.md"
repository: theGAI/Qwen3-8B-KTO-ultrafeedback
last_updated: 2026-10-12
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
base_model:
  - Qwen/Qwen3-8B
base_model_relation: finetune
parameters: 21.8M
tensor_type:
  - F32
library_name:
  - transformer
  - lora
  - safetensors
language:
  - zh
  - en
downloads: 30
stars: 0
tags:
  - ms-swift
  - LoRA
  - RLHF
  - fine-tuning
  - KTO
---

# Qwen3-8B-KTO-ultrafeedback

> Qwen3-8B-KTO-ultrafeedback - theGAI 在 ModelScope 开源的模型。对照 #1：KTO 只要单条好/坏标注——前景理论让标注成本砍半。

theGAI/Qwen3-8B-KTO-ultrafeedback 是 ModelScope 魔搭社区上的 21.8M 参数text-generation模型，采用 apache-2.0 许可，基于 Qwen/Qwen3-8B 构建。

- **Repository**: theGAI/Qwen3-8B-KTO-ultrafeedback
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 21.8M
- **Base model**: Qwen/Qwen3-8B
- **Tags**: ms-swift, LoRA, RLHF, fine-tuning, KTO
- **Downloads**: 30
- **Stars**: 0
- **Last updated**: 2026-10-12

Source: https://www.modelscope.cn/models/theGAI/Qwen3-8B-KTO-ultrafeedback

---

<p align="center"><img src="banner.svg" width="92%" alt="Qwen3-8B-KTO-ultrafeedback"></p>

<p align="center"><img src="https://skillicons.dev/icons?i=pytorch,python,linux&theme=dark" alt="stack"></p>

<p align="center">
<img src="https://img.shields.io/badge/Method-KTO-EC4899?style=flat-square&logo=probot&logoColor=white">
<img src="https://img.shields.io/badge/Base-Qwen3--8B-475569?style=flat-square">
<img src="https://img.shields.io/badge/Data-ultrafeedback-0F766E?style=flat-square">
<img src="https://img.shields.io/badge/ms--swift-4.5.3-10B981?style=flat-square&logo=python&logoColor=white">
<img src="https://img.shields.io/badge/License-Apache--2.0-64748B?style=flat-square&logo=apache&logoColor=white">
</p>

> 对照 #1：**KTO 只要单条好/坏标注**——前景理论让标注成本砍半。

> *EN · Prospect theory in the loss (λ₋ > λ₊) on binary labels — 50% cheaper annotation; observed KL 0→2.34 with the flat-loss profile that binary signals predict.*

> 🔗 跨平台：Hugging Face 英文卡 [jjshin/Qwen3-8B-KTO-ultrafeedback](https://huggingface.co/jjshin/Qwen3-8B-KTO-ultrafeedback) · GitHub 主页 [jjxyxsjr](https://github.com/jjxyxsjr)（HF 卡为英文精选版，本卡为中文全量版）

## <img height="22" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/database-24.svg"> 训练数据

| 数据集 | 说明 |
|--------|------|
| [`AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto`](https://modelscope.cn/datasets/AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto) | UltraFeedback 二元化偏好（KTO 格式：单条好/坏标注） |

> 数据集 id 取自训练 run 的原始参数（args.json），非事后补记。

## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/light-bulb-24.svg"> 算法说明

<p align="center"><img src="algo.svg" width="96%" alt="算法图解"></p>

KTO（Kahneman–Tversky Optimization）将前景理论的价值函数引入偏好优化：损失侧加权高于收益侧（λ₋ > λ₊），模型对负例的梯度响应强于正例；数据形态为单条二元标注（desirable/undesirable），相比成对偏好对降低 50% 标注成本。观测：KL 散度 0 → 2.34 单调上升（参考策略持续偏移），损失 0.50 → 0.48 的低斜率与二元标签的信息熵上限一致——观测形态与理论预期吻合。

## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/graph-24.svg"> 训练配置（实测值）

| 项 | 值 | 项 | 值 |
|---|---|---|---|
| 学习率 | `5e-06` | LoRA r / α | `8` / `32.0` |
| 训练步数 | 60 steps | 截断长度 | `1024` |
| 形态 | LoRA adapter | 框架 | ms-swift 4.5.3 |

## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/pulse-24.svg"> 训练曲线（MLflow 实测）

<p align="center"><img src="https://quickchart.io/chart?w=560&h=300&format=svg&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%221%22%2C%225%22%2C%2210%22%2C%2220%22%2C%2225%22%2C%2230%22%2C%2235%22%2C%2240%22%2C%2245%22%2C%2255%22%2C%2260%22%5D%2C%22datasets%22%3A%5B%7B%22label%22%3A%22chosen%20%5Cu2713%22%2C%22data%22%3A%5B0.0%2C0.0067%2C0.0643%2C0.0981%2C0.2891%2C0.3348%2C0.3159%2C0.3223%2C0.2173%2C0.3833%2C0.3465%5D%2C%22backgroundColor%22%3A%22%23EC4899%22%7D%2C%7B%22label%22%3A%22rejected%20%5Cu2717%22%2C%22data%22%3A%5B0.0%2C0.0222%2C-0.0082%2C0.1716%2C0.1664%2C0.0251%2C0.2076%2C0.3611%2C0.3883%2C0.3386%2C0.1815%5D%2C%22backgroundColor%22%3A%22%2394A3B8%22%7D%5D%7D%2C%22options%22%3A%7B%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%5Cu504f%5Cu597d%5Cu5956%5Cu52b1%5Cu5206%5Cu5316%22%2C%22fontSize%22%3A14%7D%2C%22legend%22%3A%7B%22position%22%3A%22bottom%22%2C%22labels%22%3A%7B%22boxWidth%22%3A12%7D%7D%7D%7D" alt="训练指标图"></p>
<p align="center"><img src="https://quickchart.io/chart?w=560&h=300&format=svg&c=%7B%22type%22%3A%22line%22%2C%22data%22%3A%7B%22labels%22%3A%5B%221%22%2C%225%22%2C%2210%22%2C%2220%22%2C%2225%22%2C%2230%22%2C%2235%22%2C%2240%22%2C%2245%22%2C%2255%22%2C%2260%22%5D%2C%22datasets%22%3A%5B%7B%22label%22%3A%22%5Cu5956%5Cu52b1%5Cu8fb9%5Cu9645%20margins%22%2C%22data%22%3A%5B0.0%2C-0.015498%2C0.07253%2C-0.073438%2C0.122765%2C0.309696%2C0.108257%2C-0.038831%2C-0.171022%2C0.044668%2C0.164991%5D%2C%22borderColor%22%3A%22%23EC4899%22%2C%22backgroundColor%22%3A%22%23EC489930%22%2C%22fill%22%3Atrue%2C%22pointRadius%22%3A1.5%2C%22borderWidth%22%3A2.2%2C%22tension%22%3A0.25%7D%5D%7D%2C%22options%22%3A%7B%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%5Cu5956%5Cu52b1%5Cu8fb9%5Cu9645%20margins%22%2C%22fontSize%22%3A14%7D%2C%22legend%22%3A%7B%22display%22%3Afalse%7D%7D%7D" alt="训练指标图"></p>
<p align="center"><img src="https://quickchart.io/chart?w=560&h=300&format=svg&c=%7B%22type%22%3A%22scatter%22%2C%22data%22%3A%7B%22datasets%22%3A%5B%7B%22label%22%3A%22KL%20%5Cu6563%5Cu5ea6%22%2C%22data%22%3A%5B%7B%22x%22%3A1%2C%22y%22%3A0.0%7D%2C%7B%22x%22%3A5%2C%22y%22%3A0.20339%7D%2C%7B%22x%22%3A10%2C%22y%22%3A0.47187%7D%2C%7B%22x%22%3A15%2C%22y%22%3A0.32437%7D%2C%7B%22x%22%3A20%2C%22y%22%3A1.05702%7D%2C%7B%22x%22%3A25%2C%22y%22%3A1.54153%7D%2C%7B%22x%22%3A30%2C%22y%22%3A1.80639%7D%2C%7B%22x%22%3A35%2C%22y%22%3A2.10265%7D%2C%7B%22x%22%3A40%2C%22y%22%3A2.27221%7D%2C%7B%22x%22%3A45%2C%22y%22%3A2.33647%7D%2C%7B%22x%22%3A50%2C%22y%22%3A2.39261%7D%2C%7B%22x%22%3A55%2C%22y%22%3A2.71911%7D%2C%7B%22x%22%3A60%2C%22y%22%3A2.34467%7D%5D%2C%22pointRadius%22%3A3.5%2C%22backgroundColor%22%3A%22%23EC4899%22%2C%22showLine%22%3Afalse%7D%5D%7D%2C%22options%22%3A%7B%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22KL%20%5Cu6563%5Cu5ea6%22%2C%22fontSize%22%3A14%7D%2C%22legend%22%3A%7B%22display%22%3Afalse%7D%7D%7D" alt="训练指标图"></p>
<p align="center"><img src="https://quickchart.io/chart?w=560&h=300&format=svg&c=%7B%22type%22%3A%22line%22%2C%22data%22%3A%7B%22labels%22%3A%5B%221%22%2C%225%22%2C%2210%22%2C%2215%22%2C%2220%22%2C%2225%22%2C%2230%22%2C%2235%22%2C%2240%22%2C%2245%22%2C%2250%22%2C%2255%22%2C%2260%22%5D%2C%22datasets%22%3A%5B%7B%22label%22%3A%22loss%20%5Cu6536%5Cu655b%22%2C%22data%22%3A%5B0.5%2C0.499413%2C0.49686%2C0.496515%2C0.503744%2C0.482768%2C0.456123%2C0.484571%2C0.489373%2C0.511724%2C0.451839%2C0.477808%2C0.480945%5D%2C%22borderColor%22%3A%22%23EC4899%22%2C%22backgroundColor%22%3A%22%23EC489930%22%2C%22fill%22%3Atrue%2C%22pointRadius%22%3A1.5%2C%22borderWidth%22%3A2.2%2C%22tension%22%3A0.25%7D%5D%7D%2C%22options%22%3A%7B%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22loss%20%5Cu6536%5Cu655b%22%2C%22fontSize%22%3A14%7D%2C%22legend%22%3A%7B%22display%22%3Afalse%7D%7D%7D" alt="训练指标图"></p>

> 数据来自 90.2 实验跟踪系统（MLflow）逐步记录；QuickChart × Image-Charts 双 CDN 渲染——分化用分组柱状、终值用仪表盘、边际与 loss 用面积线、KL 用散点、token 准确率用柱状/折线；无信息量的平直曲线已筛除。

## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/rocket-24.svg"> 快速上手

<details open>
<summary><b>① PEFT 挂载推理</b>（最轻量）</summary>

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base, "theGAI/Qwen3-8B-KTO-ultrafeedback"); model.eval()
msgs = [{"role": "user", "content": "人在职场被抢了功劳，该怎么调整心态？"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[1]:], skip_special_tokens=True))
```
</details>

<details>
<summary><b>② swift CLI 一条命令</b></summary>

```bash
swift infer --model Qwen/Qwen3-8B --adapters theGAI/Qwen3-8B-KTO-ultrafeedback --stream true
```
</details>

<details>
<summary><b>③ 合并导出 + vLLM 部署</b></summary>

```bash
swift export --model Qwen/Qwen3-8B --adapters theGAI/Qwen3-8B-KTO-ultrafeedback --merge_lora true --output_dir ./merged
vllm serve ./merged --served-model-name Qwen3-8B-KTO-ultrafeedback
```
</details>

## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/beaker-24.svg"> 复现训练

```bash
swift rlhf --rlhf_type kto --model Qwen/Qwen3-8B \
  --dataset AI-ModelScope/ultrafeedback-binarized-preferences-cleaned-kto --train_type lora --lora_rank 8
```

完整超参：[theGAI/swift-ft-recipes](https://modelscope.cn/datasets/theGAI/swift-ft-recipes) 数据集（`Qwen3-8B-KTO-ultrafeedback.json`）

## theGAI 方法谱系（32 模型 · 24 法）

| 家族 | 模型 |
|---|---|
| ![SFT](https://img.shields.io/badge/SFT-10B981?style=flat-square) | [MiniCPM5-1B-FullSFT](https://modelscope.cn/models/theGAI/MiniCPM5-1B-FullSFT-swift-mixture)（全参） · [GLM-4.7-QLoRA](https://modelscope.cn/models/theGAI/GLM-4.7-Flash-QLoRA-SFT-swift-mixture)（61B） · [Qwen3.5-35B-QLoRA](https://modelscope.cn/models/theGAI/Qwen3.5-35B-A3B-QLoRA-SFT-swift-mixture)（35B MoE） · [GLM-4.6V-ChartQA](https://modelscope.cn/models/theGAI/GLM-4.6V-Flash-LoRA-SFT-ChartQA)（视觉真数据） · [Omni-aishell](https://modelscope.cn/models/theGAI/Qwen2.5-Omni-3B-LoRA-SFT-aishell)（语音真数据） |
| ![RL](https://img.shields.io/badge/RL-F97316?style=flat-square) | [GRPO-MATH](https://modelscope.cn/models/theGAI/Qwen3.5-4B-GRPO-MATH)（基型） · [DAPO-fix](https://modelscope.cn/models/theGAI/Qwen3.5-4B-GRPO-DAPO-MATH-fix) / [GSPO-fix](https://modelscope.cn/models/theGAI/Qwen3.5-4B-GRPO-GSPO-MATH-fix) / [RLOO-fix](https://modelscope.cn/models/theGAI/Qwen3.5-4B-GRPO-RLOO-MATH-fix) / [RPP-fix](https://modelscope.cn/models/theGAI/Qwen3.5-4B-GRPO-RPP-MATH-fix)（修复版 ×4，reward 0→0.2） · [MultiTurn](https://modelscope.cn/models/theGAI/Qwen3.5-4B-GRPO-MultiTurn-MATH)（Agentic） · [OPD-RL](https://modelscope.cn/models/theGAI/Qwen3-VL-2B-GRPO-OPDRL-gsm8k)（蒸馏） |
| ![PREF](https://img.shields.io/badge/PREF-EC4899?style=flat-square) | [Intern-S1-DPO](https://modelscope.cn/models/theGAI/Intern-S1-mini-DPO-Zhihu-RLHF) · [KTO](https://modelscope.cn/models/theGAI/Qwen3-8B-KTO-ultrafeedback) · [ORPO](https://modelscope.cn/models/theGAI/Qwen3-8B-ORPO-hhrlhf) · [SimPO](https://modelscope.cn/models/theGAI/Qwen3-8B-SimPO-zhihu) · [CPO](https://modelscope.cn/models/theGAI/Qwen3-8B-CPO-zhihu) · [VL-8B-DPO](https://modelscope.cn/models/theGAI/Qwen3-VL-8B-LoRA-DPO-zhihu)（多模态） |
| ![RERANK/EMBED](https://img.shields.io/badge/RERANK/EMBED-3B82F6?style=flat-square) | [bge-SciDocs](https://modelscope.cn/models/theGAI/bge-reranker-v2-m3-LoRA-rerank-SciDocs)（重排真数据） · [gte-zhihu](https://modelscope.cn/models/theGAI/gte-modernbert-LoRA-embed-zhihu)（嵌入真数据） |
| ![决策](https://img.shields.io/badge/决策/自进化-0EA5E9?style=flat-square) | [ReFT](https://modelscope.cn/models/theGAI/Qwen3.5-4B-ReFT-gsm8k)（拒绝采样） · [laya](https://modelscope.cn/models/theGAI/laya-fullft-typed-decisions)（RLCD 全参） · [von](https://modelscope.cn/models/theGAI/von-RLCD-synthetic-8k)（合成数据 RLCD） · [nimble-9B](https://modelscope.cn/models/theGAI/Qwen3.5-9B-LoRA-schema-classification)（schema 分类） |
| ![对照](https://img.shields.io/badge/对照-64748B?style=flat-square) | 冒烟版 ×4（[视觉](https://modelscope.cn/models/theGAI/GLM-4.6V-Flash-LoRA-SFT-vision) / [语音](https://modelscope.cn/models/theGAI/Qwen2.5-Omni-3B-LoRA-SFT-audio) / [重排](https://modelscope.cn/models/theGAI/bge-reranker-v2-m3-LoRA-rerank) / [嵌入](https://modelscope.cn/models/theGAI/gte-modernbert-base-LoRA-embed)）+ GRPO gsm8k 初版 ×4（reward 恒零→修复实录的对照组） |

## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/law-24.svg"> 声明

- LoRA 轻量形态，社区惯例可直接分发使用；遵循基模许可协议
- 训练于私有 GPU 集群，超参见 recipes 数据集存档## <img height="24" src="https://cdn.jsdelivr.net/npm/@primer/octicons@19.8.0/build/svg/graph-24.svg"> 独立评估（ARC 200 题）

ARC（AI2 科学问答）200 例 mean_acc = **0.89**；裸基座探针 0.920 作对照——域内训练（数学/偏好/多模态指令）的增益不外溢到科学知识问答属预期，分数与基座水位的关系是 adapter 无副作用的证据。原始报告存平台 S3 `eval-results/`。


### 加深实验：判别面零迁移（2026-10-04 平台实验，重要诚实发现）

"步数不足"假设被证伪：DPO 300 步后 loss 距 ln2 理论零点仅 0.009（比 80 步分化更小）；KTO 300 步训练面大幅在动（margins 0→5.31）且 LoRA 权重非零——**但 held-out 对的 logp 判别与 base 逐位同分（en 0.61 / zh 0.44，merge 制评测三臂一致）**。

怎么读：本卡的准确率数字是**训练批次口径**（训练时 rewards/accuraries，真实有效）；加深实验证明的是**该配置的偏好训练不迁移到 held-out logp 判别面**——训练域 margin 在学，泛化判别不动。偏好家族的正确叙事是"**方法-行为面脱节**（双层口径实证）"而非"训练不足"：logp 判别（三臂逐位同分）与生成胜率（50 对成对生成 + LLM-judge 位置交换双跑，3:3:44 完全对称、decided 胜率 0.50）**双口径均测不出微调与 base 的差异**——量尺假设关闭，margins 变化局限在训练样本上。β 线收手（swift 4.5.3 CLI 未暴露 pref_beta + 双口径无差异下预期收益低优先）；若续试方向为配置文件级 β 或 target_modules。产物 `prod-verify/pref-deep-*.json` + `winrate-kto300-vs-base.json`。
