---
title: DialFactSum-Base-8B
canonical_url: "https://www.modelscope.cn/models/dada122/DialFactSum-Base-8B"
md_url: "https://www.modelscope.cn/models/dada122/DialFactSum-Base-8B.md"
repository: dada122/DialFactSum-Base-8B
last_updated: 2026-04-24
pipeline_tag: text-summarization
tasks:
  - text-summarization
model_type:
  - qwen3
architectures:
  - Qwen3ForCausalLM
parameters: 8.2B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 15
stars: 0
---

# DialFactSum-Base-8B

> DialFactSum-Base-8B - dada122 在 ModelScope 开源的模型。DialFactSum: RoSE Benchmark Evaluation on Salesforce/rose Dataset

dada122/DialFactSum-Base-8B 是 ModelScope 魔搭社区上的 8.2B 参数text-summarization模型，可用 deploy_task text/emb、lmdeploy_turbomind 0.9.1、sglang 0.5.2 部署。

- **Repository**: dada122/DialFactSum-Base-8B
- **Tasks**: text-summarization
- **Parameters**: 8.2B
- **Inference backends**: deploy_task text/emb, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Downloads**: 15
- **Stars**: 0
- **Last updated**: 2026-04-24

Source: https://www.modelscope.cn/models/dada122/DialFactSum-Base-8B

---

# DialFactSum: RoSE Benchmark Evaluation on Salesforce/rose Dataset

This repository presents the evaluation results of our **DialFactSum** models on the **Salesforce/rose** dataset under the **RoSE (Revisiting, Organizing, and Structuring Evaluation for Dialogue Summarization)** Benchmark.

We report the standard **ACU** and **Normalized ACU** metrics, demonstrating that our **DialFactSum-ACU-8B** (trained with GRPO) achieves state-of-the-art performance compared to existing strong baselines.  
All baseline results have been re‑evaluated using **GPT‑4o + G‑eval** for a fair and unified comparison.

## Dataset & Benchmark

- **Dataset:** [Salesforce/rose](https://huggingface.co/datasets/Salesforce/rose)
- **Benchmark:** RoSE Benchmark (Dialogue Summarization)
- **Evaluation Metrics:**
    - **ACU (Atomic Content Unit):** Measures the precision of atomic factual units in the summary.
    - **Normalized ACU:** A normalized version of ACU to account for length variations.
- **Unified Evaluation Protocol:** GPT‑4o with G‑eval (consistent across all models)

## Model Performance & Comparison

We evaluate our models against a comprehensive set of dialogue summarization baselines. Our approaches leverage **Qwen3-8B** fine‑tuned with **SFT (Supervised Fine‑Tuning)** and **GRPO (Group Relative Policy Optimization)** .

### Quantitative Results

Below is the full leaderboard comparison on the RoSE Benchmark test split, with all baseline scores obtained under the same **GPT‑4o + G‑eval** evaluator.

| Model | ACU ↑ | Normalized ACU ↑ |
| :--- | :---: | :---: |
| **DialFactSum-ACU-8B (T=0)** | **0.6947** | **0.4649** |
| **DialFactSum-ACU-8B (Multi-Sample, T=0.7)** | **0.6928** | **0.4540** |
| DialFactSum-Base-8B (T=0) | 0.5205 | 0.4421 |
| DialFactSum-Base-8B (Multi-Sample, T=0.7) | 0.4831 | 0.4011 |
| Ctrl‑DiaSumm | 0.5568 | 0.4433 |
| MV‑BART | 0.5334 | 0.4166 |
| PLM‑BART | 0.5115 | 0.4165 |
| BART | 0.4820 | 0.3972 |
| CODS | 0.4396 | 0.3713 |
| PEGASUS | 0.4341 | 0.3736 |
| S‑BART | 0.4111 | 0.3352 |
| UniLM | 0.3705 | 0.3137 |

### Visual Comparison

The figures below provide an intuitive comparison of model performance across the two metrics. Our **DialFactSum-ACU-8B** variants are highlighted in red, clearly demonstrating a substantial margin over all prior baselines.

#### 1. ACU Score Comparison

![ACU Comparison](./rose_benchmark_ACU_comparison.png)

#### 2. Normalized ACU Score Comparison

![Normalized ACU Comparison](./rose_benchmark_Normalized_ACU_comparison.png)

### Key Findings & Analysis

1.  **State‑of‑the‑Art Results:** Our **DialFactSum-ACU-8B** models significantly outperform all previous baselines. The **Base (T=0)** configuration achieves the highest scores on both **ACU (0.6947)** and **Normalized ACU (0.4649)** .
2.  **GRPO vs. SFT:** The models trained with **GRPO** exhibit a substantial performance leap over the standard **SFT** approach (+17.4 absolute points in ACU). This validates the effectiveness of reinforcement learning from preferences for improving factual consistency and content selection in dialogue summarization.
3.  **Comparison to Baselines (Unified GPT‑4o Evaluation):** Our DialFactSum-ACU-8B model outperforms the previous best baseline, `Ctrl‑DiaSumm`, by **13.8 percentage points** in ACU (0.6947 vs. 0.5568), demonstrating a clear and consistent advantage under the same evaluation protocol.