---
title: Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8
canonical_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8"
md_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8.md"
repository: Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8
chinese_name: Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8
last_updated: 2026-06-28
license: apache-2.0
model_type:
  - qwen3_5
architectures:
  - Qwen3_5ForConditionalGeneration
base_model:
  - Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill
base_model_relation: quantized
parameters: 27.8B
tensor_type:
  - BF16
  - F8_E4M3
library_name:
  - safetensors
language:
  - en
  - zh
downloads: 2303
stars: 5
tags:
  - qwen3
  - dense
  - distillation
  - reasoning
  - thinking
  - agentic
  - mtp
  - speculative-decoding
  - fp8
---

# Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8

> Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 - Merkyor 在 ModelScope 开源的模型。Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8

Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 是 ModelScope 魔搭社区上的 27.8B 参数机器学习模型，采用 apache-2.0 许可，基于 Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill 构建。

- **Repository**: Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8
- **License**: apache-2.0
- **Parameters**: 27.8B
- **Base model**: Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill
- **Tags**: qwen3, dense, distillation, reasoning, thinking, agentic, mtp, speculative-decoding, fp8
- **Downloads**: 2303
- **Stars**: 5
- **Last updated**: 2026-06-28

Source: https://www.modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8

---

# Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8

> **🇨🇳 中文 · 🇬🇧 English ⬇️**

---

# 🇨🇳 中文版

[**Qwen3.6-27B-DSV4Pro-Thinking-Distill**](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill) 的 **FP8(block-128 e4m3)+ 原生 MTP** 版 —— 面向 **SGLang / vLLM 服务化**(尤其并发 serving),FP8 张量核 + 批量在这里发挥。

**蒸馏方法、出处归因、评测口径、局限**见[父模型(BF16)卡片](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill)。本卡只讲 FP8 版。

## 这是什么
- **FP8 block-128、e4m3、动态激活**量化蒸馏主干 —— 与官方 `Qwen/Qwen3.6-27B-FP8` 同方案(每 128×128 块一个 `weight_scale_inv`)。norm、embedding、`lm_head`、Gated-DeltaNet 小投影、vision 塔保持 **BF16**(对齐官方布局)。
- 打包**原生 MTP(nextn)头**(`mtp.safetensors`),供 EAGLE/NEXTN 投机解码。
- **构造上近无损** —— block-128 FP8 正是 Qwen 官方"性能与原模型几乎一致"的同精度。

## 速度 —— **蒸馏 FP8** 实测 **DGX Spark(GB10,Blackwell sm_121)**,SGLang `dev-cu13` + flashinfer + EAGLE/NEXTN
| 模式 | 吞吐 | MTP 接受率 | 接受长度 |
|---|---|---|---|
| 单流,**关 MTP**(base) | **~7.7 tok/s** | — | — |
| 单流,**开 MTP** | **~15 tok/s**(代码段 → ~19) | 0.65–0.89 | 2.6–3.55 |
| 并发 ×16 —— *官方 FP8 参考* | **≈136–143 tok/s** 聚合 | — | — |

> **MTP 单流约 2× 加速**(base 7.7 → 15 tok/s,代码这类可预测内容升到 ~19 / 接受率 0.89)。均为**本蒸馏 FP8 实测**(SGLang `dev-cu13`,EAGLE topk=1 / 3 draft / mamba `extra_buffer`):生成正确、无 FP8 损坏。并发数来自结构一致的官方 `Qwen/Qwen3.6-27B-FP8`(蒸馏版并发未单独实测)。

> **FP8+MTP 的主场**:在带宽受限的机器上单流**不是**它的强项(那里 Q4_K_M-imatrix GGUF 更快 ~25 tok/s)—— **它强在并发 serving**:FP8 核 + 批量 + 近无损质量。Blackwell RTX-50(FP8 更快)上更快。

## 质量 —— 同 harness 正式 FP8 评测
本**蒸馏 FP8** 与**官方 base FP8**(`Qwen/Qwen3.6-27B-FP8`)跑*完全相同*的流式 harness(DGX Spark,SGLang `dev-cu13`,thinking-on,`temperature 0.6 / top_p 0.95`,`context 36864`):GPQA-Diamond 全 198 + MMLU-500 5-shot。

| 维度 | **蒸馏 FP8** | Base FP8 | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **82.32%** (163/198) | 66.67% (132/198) | **+15.65** |
| MMLU-500 (5-shot) | **90.00%** (450/500) | 88.60% (443/500) | +1.40 |
| GPQA `finish=length`(跑飞) | **0** | 12 | — |

蒸馏 FP8 在 **GPQA 上 +15.65 分**,而最亮的是收口:**0 次长度截断 vs base 的 12 次**。难题上未蒸馏的 base 思维链收不住、一路撞到 token 上限(那 12 个没答完的判错);蒸馏正是为修这个而训,同时 MMLU 也保持领先。FP8 本身近无损——数据贴合父 BF16 蒸馏版(canonical GPQA 80.81 / MMLU 91.8)。早前 canonical 参考(未在 FP8 下重跑):coding-100 **86** vs 83,Agentic SOLO-20 **16** vs 13。
## 服务化(SGLang,已验证启动)
```bash
SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server \
  --model-path Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer \
  --attention-backend flashinfer --trust-remote-code
```
- 需 **SGLang ≥ 0.5.10**(Qwen3.6)。`extra_buffer` mamba 调度 + `SPEC_V2` 是 Gated-DeltaNet(mamba 式)层做投机解码的必需项;Blackwell 用 `--attention-backend flashinfer`。
- thinking-on,**`temperature=0.6, top_p=0.95`**(切勿 greedy)。

## 文件
- `model-*-of-trunk.safetensors` —— FP8(e4m3, block-128)蒸馏主干 + BF16 norm/embed/lm_head
- `mtp.safetensors` —— 原生 nextn MTP 头(FP8)
- `model-vision.safetensors` —— vision 塔(BF16,文本不用)
- `config.json` 带 `quantization_config`(fp8 e4m3)

---
---

# 🇬🇧 English

**FP8 (block-128 e4m3) + native MTP** build of [**Qwen3.6-27B-DSV4Pro-Thinking-Distill**](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill) — for **SGLang / vLLM serving** (especially concurrent serving), where FP8 tensor cores + batching shine.

For the **distillation method, attribution, eval protocol and limitations**, see the [parent BF16 model card](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill). This card only covers the FP8 build.

## What this is
- **FP8 block-128, e4m3, dynamic activation** quantization of the distilled trunk — same scheme as the official `Qwen/Qwen3.6-27B-FP8` (`weight_scale_inv` per 128×128 block). Norms, embeddings, `lm_head`, the small Gated-DeltaNet projections, and the vision tower are kept **BF16** (matching the official layout).
- **Native MTP (nextn) head** bundled (`mtp.safetensors`) for EAGLE/NEXTN speculative decoding.
- **Near-lossless by construction** — block-128 FP8 is the same precision the Qwen team ships as "performance nearly identical to the original".

## Speed — **distill FP8** measured on **DGX Spark (GB10, Blackwell sm_121)**, SGLang `dev-cu13` + flashinfer + EAGLE/NEXTN
| Mode | Throughput | MTP accept rate | accept length |
|---|---|---|---|
| single-stream, **no MTP** (base) | **~7.7 tok/s** | — | — |
| single-stream, **MTP on** | **~15 tok/s** (→ ~19 on code) | 0.65–0.89 | 2.6–3.55 |
| concurrent ×16 — *official-FP8 ref* | **≈136–143 tok/s** aggregate | — | — |

> **MTP gives ~2× single-stream** (base 7.7 → 15 tok/s, up to ~19 / accept 0.89 on predictable code). All measured on **this distilled FP8** (SGLang `dev-cu13`, EAGLE topk=1 / 3 draft / mamba `extra_buffer`); generation correct, no FP8 corruption. The concurrent figure is from the structurally-identical official `Qwen/Qwen3.6-27B-FP8` (distill concurrent not separately benchmarked).

> **Where FP8+MTP wins**: single-stream on a bandwidth-bound box is *not* its arena (a Q4_K_M-imatrix GGUF is faster there ~25 tok/s) — **FP8+MTP's strength is concurrent serving**: FP8 tensor cores + batching + near-lossless quality. On Blackwell RTX-50 (faster FP8) it's faster still.

## Quality — formal same-harness FP8 eval
Both this **distilled FP8** and the **official base FP8** (`Qwen/Qwen3.6-27B-FP8`) were run through the *identical* streaming harness on **DGX Spark** (SGLang `dev-cu13`, thinking-on, `temperature 0.6 / top_p 0.95`, `context 36864`): GPQA-Diamond full-198 and MMLU-500 5-shot.

| Dimension | **Distill FP8** | Base FP8 | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **82.32%** (163/198) | 66.67% (132/198) | **+15.65** |
| MMLU-500 (5-shot) | **90.00%** (450/500) | 88.60% (443/500) | +1.40 |
| GPQA `finish=length` (runaway) | **0** | 12 | — |

The distilled FP8 is **+15.65 pts on GPQA** over the un-distilled base — and the headline is convergence: **0 length-truncations vs the base's 12**. On hard GPQA items the un-distilled base lets its chain-of-thought run away until it slams into the token ceiling (those 12 unfinished answers are scored wrong); the distill was trained to rein exactly that in, while staying ahead on MMLU too. FP8 itself is near-lossless — numbers track the parent BF16 distill (canonical GPQA 80.81 / MMLU 91.8). Prior canonical reference (not re-run under FP8): coding-100 **86** vs 83, Agentic SOLO-20 **16** vs 13.
## Serving (SGLang, validated launch)
```bash
SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server \
  --model-path Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer \
  --attention-backend flashinfer --trust-remote-code
```
- Requires **SGLang ≥ 0.5.10** (Qwen3.6). The `extra_buffer` mamba scheduler + `SPEC_V2` are required for speculative decoding on the Gated-DeltaNet (mamba-style) layers. On Blackwell use `--attention-backend flashinfer`.
- thinking-on, **`temperature=0.6, top_p=0.95`** (never greedy).

## Files
- `model-*-of-trunk.safetensors` — FP8 (e4m3, block-128) distilled trunk + BF16 norms/embed/lm_head
- `mtp.safetensors` — native nextn MTP head (FP8)
- `model-vision.safetensors` — vision tower (BF16; unused for text)
- `config.json` carries the `quantization_config` (fp8 e4m3)


## Claude Code（实验性 / experimental）

Claude Code 不是本地 GGUF/HF 加载器——它对接兼容 Anthropic `/v1/messages` 的后端、且要求可靠的工具调用，不能直接喂仓名/路径。本 **FP8 仓是 vLLM 路线的直接载体**：

```bash
vllm serve Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 \
  --served-model-name qwen36-27b-fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml
```

Claude Code 里模型名填 `qwen36-27b-fp8`（即 `--served-model-name`，**不要填带 `/` 的仓名**），`ANTHROPIC_BASE_URL` 指向你的 vLLM 端点。要 GGUF + LM Studio 路线见 **[GGUF 仓](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF)**。参考 [vLLM Claude Code](https://docs.vllm.ai/en/stable/serving/integrations/claude_code/)。

---

## Claude Code (experimental)

Claude Code is not a local GGUF/HF loader — it talks to an Anthropic-compatible `/v1/messages` backend and needs reliable tool calling, so you cannot point it at a repo id directly. This **FP8 repo is the direct vLLM path** — serve it with `--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3` (command above), then set Claude Code's model name to the `--served-model-name` (no slashes) and `ANTHROPIC_BASE_URL` to your vLLM endpoint. For the GGUF + LM Studio route, see the **[GGUF repo](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF)**.