---
title: Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8
canonical_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8"
md_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8.md"
repository: Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8
chinese_name: Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8
last_updated: 2026-06-28
license: apache-2.0
model_type:
  - qwen3_5_moe
architectures:
  - Qwen3_5MoeForConditionalGeneration
base_model:
  - Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill
base_model_relation: quantized
parameters: 37.0B
tensor_type:
  - F8_E4M3
  - BF16
library_name:
  - safetensors
language:
  - zh
  - en
downloads: 462
stars: 2
tags:
  - qwen3
  - moe
  - a3b
  - distillation
  - reasoning
  - thinking
  - agentic
  - mtp
  - speculative-decoding
  - fp8
---

# Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8

> Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 - Merkyor 在 ModelScope 开源的模型。Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8

Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 是 ModelScope 魔搭社区上的 37.0B 参数机器学习模型，采用 apache-2.0 许可，基于 Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 构建。

- **Repository**: Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8
- **License**: apache-2.0
- **Parameters**: 37.0B
- **Base model**: Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill
- **Tags**: qwen3, moe, a3b, distillation, reasoning, thinking, agentic, mtp, speculative-decoding, fp8
- **Downloads**: 462
- **Stars**: 2
- **Last updated**: 2026-06-28

Source: https://www.modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8

---

# Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8

> **🇨🇳 中文 · 🇬🇧 English ⬇️**

---

# 🇨🇳 中文版

[**Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill**](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill) 的 **FP8(block-128 e4m3)+ 原生 MTP** 版 —— 面向 **SGLang / vLLM 服务化**。35B-A3B 是 **MoE(256 专家,3B 激活)**,稀疏激活让它**单流就很快**。

**蒸馏方法、出处归因、评测口径、局限**见[父模型(BF16)卡片](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill)。本卡只讲 FP8 版。

## 这是什么
- **FP8 block-128、e4m3、动态激活**量化蒸馏主干 —— 与官方 `Qwen/Qwen3.6-35B-A3B-FP8` 同方案(每 128×128 块一个 `weight_scale_inv`)。**256 个专家逐专家(per-expert)量化**(gate/up/down 拆分对齐官方布局,SGLang/vLLM 直认)。norm、embedding、`lm_head`、router、Gated-DeltaNet 小投影、shared_expert 之外的 BF16 项、vision 塔保持 **BF16**。
- 打包**原生 MTP(nextn)头**(`mtp.safetensors`),供 EAGLE/NEXTN 投机解码。
- **构造上近无损** —— block-128 FP8 正是 Qwen 官方"性能与原模型几乎一致"的同精度。

## 速度 —— **蒸馏 FP8** 实测 **DGX Spark(GB10,Blackwell sm_121)**,SGLang `dev-cu13` + flashinfer + EAGLE/NEXTN
| 模式 | 吞吐 | MTP 接受率 | 接受长度 |
|---|---|---|---|
| 单流,**关 MTP**(base) | **~51 tok/s** | — | — |
| 单流,**开 MTP** | **~63–65 tok/s** | 0.68–0.78 | 2.7–3.1 |

> **MTP 单流约 1.25×(+25%)加速**(base ~51 → 63–65 tok/s,接受率稳定 ~0.75 / 接受长度 ~3.0)。均为**本蒸馏 FP8 实测**(SGLang `dev-cu13`,EAGLE topk=1 / 3 draft / mamba `extra_buffer`):生成正确、无 FP8 损坏。
>
> **增益温和但单流本就快**:MTP 增益**温和(~25%)** —— A3B 稀疏、base 本已快,投机空间小(**密度律**:激活越稀疏,投机解码 ROI 越低)。但原生 nextn 够便宜,**仍净正收益**(不像外挂 drafter 在稀疏模型上反亏)。真正红利是**稀疏让单流就快**:35B-A3B(3B 激活)51→65 tok/s 远超 dense 27B(~15 tok/s)。Blackwell RTX-50(FP8 更快)上更快。

## 质量
FP8 近无损,且在 **GB10 上冒烟验证正确**(中英推理、`<think>` + 收口结构完整,无 FP8 损坏)。质量 **≈ 父蒸馏版**,其 canonical(thinking-on)数据:

| 维度 | 父蒸馏版 | 原版 base | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **80.3%**(32K) | 72.7% | **+7.6** |
| MMLU-500 (5-shot) | **90.2%** | 91.4% | -1.2 |
| GPQA 未收口空答 (parse_fail) | **1** | 12 | **-11** |

知识广度基本不变,**硬推理显著提升(GPQA +7.6pp)**,且"想完就收口"(空答 12→1)。

## 多模态说明
35B-A3B 的**多模态架构继承自原版 base**(vision 塔为官方原版,已随 FP8 打包,`outside.safetensors`)。**蒸馏仅针对文本推理 / agentic**,未训练视觉。FP8 版可走 SGLang/vLLM 多模态;**GGUF 纯文本**(视觉单独走 [父仓 `gguf/` 里的 mmproj 文件](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill))。

## 用法(SGLang + MTP)
```bash
docker run --gpus all --network host lmsysorg/sglang:dev-cu13 \
  python3 -m sglang.launch_server \
  --model-path Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer --attention-backend flashinfer \
  --trust-remote-code --mem-fraction-static 0.75
```
> **采样**:`temperature 0.6 / top_p 0.95`(thinking 模型必需,greedy 会重复死循环)。

---

# 🇬🇧 English

The **FP8 (block-128 e4m3) + native MTP** build of [**Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill**](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill) — for **SGLang / vLLM serving**. 35B-A3B is **MoE (256 experts, 3B active)**; sparse activation makes it **fast even single-stream**.

For **distillation method, attribution, eval methodology, and limitations**, see the [parent (BF16) card](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill). This card covers only the FP8 build.

## What this is
- **FP8 block-128, e4m3, dynamic activation** quant of the distilled backbone — same scheme as official `Qwen/Qwen3.6-35B-A3B-FP8` (one `weight_scale_inv` per 128×128 block). **All 256 experts quantized per-expert** (gate/up/down split to match the official layout, loads directly in SGLang/vLLM). norm, embedding, `lm_head`, router, Gated-DeltaNet small projections, and the vision tower stay **BF16**.
- Bundles the **native MTP (nextn) head** (`mtp.safetensors`) for EAGLE/NEXTN speculative decoding.
- **Near-lossless by construction** — block-128 FP8 is Qwen's own "virtually identical to the original" precision.

## Speed — **this distilled FP8**, measured on **DGX Spark (GB10, Blackwell sm_121)**, SGLang `dev-cu13` + flashinfer + EAGLE/NEXTN
| Mode | Throughput | MTP accept rate | accept len |
|---|---|---|---|
| Single-stream, **MTP off** (base) | **~51 tok/s** | — | — |
| Single-stream, **MTP on** | **~63–65 tok/s** | 0.68–0.78 | 2.7–3.1 |

> **~1.25× (+25%) single-stream speedup** from MTP (base ~51 → 63–65 tok/s, stable accept ~0.75 / accept len ~3.0). All **measured on this distilled FP8** (SGLang `dev-cu13`, EAGLE topk=1 / 3 draft / mamba `extra_buffer`): correct generation, no FP8 corruption.
>
> **Modest MTP gain, but already-fast single-stream**: the MTP single-stream gain is **modest (~25%)** — A3B is sparse and the base is already fast, so speculation headroom is limited (**density rule**: the sparser the activation, the lower the speculative-decoding ROI). But the native nextn head is cheap enough to stay **net-positive** (unlike an external drafter, which *loses* on sparse models). The real payoff is that **sparsity makes single-stream fast**: 35B-A3B (3B active) 51→65 tok/s, far above the dense 27B (~15 tok/s). Faster on Blackwell RTX-50 (faster FP8).

## Quality
FP8 is near-lossless and **smoke-verified correct on GB10** (EN/ZH reasoning, intact `<think>` + closure, no FP8 corruption). Quality **≈ parent distill**, whose canonical (thinking-on) numbers:

| Dimension | Parent distill | Base | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **80.3%** (32K) | 72.7% | **+7.6** |
| MMLU-500 (5-shot) | **90.2%** | 91.4% | -1.2 |
| GPQA non-closure empty (parse_fail) | **1** | 12 | **-11** |

## Multimodal note
The multimodal architecture is **inherited from the original base** (the vision tower is the stock official one, bundled with the FP8 as `outside.safetensors`). **Distillation targeted text reasoning / agentic only**, not vision. The FP8 build supports SGLang/vLLM multimodal; **GGUF is text-only** (vision via the separate mmproj in the [parent repo's `gguf/`](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill)).

## Usage (SGLang + MTP)
```bash
docker run --gpus all --network host lmsysorg/sglang:dev-cu13 \
  python3 -m sglang.launch_server \
  --model-path nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer --attention-backend flashinfer \
  --trust-remote-code --mem-fraction-static 0.75
```
> **Sampling**: `temperature 0.6 / top_p 0.95` (required for thinking models; greedy loops).


## Claude Code（实验性 / experimental）

Claude Code 不是本地 GGUF/HF 加载器——它对接兼容 Anthropic `/v1/messages` 的后端、且要求可靠的工具调用，不能直接喂仓名/路径。本 **FP8 仓是 vLLM 路线的直接载体**：

```bash
vllm serve Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8 \
  --served-model-name qwen36-35b-fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml
```

Claude Code 里模型名填 `qwen36-35b-fp8`（即 `--served-model-name`，**不要填带 `/` 的仓名**），`ANTHROPIC_BASE_URL` 指向你的 vLLM 端点。要 GGUF + LM Studio 路线见 **[GGUF 仓](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF)**。参考 [vLLM Claude Code](https://docs.vllm.ai/en/stable/serving/integrations/claude_code/)。

---

## Claude Code (experimental)

Claude Code is not a local GGUF/HF loader — it talks to an Anthropic-compatible `/v1/messages` backend and needs reliable tool calling, so you cannot point it at a repo id directly. This **FP8 repo is the direct vLLM path** — serve it with `--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3` (command above), then set Claude Code's model name to the `--served-model-name` (no slashes) and `ANTHROPIC_BASE_URL` to your vLLM endpoint. For the GGUF + LM Studio route, see the **[GGUF repo](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF)**.