---
title: Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF
canonical_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF"
md_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF.md"
repository: Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF
last_updated: 2026-07-12
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
base_model:
  - Qwen/Qwen3.6-27B
base_model_relation: quantized
library_name:
  - gguf
downloads: 2692
stars: 11
tags:
  - gguf
  - llama.cpp
  - mtp
  - imatrix
  - reasoning
  - distillation
---

# Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF

> Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF - Merkyor 在 ModelScope 开源的模型。Qwen3.6-27B-DSV4Pro-Thinking-Distill — GGUF (原生 MTP + imatrix)

Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF 是 ModelScope 魔搭社区上的text-generation模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.6-27B 构建。

- **Repository**: Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF
- **License**: apache-2.0
- **Tasks**: text-generation
- **Base model**: Qwen/Qwen3.6-27B
- **Tags**: gguf, llama.cpp, mtp, imatrix, reasoning, distillation
- **Downloads**: 2692
- **Stars**: 11
- **Last updated**: 2026-07-12

Source: https://www.modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF

---

# Qwen3.6-27B-DSV4Pro-Thinking-Distill — GGUF (原生 MTP + imatrix)

## Lynn Agent 默认本地模型

这个 GGUF 仓是 **Lynn Agent** 的默认本地模型推荐。Lynn v0.85.6+ 会把 `Qwen3.6-27B-DSV4Pro-Distill-MTP-Q5_K_M-imatrix.gguf` 作为 24GB+ VRAM / 统一内存机器的首选端侧模型；32GB+ 可选择 35B-A3B Q5_K_M，高低配机器可手动降级到 9B / 4B。

- **下载 Lynn Agent**：[GitHub Releases](https://github.com/LynnMerkyor/Lynn/releases)（当前桌面/CLI 发版：[v0.85.6](https://github.com/LynnMerkyor/Lynn/releases/tag/v0.85.6)）
- **Lynn 默认量化**：Q5_K_M imatrix + 原生 MTP (`--spec-draft-n-max 3`)
- **推荐用途**：端侧推理智能体、本地 coding / agentic 任务、无云依赖兜底。

[nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill](https://huggingface.co/nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill) 的 GGUF 量化版 —— Qwen3.6-27B(dense)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了**原生 MTP 头**,并做了 **imatrix 校正**(Q8_0/BF16 除外)。蒸馏只改*思维方式*(学会"收口"),coding 与知识不降。

## 量化档位

| 文件 | 大小 | 说明 |
|---|---|---|
| `…-MTP-IQ4_XS-imatrix.gguf` | 15.3 GB | **16G 显存底线** —— 16G 不 offload 可跑,最小档 |
| `…-MTP-Q4_K_S-imatrix.gguf` | 15.8 GB | 速度优先 |
| `…-MTP-Q4_K_M-imatrix.gguf` | 16.8 GB | 平衡档 / 低显存回退 |
| `…-MTP-Q5_K_M-imatrix.gguf` | 19.5 GB | **Lynn 默认推荐(24GB+)** |
| `…-MTP-Q6_K-imatrix.gguf` | 22.4 GB | 接近无损 |
| `…-MTP-Q8_0.gguf` | 29.0 GB | 几乎无损 |
| `…-MTP-BF16.gguf` | 54.7 GB | 全精度参考 |
| `…-mmproj-F16.gguf` | 0.9 GB | 视觉投影(图像输入) |

> **IQ4_XS 是底线档。** 为 **16 GB 显存、不 offload** 而做(14.3 GiB 权重 + KV 余量)。冒烟实测**收口干净**(3/3 推理题正确、0 长度跑飞、中英),单流 **31 t/s** + 原生 MTP 头(`--spec-type draft-mtp`,接受率 0.80、平均接受长 3.4)。这是我们发布的**最小档** —— 27B 推理蒸馏低于 IQ4_XS 会开始丢收口,到此为止。

> **📏 GB vs GiB —— 为什么 15.3 GB 的文件能进"16 GB"显卡。** 本表大小用十进制 **GB(10⁹ 字节)**；显卡显存按 **GiB(2³⁰ 字节)** 造却标"GB"，所以一张"16 GB"卡其实是 **16 GiB ≈ 17.2 GB**。于是 IQ4_XS(15.3 GB = **14.3 GiB**)能完整载入 16 GiB 卡，**剩 ~1.7 GiB 给 KV —— 不用 offload**。经验法则：文件的 **GiB** 数 < 显卡标称"GB"数，就能不 offload 跑（这里 14.3 < 16 ✓）。

## 评测 —— 蒸馏 vs 原版,同规格 GGUF,thinking-on

两行同量化 + 同 MTP,唯一变量是"是否蒸馏"。禁止跨量化档比较(Q4 与 Q5 用了不同 harness)。

**Q4_K_M(旧 harness):**

| 维度 | 蒸馏 | 原版 | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **80.81%**(160/198)| 73.7% | **+7.1** |
| MMLU-500(5-shot)| **91.8%** | 91.6% | +0.2 |
| GPQA 未收口空答 | **0** | 14 | −14 |
| coding-100(真实沙箱)| **86/100** | 83/100 | +3 |

**Q5_K_M(流式 harness · 并发4 · timeout 1800 · max_tokens 32000,逐题记 `finish_reason`):**

| 维度 | 蒸馏 | 原版 | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **81.82%**(162/198)| 68.69%(136/198)| **+13.13pp** |
| MMLU-500 | **90.0%** | 89.6% | +0.4 |
| GPQA `finish=stop`(收口)| **198/198** | 186/198 | |
| GPQA `finish=length`(撞 32K 墙)| **0** | 12 | |

硬推理大涨(GPQA +7~13pp),知识不降,且蒸馏**学会收口** —— 原版有 12 题撞 32K 墙(始终不出答案),蒸馏 198 题全收口。

## MTP —— 原生多 token 预测(单流加速)

GGUF 内含原生 MTP 头(mainline llama.cpp `--spec-type draft-mtp`,无需外挂草稿模型)。DGX Spark GB10 单流实测,Q4_K_M:

| `--spec-draft-n-max` | TPS | 接受率 | 接受长度 |
|---|---|---|---|
| 裸(无 MTP)| 10.4 | — | — |
| **n-max=3 ⭐** | **26.8** | 0.72 | 3.16 |

单流 2.3–2.6× 加速,无损(贪心投机只接受目标 argmax token)。投机是单流手段 —— 并发会摊薄它,高吞吐场景用裸多并发。

## 运行

```bash
llama-server -m Qwen3.6-27B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080
```

思考型模型 —— 必须 **temp 0.6 / top_p 0.95**(贪心会死循环)。图像输入加 `--mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf`。

---

# English

GGUF quantizations of [nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill](https://huggingface.co/nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill) — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-27B (dense). Every GGUF carries a **native MTP (multi-token-prediction) head** and is **imatrix-corrected** (except Q8_0/BF16). Distillation only changes the *thinking style* (the model learns to converge), not coding or knowledge.

## Quantizations

| File | Size | Notes |
|---|---|---|
| `…-MTP-IQ4_XS-imatrix.gguf` | 15.3 GB | **16 GB-VRAM floor** — fits 16 GB no-offload; smallest we ship |
| `…-MTP-Q4_K_S-imatrix.gguf` | 15.8 GB | speed-first |
| `…-MTP-Q4_K_M-imatrix.gguf` | 16.8 GB | **recommended balance** |
| `…-MTP-Q5_K_M-imatrix.gguf` | 19.5 GB | higher quality |
| `…-MTP-Q6_K-imatrix.gguf` | 22.4 GB | near-lossless |
| `…-MTP-Q8_0.gguf` | 29.0 GB | virtually lossless |
| `…-MTP-BF16.gguf` | 54.7 GB | full-precision reference |
| `…-mmproj-F16.gguf` | 0.9 GB | vision projector (image input) |

> **IQ4_XS is the floor.** Sized for a **16 GB-VRAM card with zero CPU offload** (14.3 GiB weights + headroom for KV). Smoke-verified **converging** (3/3 reasoning prompts correct, 0 length-runaway, EN + ZH) at **31 t/s single-stream** with the native MTP head (`--spec-type draft-mtp`, accept 0.80, mean accept-len 3.4). This is the smallest tier we ship — below IQ4_XS a 27B reasoning distill starts to lose convergence, so we stop here.

> **📏 GB vs GiB — why a 15.3 GB file fits a "16 GB" GPU.** Sizes in this table are decimal **GB (10⁹ bytes)**. GPU VRAM is built in **GiB (2³⁰ bytes)** but marketed as "GB" — so a "16 GB" card is really **16 GiB ≈ 17.2 GB**. IQ4_XS (15.3 GB = **14.3 GiB**) therefore loads fully on a 16 GiB card with **~1.7 GiB left for KV cache — no CPU offload**. Rule of thumb: a file runs no-offload when its **GiB** size < the card's nominal "GB" number (here 14.3 < 16 ✓).

## Evaluation — distill vs base, same-spec GGUF, thinking-on

Both rows use the same quant + MTP; the only variable is "distilled or not". Do not cross-compare quant tiers (Q4 and Q5 use different harnesses).

**Q4_K_M (archived harness):**

| Dimension | Distill | Base | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **80.81%** (160/198) | 73.7% | **+7.1** |
| MMLU-500 (5-shot) | **91.8%** | 91.6% | +0.2 |
| GPQA unconverged empties | **0** | 14 | −14 |
| coding-100 (real sandbox) | **86/100** | 83/100 | +3 |

**Q5_K_M (streaming harness · conc4 · timeout 1800 · max_tokens 32000, `finish_reason` logged):**

| Dimension | Distill | Base | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **81.82%** (162/198) | 68.69% (136/198) | **+13.13pp** |
| MMLU-500 | **90.0%** | 89.6% | +0.4 |
| GPQA `finish=stop` (converged) | **198/198** | 186/198 | |
| GPQA `finish=length` (hit 32K wall) | **0** | 12 | |

Hard reasoning jumps (GPQA +7~13pp), knowledge holds, and the distill **learns to converge** — the base hits the 32K wall on 12 questions (never answers); the distill converges on all 198.

## MTP — native multi-token-prediction (single-stream speedup)

The GGUF has a native MTP head (mainline llama.cpp `--spec-type draft-mtp`, no external draft model). Measured single-stream on DGX Spark GB10, Q4_K_M:

| `--spec-draft-n-max` | TPS | accept rate | accept len |
|---|---|---|---|
| bare (no MTP) | 10.4 | — | — |
| **n-max=3 ⭐** | **26.8** | 0.72 | 3.16 |

2.3–2.6× single-stream speedup, lossless (greedy speculative only accepts the target-argmax token). Speculation is single-stream — concurrency degrades it, so use bare multi-concurrency for throughput.

## Run

```bash
llama-server -m Qwen3.6-27B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080
```

Thinking model — use **temp 0.6 / top_p 0.95** (greedy loops to death). For image input add `--mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf`.


<!-- CLAUDE-CODE-COMPAT-TEMPLATE-ZH-START -->
### Claude Code / llama-swap 兼容模板
如果对话历史中间出现新的 system 或 developer 消息，并报错 `System message must be at the beginning`，请使用 [qwen35-claude-compatible.jinja](./compat/qwen35-claude-compatible.jinja)。该模板会按原顺序把所有 system/developer 内容合并到开头的 system 块，同时保留 Qwen3.5 原有的 thinking、工具调用与 tool response 逻辑。

启动示例：
```bash
llama-server -m model.gguf --jinja \
  --chat-template-file qwen35-claude-compatible.jinja
```
<!-- CLAUDE-CODE-COMPAT-TEMPLATE-ZH-END -->
## Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic `/v1/messages` 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。**GGUF 走 LM Studio**：LM Studio 0.4.1+ 加载本仓 GGUF（Q5_K_M 推荐），自带 Claude Code `/v1/messages` 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 **[FP8 仓](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8)** 或 BF16 主仓，加 `--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3`，模型名用 `--served-model-name`（不带斜杠）。参考 [LM Studio](https://lmstudio.ai/blog/claudecode) · [vLLM Claude Code](https://docs.vllm.ai/en/stable/serving/integrations/claude_code/)。

---

## Claude Code (experimental)

Claude Code needs an Anthropic-compatible `/v1/messages` backend + reliable tool calling — you cannot hand it a GGUF path. **GGUF goes via LM Studio**: LM Studio 0.4.1+ loads this repo's GGUF (Q5_K_M recommended) and exposes a built-in Claude Code `/v1/messages` endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the **[FP8 repo](https://modelscope.cn/models/Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8)** or the BF16 parent with `--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3`, model name = `--served-model-name` (no slashes).