---
title: Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF
canonical_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF"
md_url: "https://www.modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF.md"
repository: Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF
last_updated: 2026-06-28
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
base_model:
  - Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
library_name:
  - gguf
downloads: 1008
stars: 4
tags:
  - gguf
  - llama.cpp
  - mtp
  - imatrix
  - reasoning
  - distillation
  - moe
---

# Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF

> Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF - Merkyor 在 ModelScope 开源的模型。Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — GGUF (原生 MTP + imatrix)

Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF 是 ModelScope 魔搭社区上的text-generation模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.6-35B-A3B 构建。

- **Repository**: Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF
- **License**: apache-2.0
- **Tasks**: text-generation
- **Base model**: Qwen/Qwen3.6-35B-A3B
- **Tags**: gguf, llama.cpp, mtp, imatrix, reasoning, distillation, moe
- **Downloads**: 1008
- **Stars**: 4
- **Last updated**: 2026-06-28

Source: https://www.modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF

---

# Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — GGUF (原生 MTP + imatrix)

## Lynn Agent 高配本地模型

这个 GGUF 仓是 **Lynn Agent** 的高配本地模型选项。Lynn v0.85.6+ 默认推荐 27B Q5_K_M GGUF；32GB+ VRAM / 统一内存机器可选择这里的 35B-A3B Q5_K_M imatrix MTP。

- **下载 Lynn Agent**：[GitHub Releases v0.85.6](https://github.com/LynnMerkyor/Lynn/releases/tag/v0.85.6)
- **Lynn 高配量化**：Q5_K_M imatrix + 原生 MTP (`--spec-draft-n-max 3`)
- **默认本地路径**：24GB+ 首推 27B Q5_K_M；9B / 4B 保留为低配置手动降级。

[nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill) 的 GGUF 量化版 —— Qwen3.6-35B-A3B(MoE,35B 总参 / 约 3B 激活)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了**原生 MTP 头**,并做了 **imatrix 校正**(Q8_0/F16 除外)。MoE 架构 + MTP 投机解码,单流加速尤其明显(见下)。

## 量化档位

| 文件 | 大小 | 说明 |
|---|---|---|
| `…-MTP-Q4_K_M-imatrix.gguf` | ~21 GB | **推荐平衡档** |
| `…-MTP-Q5_K_M-imatrix.gguf` | ~25 GB | 更高质量 |
| `…-MTP-Q6_K-imatrix.gguf` | ~29 GB | 接近无损 |
| `…-MTP-Q8_0.gguf` | ~37 GB | 几乎无损 |
| `…-MTP-F16.gguf` | ~69 GB | 全精度参考 |
| `…-mmproj-F16.gguf` | ~1 GB | 视觉投影(图像输入) |

## 评测 —— 蒸馏 vs 原版,Q4_K_M GGUF,同 harness,thinking-on

| 维度 | 蒸馏 | 原版 | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **80.3%** | 72.7% | **+7.6pp** |
| MMLU-500(5-shot)| 90.2% | 91.4% | −1.2 |
| 端到端编排 | **26.6s** | 60.7s | **快 2.3×** |
| 编排假验证(20×5)| **0/20** | 0/20 | |
| GPQA 未收口空答 | **1** | 12 | −11 |

GPQA +7.6pp、编排快 2.3×(学会收口,空答 12→1)。MMLU 微降 1.2:35B-A3B 容量比 27B dense 紧,蒸馏轻微挤占知识(已知权衡)。

## MTP —— 原生多 token 预测(MoE → 加速更猛)

GGUF 内含原生 MTP 头(mainline llama.cpp `--spec-type draft-mtp`)。MoE 比 dense 更吃投机解码红利。DGX Spark GB10 单流实测:

| 量化档 | 裸 t/s | MTP t/s | 加速 |
|---|---|---|---|
| Q4_K_M | 79.2 | 86.3 | 1.09× |
| Q5_K_M | 73.0 | 88.0 | 1.20× |
| Q6_K | 66.0 | 75.2 | 1.14× |
| Q8_0 | 49.8 | 81.2 | **1.63×** |

越高档加速越明显(带宽受限的裸推理被廉价草稿加速)。无损(贪心投机)。并发会摊薄投机 —— 高吞吐用裸多并发。

## 运行

```bash
llama-server -m Qwen3.6-35B-A3B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080
```

思考型模型 —— 必须 **temp 0.6 / top_p 0.95**。图像输入加 `--mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf`。

---

# English

GGUF quantizations of [nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill) — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-35B-A3B (MoE, 35B total / ~3B active). Every GGUF carries a **native MTP head** and is **imatrix-corrected** (except Q8_0/F16). The MoE architecture plus MTP speculative decoding gives an especially strong single-stream speedup (below).

## Quantizations

| File | Size | Notes |
|---|---|---|
| `…-MTP-Q4_K_M-imatrix.gguf` | ~21 GB | **recommended balance** |
| `…-MTP-Q5_K_M-imatrix.gguf` | ~25 GB | higher quality |
| `…-MTP-Q6_K-imatrix.gguf` | ~29 GB | near-lossless |
| `…-MTP-Q8_0.gguf` | ~37 GB | virtually lossless |
| `…-MTP-F16.gguf` | ~69 GB | full-precision reference |
| `…-mmproj-F16.gguf` | ~1 GB | vision projector (image input) |

## Evaluation — distill vs base, Q4_K_M GGUF, same harness, thinking-on

| Metric | Distill | Base | Δ |
|---|---|---|---|
| GPQA-Diamond-198 | **80.3%** | 72.7% | **+7.6pp** |
| MMLU-500 (5-shot) | 90.2% | 91.4% | −1.2 |
| End-to-end orchestration | **26.6s** | 60.7s | **2.3× faster** |
| Orchestration false-verify (20×5) | **0/20** | 0/20 | |
| GPQA non-terminating empties | **1** | 12 | −11 |

GPQA jumps +7.6pp and orchestration is 2.3× faster — the distill stops over-thinking and converges (non-terminating empties 12→1). MMLU dips 1.2pp: the 35B-A3B has less spare capacity than the 27B dense, so distillation slightly crowds out knowledge (a known trade-off).

## MTP — native multi-token-prediction (MoE → strong speedup)

The GGUF has a native MTP head (mainline llama.cpp `--spec-type draft-mtp`). MoE benefits more from speculative decoding than dense. Single-stream measured on DGX Spark GB10:

| Quant | base t/s | MTP t/s | speedup |
|---|---|---|---|
| Q4_K_M | 79.2 | 86.3 | 1.09× |
| Q5_K_M | 73.0 | 88.0 | 1.20× |
| Q6_K | 66.0 | 75.2 | 1.14× |
| Q8_0 | 49.8 | 81.2 | **1.63×** |

Higher quant tiers gain more (the bandwidth-bound base is accelerated by a cheap draft). Lossless (greedy speculative). Concurrency degrades speculation — use bare multi-concurrency for throughput.

## Run

```bash
llama-server -m Qwen3.6-35B-A3B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080
```

Thinking model — use **temp 0.6 / top_p 0.95**. For image input add `--mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf`.


## Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic `/v1/messages` 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。**GGUF 走 LM Studio**：LM Studio 0.4.1+ 加载本仓 GGUF（Q5_K_M 推荐），自带 Claude Code `/v1/messages` 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 **[FP8 仓](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8)** 或 BF16 主仓，加 `--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3`，模型名用 `--served-model-name`（不带斜杠）。参考 [LM Studio](https://lmstudio.ai/blog/claudecode) · [vLLM Claude Code](https://docs.vllm.ai/en/stable/serving/integrations/claude_code/)。

---

## Claude Code (experimental)

Claude Code needs an Anthropic-compatible `/v1/messages` backend + reliable tool calling — you cannot hand it a GGUF path. **GGUF goes via LM Studio**: LM Studio 0.4.1+ loads this repo's GGUF (Q5_K_M recommended) and exposes a built-in Claude Code `/v1/messages` endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the **[FP8 repo](https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8)** or the BF16 parent with `--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3`, model name = `--served-model-name` (no slashes).