---
title: Qwen3.8-9B-Distill-GGUF
canonical_url: "https://www.modelscope.cn/models/empero-ai/Qwen3.8-9B-Distill-GGUF"
md_url: "https://www.modelscope.cn/models/empero-ai/Qwen3.8-9B-Distill-GGUF.md"
repository: empero-ai/Qwen3.8-9B-Distill-GGUF
last_updated: 2026-08-19
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
base_model:
  - empero-ai/Qwen3.8-9B
base_model_relation: quantized
library_name:
  - gguf
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 3322
stars: 19
tags:
  - gguf
  - llama.cpp
  - quantized
  - empero-ai
  - qwen3.5
  - qwen3.8
  - distillation
  - reasoning
  - gated-deltanet
---

# Qwen3.8-9B-Distill-GGUF

> Qwen3.8-9B-Distill-GGUF - empero-ai 在 ModelScope 开源的模型。GGUF quantizations of empero-ai/Qwen3.8-9B — a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-9B architecture — for llama.cpp, Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF…

empero-ai/Qwen3.8-9B-Distill-GGUF 是 ModelScope 魔搭社区上的text-generation模型，采用 apache-2.0 许可，基于 empero-ai/Qwen3.8-9B 构建。

- **Repository**: empero-ai/Qwen3.8-9B-Distill-GGUF
- **License**: apache-2.0
- **Tasks**: text-generation
- **Base model**: empero-ai/Qwen3.8-9B
- **Tags**: gguf, llama.cpp, quantized, empero-ai, qwen3.5, qwen3.8, distillation, reasoning, gated-deltanet
- **Downloads**: 3322
- **Stars**: 19
- **Last updated**: 2026-08-19

Source: https://www.modelscope.cn/models/empero-ai/Qwen3.8-9B-Distill-GGUF

---

# Qwen3.8-9B — GGUF

**Developed by [Empero](https://empero.org)**

GGUF quantizations of **[empero-ai/Qwen3.8-9B](https://huggingface.co/empero-ai/Qwen3.8-9B)** — a full-parameter distillation of **Qwen3.8 2.4T A95B** into the Qwen3.5-9B architecture — for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF runtimes.

This card is about choosing a file and running it. The capability writeup, full benchmark results, and best practices live on the **[main model card](https://huggingface.co/empero-ai/Qwen3.8-9B)**.

Headline results for the source model (CoT protocols, `lm-evaluation-harness`, identical settings base vs. student):

| Task | Qwen3.5-9B (base) | **Qwen3.8-9B** | Δ |
|---|---:|---:|---:|
| mmlu (CoT, 57 subjects) | 0.546 | **0.751** | **+0.205** |
| gsm8k_cot | 0.885 | 0.870 | −0.015 |

> [!Note]
> Qwen3.5-class models are hybrids: three Gated DeltaNet layers for every full-attention layer. A **recent llama.cpp build with Qwen3.5 / Gated DeltaNet support** is required — older builds will fail to load the architecture.

## Files

| File | Quant | Size | Notes |
|---|---|---:|---|
| `Qwen3.8-9B-Q4_K_M.gguf` | Q4_K_M | 5.780 GB | **Recommended.** Best quality/size balance for most users. |
| `Qwen3.8-9B-Q5_K_M.gguf` | Q5_K_M | 6.643 GB | Higher quality, still fits an 8 GB card at short context. |
| `Qwen3.8-9B-Q6_K.gguf` | Q6_K | 7.559 GB | Near-lossless. |
| `Qwen3.8-9B-Q8_0.gguf` | Q8_0 | 9.786 GB | Highest-quality quantization. |
| `Qwen3.8-9B-BF16.gguf` | BF16 | 18.407 GB | Full precision reference. |

Sizes are exact decimal GB from the uploaded files (1 GB = 1,000,000,000 bytes).

### What fits on a GPU?

Practical weight-size-based guidance at modest context — the KV cache is the dominant cost at long context and may require offload regardless of weight quant:

| Quant | Guidance |
|---|---|
| Q4_K_M / Q5_K_M | Comfortable on 8–12 GB cards for everyday use. |
| Q6_K / Q8_0 | 12–16 GB recommended. |
| BF16 | 24 GB+. |

## Usage

### llama.cpp

```bash
llama-cli -m Qwen3.8-9B-Q4_K_M.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv
```

Use the built-in chat template (`-cnv`). The model is a reasoning model: every answer opens with a `<think>` block, so allow a generous `-n` and strip the `<think>...</think>` span for end users.

### Ollama / LM Studio / Jan / KoboldCpp

Download the GGUF of your choice and load it directly; the chat template is embedded in the file. Recommended sampling: `temperature=0.6, top_p=0.95, top_k=20`.

## Provenance & licensing

Quantizations of **[empero-ai/Qwen3.8-9B](https://huggingface.co/empero-ai/Qwen3.8-9B)**, a distillation of Qwen3.8 2.4T A95B into [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) trained on ~70,000 curated teacher traces from our internal Qwen3.8 distillation datasets. Weights are **Apache-2.0**, inherited from the Qwen base, shared as-is.

## Stay in the loop

Sign up for the Empero newsletter at **[empero.org](https://empero.org)** for releases, evals, and research notes.

## Support / Donate

If this model helped you, consider supporting the project:

- **BTC**: `bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v`
- **LTC**: `ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x`

## Acknowledgements

- Developed and released by [Empero](https://empero.org)
- Base model: [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Alibaba Qwen team)
- GGUF quantization: [llama.cpp](https://github.com/ggml-org/llama.cpp) (ggml-org)