--- title: Qwen3.8-35B-A3B-Distill-GGUF canonical_url: "https://www.modelscope.cn/models/empero-ai/Qwen3.8-35B-A3B-Distill-GGUF" md_url: "https://www.modelscope.cn/models/empero-ai/Qwen3.8-35B-A3B-Distill-GGUF.md" repository: empero-ai/Qwen3.8-35B-A3B-Distill-GGUF last_updated: 2026-09-17 license: apache-2.0 pipeline_tag: text-generation tasks: - text-generation base_model: - empero-ai/Qwen3.8-35B-A3B-Distill base_model_relation: quantized library_name: - gguf - pytorch frameworks: - pytorch language: - en downloads: 405 stars: 3 tags: - gguf - llama.cpp - quantized - empero-ai - qwen3.6 - qwen3.8 - distillation - reasoning - moe - gated-deltanet --- # Qwen3.8-35B-A3B-Distill-GGUF > Qwen3.8-35B-A3B-Distill-GGUF - empero-ai 在 ModelScope 开源的模型。Qwen3.8-35B-A3B — GGUF empero-ai/Qwen3.8-35B-A3B-Distill-GGUF 是 ModelScope 魔搭社区上的text-generation模型，采用 apache-2.0 许可，基于 empero-ai/Qwen3.8-35B-A3B-Distill 构建。 - **Repository**: empero-ai/Qwen3.8-35B-A3B-Distill-GGUF - **License**: apache-2.0 - **Tasks**: text-generation - **Base model**: empero-ai/Qwen3.8-35B-A3B-Distill - **Tags**: gguf, llama.cpp, quantized, empero-ai, qwen3.6, qwen3.8, distillation, reasoning, moe, gated-deltanet - **Downloads**: 405 - **Stars**: 3 - **Last updated**: 2026-09-17 Source: https://www.modelscope.cn/models/empero-ai/Qwen3.8-35B-A3B-Distill-GGUF --- # Qwen3.8-35B-A3B — GGUF **Developed by [Empero](https://empero.org)** GGUF quantizations of **[empero-ai/Qwen3.8-35B-A3B-Distill](https://huggingface.co/empero-ai/Qwen3.8-35B-A3B-Distill)** — a distillation of the Qwen3.8 frontier models into the Qwen3.6-35B-A3B Mixture-of-Experts architecture — for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF runtimes. This card is about choosing a file and running it. The capability writeup, benchmark results, and best practices live on the **[main model card](https://huggingface.co/empero-ai/Qwen3.8-35B-A3B-Distill)**. 35B total parameters with ~3B active per token — the MoE sparsity means it runs considerably faster than a dense 35B at the same quant, but the **whole weight file still has to fit in RAM or VRAM**. > [!Note] > Qwen3.6-class models are hybrids: 30 Gated DeltaNet layers and 10 full-attention layers, with 256 experts routed 8-per-token. A **recent llama.cpp build with Qwen3.6 / Gated DeltaNet MoE support** is required — older builds will fail to load the architecture. ## Files | File | Quant | Size | Notes | |---|---|---:|---| | `Qwen3.8-35B-A3B-IQ2_M.gguf` | IQ2_M | 12.558 GB | Smallest usable. Fits a 16 GB card. | | `Qwen3.8-35B-A3B-Q2_K.gguf` | Q2_K | 13.839 GB | 2-bit K-quant; widest runtime support at this size. | | `Qwen3.8-35B-A3B-IQ3_M.gguf` | IQ3_M | 16.340 GB | Strong quality per byte at 3-bit. | | `Qwen3.8-35B-A3B-Q3_K_M.gguf` | Q3_K_M | 17.664 GB | Conventional 3-bit K-quant. | | `Qwen3.8-35B-A3B-IQ4_XS.gguf` | IQ4_XS | 19.628 GB | Near Q4_K_M quality, ~2 GB smaller. | | `Qwen3.8-35B-A3B-Q4_K_M.gguf` | Q4_K_M | 21.713 GB | **Recommended.** Best quality/size balance for most users. | | `Qwen3.8-35B-A3B-Q5_K_M.gguf` | Q5_K_M | 25.348 GB | Higher quality, modest size increase. | | `Qwen3.8-35B-A3B-Q6_K.gguf` | Q6_K | 29.209 GB | Near-lossless. | | `Qwen3.8-35B-A3B-Q8_0.gguf` | Q8_0 | 37.802 GB | Highest-quality quantization. | | `Qwen3.8-35B-A3B-BF16.gguf` | BF16 | 71.067 GB | Full precision reference. | | `mmproj-Qwen3.8-35B-A3B-F16.gguf` | F16 | 0.899 GB | Vision projector. Pair with any text quant above for image input. | The IQ\* quants and the 2/3-bit K-quants are calibrated with an importance matrix, which is what keeps them coherent at these bit-widths. Sizes are exact decimal GB from the uploaded files (1 GB = 1,000,000,000 bytes). ### What fits? Weight-size guidance at modest context. The KV cache is the dominant cost at long context and may require offload regardless of weight quant: | Quant | Guidance | |---|---| | IQ2_M / Q2_K | 16 GB VRAM, or 16 GB system RAM. The smallest that stay coherent. | | IQ3_M / Q3_K_M | 20-24 GB VRAM, or 24 GB system RAM. | | IQ4_XS / Q4_K_M | 24 GB VRAM for a full GPU load; comfortable on CPU with 32 GB RAM. | | Q5_K_M / Q6_K | 32 GB VRAM, or 48 GB system RAM. | | Q8_0 | 48 GB VRAM, or 64 GB system RAM. | | BF16 | 80 GB+ VRAM, or 96 GB system RAM. Reference only. | Because only ~3B parameters are active per token, CPU and partial-offload inference are far more practical here than for a dense model of the same file size. ## Usage ### llama.cpp ```bash llama-cli -m Qwen3.8-35B-A3B-Q4_K_M.gguf \ --temp 0.6 --top-p 0.95 --top-k 20 \ -n 16384 -cnv ``` Use the built-in chat template (`-cnv`). The model is a reasoning model: every answer opens with a `` block, so allow a generous `-n` and strip the `...` span for end users. ### Vision Pair the projector with any text quant: ```bash llama-mtmd-cli -m Qwen3.8-35B-A3B-Q4_K_M.gguf \ --mmproj mmproj-Qwen3.8-35B-A3B-F16.gguf \ --image photo.jpg -p "Describe this image." ``` The vision tower is inherited unchanged from the Qwen3.6-35B-A3B base — the distillation was text-only and vision behaviour was not evaluated. ### Ollama / LM Studio / Jan / KoboldCpp Download the GGUF of your choice and load it directly; the chat template is embedded in the file. Recommended sampling: `temperature=0.6, top_p=0.95, top_k=20`. ## Verifying downloads `SHA256SUMS` in this repo lists checksums for every GGUF: ```bash sha256sum -c SHA256SUMS --ignore-missing ``` ## Provenance & licensing Quantizations of **[empero-ai/Qwen3.8-35B-A3B-Distill](https://huggingface.co/empero-ai/Qwen3.8-35B-A3B-Distill)**, a distillation of the Qwen3.8 frontier models into [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). Weights are **Apache-2.0**, inherited from the Qwen base, shared as-is. ## Stay in the loop Sign up for the Empero newsletter at **[empero.org](https://empero.org)** for releases, evals, and research notes. ## Support / Donate If this model helped you, consider supporting the project: - **BTC**: `bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v` - **LTC**: `ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x` ## Acknowledgements - Developed and released by [Empero](https://empero.org) - Base model: [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Alibaba Qwen team) - GGUF quantization: [llama.cpp](https://github.com/ggml-org/llama.cpp) (ggml-org)