---
title: DeepSeek-V3.1-mlx-DQ5_K_M
canonical_url: "https://www.modelscope.cn/models/mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M"
md_url: "https://www.modelscope.cn/models/mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M.md"
repository: mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M
last_updated: 2025-09-10
license: mit
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - deepseek_v3
architectures:
  - DeepseekV3ForCausalLM
base_model:
  - deepseek-ai/DeepSeek-V3.1
base_model_relation: quantized
parameters: 128.8B
tensor_type:
  - U32
  - BF16
  - F32
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
inference_backends:
  - "deploy_task text/emb"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 93
stars: 0
tags:
  - mlx
---

# DeepSeek-V3.1-mlx-DQ5_K_M

> DeepSeek-V3.1-mlx-DQ5_K_M - mlx-community 在 ModelScope 开源的模型。mlx-community/DeepSeek-V3.1-mlx-DQ5KM

mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M 是 ModelScope 魔搭社区上的 128.8B 参数text-generation模型，采用 mit 许可，基于 deepseek-ai/DeepSeek-V3.1 构建，可用 deploy_task text/emb、sglang 0.5.2、vllm 0.9.2 部署。

- **Repository**: mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M
- **License**: mit
- **Tasks**: text-generation
- **Parameters**: 128.8B
- **Base model**: deepseek-ai/DeepSeek-V3.1
- **Inference backends**: deploy_task text/emb, sglang 0.5.2, vllm 0.9.2
- **Tags**: mlx
- **Downloads**: 93
- **Stars**: 0
- **Last updated**: 2025-09-10

Source: https://www.modelscope.cn/models/mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M

---

# mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M

This model [mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M](https://huggingface.co/mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M) was
converted to MLX format from [deepseek-ai/DeepSeek-V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)
using mlx-lm version **0.26.3**.

This is created for people using a single Apple Mac Studio M3 Ultra with 512 GB. With 512 GB, we can do better than the 4-bit version of DeepSeek V3.1. Using research results, we aim to get better than 5-bit performance using smarter quantization. We aim to not have the quant so large that it leaves no memory for a useful context window.

```bash
pip install mlx-lm

mlx_lm.generate --model mlx-community/DeepSeek-V3.1-mlx-DQ5_K_M --temp 1.3 --min-p 0.01 --max-tokens 4096 --prompt "Hallo"
```

The temperature of 1.3 is DeepSeek's recommendation for translations. For coding, you should probably use a temperature of 0.6 or lower.

---

## What is this DQ5_K_M?

In the Arxiv paper [Quantitative Analysis of Performance Drop in DeepSeek Model Quantization](https://arxiv.org/abs/2505.02390) the authors write,

> We further propose `DQ3_K_M`, a dynamic 3-bit quantization method that significantly outperforms traditional `Q3_K_M` variant on various benchmarks, which is also comparable with 4-bit quantization (`Q4_K_M`) approach in most tasks.

and

> dynamic 3-bit quantization method (`DQ3_K_M`) that outperforms the 3-bit quantization implementation in `llama.cpp` and achieves performance comparable to 4-bit quantization across multiple benchmarks.

The resulting multi-bitwidth quantization has been well tested and documented.

In this case we did not want a improved 3-bit quant, but rather the best possible "5-bit" quant. We therefore modified the `DQ3_K_M` quantization by replacing 3-bit by 5-bit, 4-bit by 6-bit, and 6-bit by 8-bit to create a new `DQ5_K_M` quant. This produces a quantization of 5.638 bpw (bits per weight).

---

## How can you create your own DQ5_K_M quants?

In the `convert.py` file of mlx-lm on your system ( [you can see the original code here](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/convert.py) ), replace the code inside `def mixed_quant_predicate()` with something like

```python
        index = (
            int(path.split(".")[layer_location])
            if len(path.split(".")) > layer_location
            else 0
        )
        # Build a mixed quant like "DQ5" similar to the "DQ3" of Arxiv paper https://arxiv.org/abs/2505.02390
        #    Quantitative Analysis of Performance Drop in DeepSeek Model Quantization
        q_bits = 6       
        if "lm_head" in path:
           q_bits = 8
        #if "tokens" in path:
        #   q_bits = 6
        if "attn.kv" in path:
           q_bits = 8
        #if "o_proj" in path:
        #   q_bits = 6
        #if "attn.q" in path:
        #   q_bits = 6
        # For all "mlp" and "shared experts"
        if "down_proj" in path:
           q_bits = 8
        #if "up_proj" in path:
        #   q_bits = 6
        #if "gate_proj" in path:
        #   q_bits = 6
        # For "switch experts"
        if "switch_mlp.up_proj" in path:
           q_bits = 5
        if "switch_mlp.gate_proj" in path:
           q_bits = 5
        if "switch_mlp.down_proj" in path:
           q_bits = 5
           # Blocks up to 5 are higher quality
           if index < 5:
              q_bits = 8
           # Every 5th block is "medium" quality
           if (index % 5) == 0:
              q_bits = 6
        #print("path:", path, "index:", index, "q_bits:", q_bits)
        return {"group_size": group_size, "bits": q_bits}
```

Should you wish to squeeze more out of your quant, and you do not need to use a larger context window, you can change the last part of the above code to

```python
        if "switch_mlp.down_proj" in path:
           q_bits = 6
           # Blocks up to 5 are higher quality
           if index < 5:
              q_bits = 8
        #print("path:", path, "index:", index, "q_bits:", q_bits)
        return {"group_size": group_size, "bits": q_bits}
```

Then create your `DQ5_K_M` quant with

```bash
mlx_lm.convert --hf-path deepseek-ai/DeepSeek-V3.1 --mlx-path your-model-DQ5_K_M -q --quant-predicate mixed_3_4
```

---

Enjoy!