---
title: Qwen2.5-32B-Instruct-Int4-W4A16
canonical_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16"
md_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16.md"
repository: okwinds/Qwen2.5-32B-Instruct-Int4-W4A16
chinese_name: "通义千问 Qwen2.5-32B-Instruct-Int4-W4A16 量化高精校准"
last_updated: 2025-04-17
license: apache-2.0
pipeline_tag: text-generation
tasks:
- text-generation
model_type:
- qwen2
architectures:
- Qwen2ForCausalLM
parameters: 5.7B
tensor_type:
- I32
- BF16
- I64
library_name:
- safetensors
- pytorch
frameworks:
- pytorch
language:
- en
- zh
inference_backends:
- "deploy_task text/emb"
- "lmdeploy 0.9.1"
- "lmdeploy_turbomind 0.9.1"
- "sglang 0.5.2"
- "vllm 0.9.2"
downloads: 1272
stars: 14
tags:
- Instruct
- Chat
---
# Qwen2.5-32B-Instruct-Int4-W4A16
> Qwen2.5-32B-Instruct-Int4-W4A16 - okwinds 在 ModelScope 开源的模型。正式版推理模型 QwQ-32B 的量化版本已出，用了更优质的数据做校准。欢迎品尝：QwQ-32B-Int4-W4A16
okwinds/Qwen2.5-32B-Instruct-Int4-W4A16 是 ModelScope 魔搭社区上的 5.7B 参数text-generation模型，采用 apache-2.0 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。
- **Repository**: okwinds/Qwen2.5-32B-Instruct-Int4-W4A16
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 5.7B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: Instruct, Chat
- **Downloads**: 1272
- **Stars**: 14
- **Last updated**: 2025-04-17
Source: https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16
---
正式版推理模型 QwQ-32B 的量化版本已出，用了更优质的数据做校准。欢迎品尝：[QwQ-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Int4-W4A16)
# 通义千问 Qwen2.5-32B-Instruct-INT4-W4A16 量化高精校准
原模型 [qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct)
友情提醒，推理本模型时，请按照如下官方指引：
> 注意：建议在运行这些模型时设置适当的温度（0.5 到 0.7 之间），否则您可能会遇到无休止的重复或不连贯的输出问题。
📖 关于量化损失方面的研究，可阅读公众号“觉察流”文章👇
### 《[Reason Model 的“瘦身计划”：量化技术的得与失](https://mp.weixin.qq.com/s/NMGq4UUkfo8GMix5LHnWCg)》
```python
# download model
from modelscope import snapshot_download
model_dir = snapshot_download('okwinds/Qwen2.5-32B-Instruct-Int4-W4A16')
```
## Qwen2.5 同系列模型
#### Finetuned Model
- **Model tree**
Base model - [Qwen/Qwen2.5-32B](https://www.modelscope.cn/models/Qwen/Qwen2.5-32B)
╰─ Finetuned - [Qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen2.5-32B-Instruct)
├─ Finetuned - [CompassJudger-1](https://www.modelscope.cn/collections/CompassJudger-1-Int-and-FP8---Mixed-Precision-85d756858db345)
└─ Finetuned - [Rombos-LLM-Qwen2.5](https://www.modelscope.cn/collections/Rombos-LLM-Qwen25-4dc8690ad7f541)
#### 同系量化
| **Model** | GPTQ Compression & FP8 | GGUF | **MLX** (macOS) |
| ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *3B params* | - | [3B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-GGUF-V3-LOT) | [3B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-4bit)
[3B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-8bit) |
| *7B params* | [7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int4-W4A16)
[7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int8-W8A16)
[7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-FP8-A16)
[Coder-7B-Instruct-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8) | [7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-GGUF-V3-LOT) | [7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-4bit)
[7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-8bit) |
| *Coder 7B params* | [Coder-7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int4-W4A16)
[Coder-7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int8-W8A16)
[Coder-7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8-A16) | [Coder-7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-GGUF-V3-LOT) | [Coder-7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-4bit)
[Coder-7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-8bit) |
| *14B params* | [14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16)
[14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int8-W8A16)
[14B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-FP8-A16) | [14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-GGUF-V3-LOT) | [14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-4bit)
[14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-8bit) |
| *Coder 14B params* | [Coder-14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int4-W4A16)
[Coder-14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int8-W8A16) | [Coder-14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-GGUF-V3-LOT) | [Coder-14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-4bit)
[Coder-14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-8bit) |
| *32B params* | [32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16)
[32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16)
[32B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16) | [32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-GGUF-V3-LOT) | [32B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-2bit)
[32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-4bit)
[32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-8bit) |
| *Coder 32B params* | [Coder-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-Int4-W4A16)
[Coder-32B-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-FP8) | [Coder-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-GGUF-V3-LOT) | [Coder-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-4bit)
[Coder-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-8bit) |
| *QwQ-Preview 32B params* | [QwQ-32B-Preview-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-Int4-W4A16)
[QwQ-32B-Preview-FP8](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-FP8) | [QwQ-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-GGUF-V3-LOT) | [QwQ-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-4bit)
[QwQ-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-8bit) |
| *72B params* | - | [72B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT) | [72B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-2bit)
[72B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-4bit)
[72B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-8bit) |
## 模型概述
> 本模型已升级，请使用精度更高的最新版本，原版本归档至 tag v1.0.0。
更多精度详情，请关注 Benchmark 部分数据。
Qwen2.5-32B-Instruct-Int4-W4A16 是一个基于 通义千问 Qwen2.5-32B-Instruct 的 INT4 量化并校准的模型。
- **模型名称:** Qwen2.5-32B-Instruct-Int4-W4A16
- **模型架构:** Qwen2.5
- **权重量化:** INT4
该模型通过将 [qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct) 的权重量化为 INT4 数据类型而实现。
量化过程将每个参数从 16bit 减少到 4bit，将模型占用磁盘空间大小，以及推理时加载模型需要的GPU显存空间，减少到了大约为原模型的1/4。只有 transformer 中的 Linear 层权重是量化的，其他层均保持为原模型数据类型，采用了混合精度的计算方式，尽可能减少模型量化后所带来的精度损失。
量化本模型，使用了[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ)，采用 Symmetric group-wise 方式量化（group size : 128）。并在量化过程中，进行了数据校准，以提升模型生成精度，修复 gptq-int4 模型出现乱吐字和无限循环的问题（*如循环输出 !!!!!*）。
> 注意：本模型需要 compute capability > 8.0（Ampere、Ada Lovelace、Hopper 架构）的 Nvidia GPU 来支持 INT4 **混合精度**计算。
## 部署推理
#### 推荐使用 vLLM
Openai api 兼容模式
```bash
>>> vllm serve "/home/gavin/llm/Qwen2.5-32B-Instruct-Int4-W4A16" --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.9 --served-model-name "Qwen2.5-32B-Instruct-Int4-W4A16"
```
## Benchmark
_本模型采用 INT4(W4A16) 量化方法，并通过构建高精校准数据集，模型在精度方面有非常好的表现。相较 BF16 原模型，在4位Int型量化的基础上，模型精度得到了最大限度的保有。_
本模型的最新版本对精度进行了更广泛的校准。
| Benchmark | [Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct)(原模型) | Int4-W4A16-v1.0.0 | Recovery-v1.0.0 | _Int4-W4A16-master(New)_ | Recovery-master(New) |
| ---------------------------- | --------------------------------------------------------------------------------------- | ----------------- | --------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Overall** | 73.21 | 72.91 | 99.59% | 72.97 | 99.67% |
| Exam | 86.33 | 85.37 | 98.89% | 86.24 | 99.90% |
| Language | 54.08 | 55.70 | 103.00% | 54.47 | 100.72% |
| Knowledge | 40.63 | 40.02 | 98.50% | 40.04 | 98.55% |
| Understanding | 92.89 | 92.65 | 99.74% | 92.46 | 99.54% |
| Coding | 85.54 | 84.35 | 98.61% | 85.15 | 99.54% |
| Reasoning | 69.56 | 69.12 | 99.37% | 69.11 | 99.35% |
| Instruct Follow | 83.45 | 83.15 | 99.64% | 83.35 | 99.88% |
| **Exam** | - | - | - | - | - |
| C-Eval | 85.87 | 84.78 | 98.73% | 86.81 | 101.09% |
| MMLU | 83.47 | 82.53 | 98.87% | 82.78 | 99.17% |
| CMMLU | 85.57 | 85.12 | 99.47% | 85.25 | 99.63% |
| GaokaoBench | 90.43 | 89.03 | 98.45% | 90.09 | 99.62% |
| **Language** | - | - | - | - | - |
| WSC | 89.42 | 92.31 | 103.23% | 89.42 | 100.00% |
| Tydiqa-goldp | 18.74 | 20.05 | 106.99% | 19.52 | 104.16% |
| **Knowledge** | - | - | - | - | - |
| TriviaQA | 60.68 | 61.33 | 101.07% | 60.13 | 99.09% |
| Natual Questions | 20.58 | 20.53 | 99.76% | 19.94 | 96.89% |
| **Understanding** | - | - | - | - | - |
| Race-middle | 94.29 | 94.22 | 99.93% | 93.87 | 99.55% |
| Race-high | 91.48 | 91.08 | 99.56% | 91.05 | 99.53% |
| **Coding** | - | - | - | - | - |
| Human eval | 87.80 | 86.59 | 98.62% | 87.80 | 100.00% |
| Sanitized MBPP | 83.27 | 82.10 | 98.59% | 82.49 | 99.06% |
| **Reasoning** | - | - | - | - | - |
| MATH | 77.66 | 77.36 | 99.61% | 77.50 | 99.79% |
| GSM8K | 84.46 | 84.61 | 100.18% | 84.46 | 100.00% |
| Winogrande | 83.27 | 82.32 | 98.86% | 82.72 | 99.34% |
| Hellaswag | 92.29 | 91.77 | 99.44% | 91.80 | 99.47% |
| BBH | 84.42 | 83.58 | 99.00% | 82.95 | 98.26% |
| TheoremQA | 22.38 | 21.89 | 97.81% | 21.38 | 95.53% |
| GPQA diamond | 42.42 | 42.42 | 100.00% | 42.93 | 101.20% |
| **Instruct Follow** | - | - | - | - | - |
| Prompt-level-strict-accuracy | 79.48 | 78.74 | 99.07% | 79.30 | 99.77% |
| Prompt-level-strict-accuracy | 85.25 | 85.37 | 100.14% | 85.97 | 100.84% |
| Prompt-level-loose-accuracy | 81.89 | 81.33 | 99.32% | 80.96 | 98.86% |
| Inst-level-loose-accuracy | 87.17 | 87.17 | 100.00% | 87.17 | 100.00% |
_以上 benchmark 以及量化工作均在 A800 80GB 显卡上完成。_
**并发说明**
在以上 benchmark 基准测试中，采用 vllm 推理，观察到以下基准测试中的并发表现。
- vLLM 0.6.1.post2
- 原模型 BF16 峰值并发（ *Nvidia A800 80GB* ），在无 Pending 的前提下，达到 Running 70 左右。
- 量化后的本模型 INT4(W4A16) 峰值并发( *Nvidia Geforce RTX 4090 24GB* )，在无 Pending 的前提下，达到 Running 35 左右。
## 附 录
##### _KV Cache Quantization_
本模型支持使用 vLLM 与 KV Cache 量化相结合，即 FP8_E4M3 KV Cache 方案。如下：
增加参数，--kv-cache-dtype fp8
```bash
>>> VLLM_ATTENTION_BACKEND=FLASHINFER python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.9 --kv-cache-dtype fp8 --served-model-name Qwen2.5-32B-Instruct-Int4-W4A16 --model /home/gavin/llm/Qwen2.5-32B-Instruct-Int4-W4A16
```
_* 以上命令行采用了 FlashInfer 作为 backend。 FlashInfer 的核心功能包括全面的注意力内核、优化共享前缀批处理解码、加速压缩/量化 KV Cache 的注意力、Grouped-Query Attention 优化、Fused-RoPE Attention 实现以及 Quantized Attention 加速。_
KV Cache 经 fp8 (FP8_E4M3) 量化以后，模型精度表现，如下参考：
| Benchmark | [Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct) (原模型) | _W4A16
- KV Cache Quantization_ | Recovery |
| ---------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | -------- |
| **Overall** | 73.21 | 71.48 | 97.64% |
| Exam | 86.33 | 84.83 | 98.26% |
| Language | 54.08 | 56.30 | 104.11% |
| Knowledge | 40.63 | 40.93 | 100.74% |
| Understanding | 92.89 | 92.39 | 99.46% |
| Coding | 85.54 | 79.63 | 93.09% |
| Reasoning | 69.56 | 68.06 | 97.84% |
| Instruct Follow | 83.45 | 78.20 | 93.71% |
| **Exam** | - | - | - |
| C-Eval | 85.87 | 84.26 | 98.13% |
| MMLU | 83.47 | 82.42 | 98.74% |
| CMMLU | 85.57 | 84.56 | 98.82% |
| GaokaoBench | 90.43 | 88.08 | 97.40% |
| **Language** | - | - | - |
| WSC | 89.42 | 92.31 | 103.23% |
| Tydiqa-goldp | 18.74 | 20.29 | 108.27% |
| **Knowledge** | - | - | - |
| TriviaQA | 60.68 | 61.33 | 101.07% |
| Natual Questions | 20.58 | 20.53 | 99.76% |
| **Understanding** | - | - | - |
| Race-middle | 94.29 | 93.59 | 99.26% |
| Race-high | 91.48 | 91.19 | 99.68% |
| **Coding** | - | - | - |
| Human eval | 87.80 | 79.88 | 90.98% |
| Sanitized MBPP | 83.27 | 79.38 | 95.33% |
| **Reasoning** | - | - | - |
| MATH | 77.66 | 76.94 | 99.07% |
| GSM8K | 84.46 | 82.64 | 97.85% |
| Winogrande | 83.27 | 82.40 | 98.96% |
| Hellaswag | 92.29 | 91.77 | 99.44% |
| BBH | 84.42 | 82.91 | 98.21% |
| TheoremQA | 22.38 | 21.88 | 97.77% |
| GPQA diamond | 42.42 | 37.88 | 89.30% |
| **Instruct Follow** | - | - | - |
| Prompt-level-strict-accuracy | 79.48 | 73.20 | 92.10% |
| Prompt-level-strict-accuracy | 85.25 | 80.58 | 94.52% |
| Prompt-level-loose-accuracy | 81.89 | 76.16 | 93.00% |
| Inst-level-loose-accuracy | 87.17 | 82.85 | 95.04% |
| | | | |
_______________________________
#### _作者在此 👇🏻 扫一扫_
