--- title: Qwen2.5-32B-Instruct-INT8-W8A16 canonical_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16" md_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16.md" repository: okwinds/Qwen2.5-32B-Instruct-Int8-W8A16 chinese_name: "通义千问 Qwen2.5-32B-Instruct-INT8-W8A16 量化高精校准" last_updated: 2025-04-06 license: qwen pipeline_tag: text-generation tasks: - text-generation model_type: - qwen2 architectures: - Qwen2ForCausalLM base_model: - Qwen/Qwen2.5-32B-Instruct base_model_relation: quantized parameters: 9.4B tensor_type: - I32 - I64 - BF16 library_name: - safetensors - pytorch frameworks: - pytorch language: - en - zh inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 3302 stars: 4 tags: - Chat - Instruct --- # Qwen2.5-32B-Instruct-INT8-W8A16 > Qwen2.5-32B-Instruct-Int8-W8A16 - okwinds 在 ModelScope 开源的模型。正式版 QwQ-32B 的量化版本已出，用了更优质的数据做校准。欢迎品尝：QwQ-32B-Int8-W8A16 okwinds/Qwen2.5-32B-Instruct-Int8-W8A16 是 ModelScope 魔搭社区上的 9.4B 参数text-generation模型，采用 qwen 许可，基于 Qwen/Qwen2.5-32B-Instruct 构建，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: okwinds/Qwen2.5-32B-Instruct-Int8-W8A16 - **License**: qwen - **Tasks**: text-generation - **Parameters**: 9.4B - **Base model**: Qwen/Qwen2.5-32B-Instruct - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2 - **Tags**: Chat, Instruct - **Downloads**: 3302 - **Stars**: 4 - **Last updated**: 2025-04-06 Source: https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16 --- 正式版 QwQ-32B 的量化版本已出，用了更优质的数据做校准。欢迎品尝：[QwQ-32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Int8-W8A16) # 通义千问 Qwen2.5-32B-Instruct-INT8-W8A16 量化高精校准 原模型 [qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct) ```python # download model from modelscope import snapshot_download model_dir = snapshot_download('okwinds/Qwen2.5-32B-Instruct-Int8-W8A16') ``` ## Qwen2.5 同系列量化模型 | **Model** | GPTQ Compression & FP8 | GGUF | **MLX** (macOS) | | ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *3B params* | - | [3B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-GGUF-V3-LOT) | [3B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-4bit)
[3B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-8bit) | | *7B params* | [7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int4-W4A16)
[7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int8-W8A16)
[7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-FP8-A16)
[Coder-7B-Instruct-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8) | [7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-GGUF-V3-LOT) | [7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-4bit)
[7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-8bit) | | *Coder 7B params* | [Coder-7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int4-W4A16)
[Coder-7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int8-W8A16)
[Coder-7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8-A16) | [Coder-7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-GGUF-V3-LOT) | [Coder-7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-4bit)
[Coder-7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-8bit) | | *14B params* | [14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16)
[14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int8-W8A16)
[14B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-FP8-A16) | [14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-GGUF-V3-LOT) | [14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-4bit)
[14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-8bit) | | *Coder 14B params* | [Coder-14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int4-W4A16)
[Coder-14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int8-W8A16) | [Coder-14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-GGUF-V3-LOT) | [Coder-14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-4bit)
[Coder-14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-8bit) | | *32B params* | [32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16)
[32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16)
[32B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16) | [32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-GGUF-V3-LOT) | [32B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-2bit)
[32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-4bit)
[32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-8bit) | | *Coder 32B params* | [Coder-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-Int4-W4A16)
[Coder-32B-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-FP8) | [Coder-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-GGUF-V3-LOT) | [Coder-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-4bit)
[Coder-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-8bit) | | *QwQ-Preview 32B params* | [QwQ-32B-Preview-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-Int4-W4A16)
[QwQ-32B-Preview-FP8](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-FP8) | [QwQ-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-GGUF-V3-LOT) | [QwQ-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-4bit)
[QwQ-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-8bit) | | *72B params* | - | [72B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT) | [72B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-2bit)
[72B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-4bit)
[72B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-8bit) | ## 模型概述 通义千问 Qwen2.5-32B-Instruct-INT8-W8A16 是一个基于 通义千问 Qwen2.5-32B-Instruct 的 INT8 量化并校准的模型。 - **模型名称:** Qwen2.5-32B-Instruct-Int8-W8A16 - **模型架构:** Qwen2.5 - **权重量化:** INT8 该模型通过将 [Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct) 的权重量化为 INT8 数据类型而实现。 量化过程将每个参数从 16bit 减少到 8bit，将模型占用磁盘空间大小，以及推理时加载模型需要的GPU显存空间，减少到了大约为原模型的1/2。 量化过程中，只有 transformer 中的 Linear 层权重是量化的，其他层均保持为原模型数据类型，采用混合精度的计算方式，尽可能减少模型量化后所带来的精度损失。 在量化过程中，使用了[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ)，采用 Symmetric group-wise 方式量化。并在量化过程中，进行了数据校准，以提升模型生成精度，修复 gptq-int8 模型出现乱吐字和“!”无限循环的问题。 > 注意：本模型需要 compute capability > 8.0（Ampere、Ada Lovelace、Hopper 架构）的 Nvidia GPU 来支持 INT8 **混合精度**计算。 ## 部署推理 #### 推荐使用 vLLM Openai api 兼容模式 ```bash >>> vllm serve "/home/gavin/llm/Qwen2.5-32B-Instruct-Int8-W8A16" --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.9 --served-model-name "Qwen2.5-32B-Instruct-Int8-W8A16" ``` ## Benchmark - INT8(W8A16) 的量化方法，对比原模型在精度上的表现，几乎无损失。在做数据校准的时候，也最大限度的保有了BF16原模型的精度。 | Benchmark | [Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct)(原模型) | _W8A16(本模型)_ | Recovery | | ---------------------------- | -------------------- |--------------| -------- | |**Overall**|73.21|73.16|99.93%| |Exam|86.33|86.34|100.01%| |Language|54.08|54.64|101.04%| |Knowledge|40.63|40.59|99.90%| |Understanding|92.89|92.75|99.85%| |Coding|85.54|84.93|99.29%| |Reasoning|69.56|69.46|99.86%| |Instruct Follow|83.45|83.38|99.92%| |**Exam**|-|-|-| |C-Eval|85.87|85.80|99.92%| |MMLU|83.47|83.48|100.01%| |CMMLU|85.57|85.75|100.21%| |GaokaoBench|90.43|90.32|99.88%| |**Language**|-|-|-| |WSC|89.42|90.38|101.07%| |Tydiqa-goldp|18.74|18.90|100.85%| |**Knowledge**|-|-|-| |TriviaQA|60.68|60.53|99.75%| |Natual Questions|20.58|20.64|100.29%| |**Understanding**|-|-|-| |Race-middle|94.29|94.08|99.78%| |Race-high|91.48|91.42|99.93%| |**Coding**|-|-|-| |Human eval|87.80|86.59|98.62%| |Sanitized MBPP|83.27|83.27|100.00%| |**Reasoning**|-|-|-| |MATH|77.66|78.18|100.67%| |GSM8K|84.46|84.69|100.27%| |Winogrande|83.27|83.11|99.81%| |Hellaswag|92.29|92.27|99.98%| |BBH|84.42|84.30|99.86%| |TheoremQA|22.38|22.75|101.65%| |GPQA diamond|42.42|40.91|96.44%| |**Instruct Follow**|-|-|-| |Prompt-level-strict-accuracy|79.48|79.48|100.00%| |Prompt-level-strict-accuracy|85.25|85.73|100.56%| |Prompt-level-loose-accuracy|81.89|81.15|99.10%| |Inst-level-loose-accuracy|87.17|87.17|100.00%| _以上 benchmark 以及量化工作均在 A800 80GB 显卡上完成。_ **并发说明** 在以上 benchmark 基准测试中，采用 vllm 推理，观察到以下基准测试中的并发表现。 - vLLM 0.6.1.post2 - 原模型 BF16 峰值并发，在无 Pending 的前提下，达到 Running 70 左右。 - 量化后的本模型 INT8(W8A16) 峰值并发，在无 Pending 的前提下，达到 Running 200 左右。
_______________________________ #### _作者在此 👇🏻 扫一扫_