--- title: Qwen2.5-14B-Instruct-Int4-W4A16 canonical_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16" md_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16.md" repository: okwinds/Qwen2.5-14B-Instruct-Int4-W4A16 chinese_name: "通义千问Qwen2.5-14B-Instruct-Int4-W4A16量化高精校准" last_updated: 2025-01-05 license: qwen pipeline_tag: text-generation tasks: - text-generation model_type: - qwen2 architectures: - Qwen2ForCausalLM base_model: - Qwen/Qwen2.5-14B-Instruct base_model_relation: finetune parameters: 3.3B tensor_type: - BF16 - I64 - I32 library_name: - safetensors - pytorch frameworks: - pytorch language: - en - zh inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 408 stars: 5 tags: - Chat - Instruct --- # Qwen2.5-14B-Instruct-Int4-W4A16 > Qwen2.5-14B-Instruct-Int4-W4A16 - okwinds 在 ModelScope 开源的模型。通义千问2.5-14B-Instruct-Int4-W4A16量化高精校准 okwinds/Qwen2.5-14B-Instruct-Int4-W4A16 是 ModelScope 魔搭社区上的 3.3B 参数text-generation模型，采用 qwen 许可，基于 Qwen/Qwen2.5-14B-Instruct 构建，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: okwinds/Qwen2.5-14B-Instruct-Int4-W4A16 - **License**: qwen - **Tasks**: text-generation - **Parameters**: 3.3B - **Base model**: Qwen/Qwen2.5-14B-Instruct - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2 - **Tags**: Chat, Instruct - **Downloads**: 408 - **Stars**: 5 - **Last updated**: 2025-01-05 Source: https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16 --- # 通义千问2.5-14B-Instruct-Int4-W4A16量化高精校准 原模型 [qwen/Qwen2.5-14B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-14B-Instruct) ```python # download model from modelscope import snapshot_download model_dir = snapshot_download('okwinds/Qwen2.5-14B-Instruct-Int4-W4A16') ``` ## Qwen2.5 同系列量化模型 | **Model** | GPTQ Compression & FP8 | GGUF | **MLX** (macOS) | | ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *3B params* | - | [3B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-GGUF-V3-LOT) | [3B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-4bit)
[3B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-8bit) | | *7B params* | [7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int4-W4A16)
[7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int8-W8A16)
[7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-FP8-A16)
[Coder-7B-Instruct-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8) | [7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-GGUF-V3-LOT) | [7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-4bit)
[7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-8bit) | | *Coder 7B params* | [Coder-7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int4-W4A16)
[Coder-7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int8-W8A16)
[Coder-7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8-A16) | [Coder-7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-GGUF-V3-LOT) | [Coder-7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-4bit)
[Coder-7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-8bit) | | *14B params* | [14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16)
[14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int8-W8A16)
[14B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-FP8-A16) | [14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-GGUF-V3-LOT) | [14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-4bit)
[14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-8bit) | | *Coder 14B params* | [Coder-14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int4-W4A16)
[Coder-14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int8-W8A16) | [Coder-14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-GGUF-V3-LOT) | [Coder-14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-4bit)
[Coder-14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-8bit) | | *32B params* | [32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16)
[32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16)
[32B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16) | [32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-GGUF-V3-LOT) | [32B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-2bit)
[32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-4bit)
[32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-8bit) | | *Coder 32B params* | [Coder-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-Int4-W4A16)
[Coder-32B-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-FP8) | [Coder-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-GGUF-V3-LOT) | [Coder-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-4bit)
[Coder-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-8bit) | | *QwQ-Preview 32B params* | [QwQ-32B-Preview-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-Int4-W4A16)
[QwQ-32B-Preview-FP8](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-FP8) | [QwQ-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-GGUF-V3-LOT) | [QwQ-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-4bit)
[QwQ-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-8bit) | | *72B params* | - | [72B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT) | [72B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-2bit)
[72B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-4bit)
[72B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-8bit) | ## 模型概述 > 本模型已升级，请使用精度更高的最新版本，原版本归档至 tag v1.0.0。
更多精度详情，请关注 Benchmark 部分数据。 通义千问2.5-14B-Instruct-Int4-W4A16 是一个基于 通义千问2.5-14B-Instruct 的 INT4 量化校准模型。 - **模型名称:** Qwen2.5-14B-Instruct-Int4-W4A16 - **模型架构:** Qwen2.5 - **权重量化:** INT4 该模型通过将 [Qwen2.5-14B-Instruct](https://www.modelscope.cn/models/qwen/qwen2.5-14B-instruct) 的权重量化为 INT4 数据类型而实现。 量化过程将每个参数从 16bit 减少到 4bit，将模型占用磁盘空间大小，以及推理时加载模型需要的GPU显存空间，减少到了大约为原模型的1/4。量化过程中，只有 transformer 中的 Linear 层权重是量化的，其他层均保持为原模型数据类型，采用混合精度的计算方式，尽可能减少模型量化后所带来的精度损失。 在量化过程中，使用了[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ)，采用 Symmetric group-wise 方式量化（group size : 128）。并在量化过程中，进行了数据校准，以提升模型生成精度，修复 gptq-int4 模型出现乱吐字和无限循环的问题。 > 注意：本模型需要 compute capability > 8.0（Ampere、Ada Lovelace、Hopper 架构）的 Nvidia GPU 来支持 INT4 **混合精度**计算。 ## 部署推理 #### 推荐使用 vLLM Openai api 兼容模式 ```bash >>> vllm serve "/home/gavin/llm/Qwen2.5-14B-Instruct-Int4-W4A16" --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.9 --served-model-name "Qwen2.5-14B-Instruct-Int4-W4A16" ``` ## Benchmark _仅 Linear layer 量化为 INT4 数据类型，其他层保持为原模型数据类型，采用混合精度计算的方式，有助于提升量化后模型的精度表现。_ 本模型的最新版本对精度进行了更广泛的校准。尤其对推理，CoT，指令遵循，Agent等能力做了强化提升。 | Benchmark | Qwen2.5-14B-Instruct | Int4-W4A16-v1.0.0 | Recovery-v1.0.0 | _Int4-W4A16-master(New)_ | Recovery-master(New) | | ------------------- | -------------------- | ----------------- | --------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------- | | **Overall** | 72.59 | 70.92 | 97.71% | 71.65 | 98.71% | | Exam | 82.67 | 81.59 | 98.70% | 81.13 | 98.14% | | Language | 57.40 | 56.02 | 97.60% | 56.20 | 97.91% | | Knowledge | 59.94 | 58.33 | 97.31% | 59.72 | 99.63% | | Understanding | 72.66 | 72.65 | 99.99% | 72.67 | 100.01% | | Coding | 80.57 | 78.35 | 97.24% | 78.68 | 97.65% | | Reasoning | 71.88 | 71.70 | 99.75% | 71.74 | 99.81% | | Instruct Follow | 82.99 | 77.82 | 93.77% | 81.41 | 98.10% | | **Exam** | - | - | - | - | - | | C-Eval | 81.38 | 80.80 | 99.29% | 80.01 | 98.32% | | MMLU | 79.44 | 78.68 | 99.04% | 78.33 | 98.60% | | CMMLU | 82.75 | 81.05 | 97.95% | 81.25 | 98.19% | | GaokaoBench | 87.09 | 85.83 | 98.55% | 84.91 | 97.50% | | **Language** | - | - | - | - | - | | WiC | 56.74 | 55.64 | 98.06% | 55.64 | 98.06% | | WSC | 81.73 | 78.85 | 96.48% | 78.85 | 96.48% | | Afqmc-DEV | 70.71 | 70.00 | 99.00% | 70.57 | 99.80% | | Tydiqa-Goldp | 20.42 | 19.59 | 95.94% | 19.72 | 96.57% | | **Knowledge** | - | - | - | - | - | | BoolQ | 84.04 | 84.40 | 100.43% | 84.80 | 100.90% | | TriviaQA | 56.89 | 53.71 | 94.41% | 53.97 | 94.87% | | GPQA Diamond | 38.89 | 36.87 | 94.81% | 40.40 | 103.88% | | **Understanding** | - | - | - | - | - | | C3 | 93.92 | 93.92 | 100.00% | 94.30 | 100.40% | | Race-Middle | 93.11 | 92.83 | 99.70% | 92.97 | 99.85% | | Race-High | 89.79 | 89.74 | 99.94% | 89.34 | 99.50% | | LCSTS | 13.82 | 14.12 | 102.17% | 14.05 | 101.66% | | **Coding** | - | - | - | - | - | | Human Eval | 82.93 | 79.27 | 95.59% | 81.10 | 97.79% | | Sanitized MBPP | 78.21 | 77.43 | 99.00% | 76.26 | 97.51% | | **Reasoning** | - | - | - | - | - | | Ocnli | 68.31 | 65.86 | 96.41% | 66.98 | 98.05% | | COPA | 98.00 | 99.00 | 101.02% | 99.00 | 101.02% | | MATH | 51.82 | 48.58 | 93.75% | 52.76 | 101.81% | | GSM8K | 89.84 | 89.01 | 99.08% | 89.69 | 99.83% | | Winogrande | 78.85 | 77.27 | 98.00% | 78.06 | 99.00% | | Hellaswag | 90.71 | 90.88 | 100.19% | 90.35 | 99.60% | | BBH | 75.88 | 76.61 | 100.96% | 75.94 | 100.08% | | TheoremQA | 21.62 | 22.62 | 104.63% | 21.12 | 97.69% | | **Instruct Follow** | - | - | - | - | - | | IFEval | 82.99 | 77.82 | 93.77% | 81.41 | 98.10% | _以上 benchmark 在 Nvidia L40 上完成，量化工作在 Nvidia 4090 RTX 24GB 显卡上完成。_ ***并发说明*** 在以上 benchmark 基准测试中，采用 vllm 推理，观察到以下基准测试中的并发表现。 - vLLM 0.6.1.post2 - 原模型 BF16 峰值并发，在无 pending 的前提下，达到 running 100左右。(*Nvidia L40*) - 量化后的本模型 W4A16 峰值并发，在无 pending 的前提下，达到 running 195左右。 (*Nvidia 4090 RTX*)
_______________________________ #### _作者在此 👇🏻 扫一扫_