--- title: Qwen2.5-32B-Instruct-FP8-A16 canonical_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16" md_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16.md" repository: okwinds/Qwen2.5-32B-Instruct-FP8-A16 chinese_name: "通义千问Qwen2.5-32B-Instruct-FP8-A16高精量化校准" last_updated: 2024-12-01 license: qwen pipeline_tag: text-generation tasks: - text-generation model_type: - qwen2 architectures: - Qwen2ForCausalLM base_model: - Qwen/Qwen2.5-32B-Instruct base_model_relation: quantized parameters: 32.8B tensor_type: - F8_E4M3 - BF16 - F32 library_name: - safetensors - pytorch frameworks: - pytorch language: - en - zh inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 592 stars: 5 tags: - Chat - Instruct --- # Qwen2.5-32B-Instruct-FP8-A16 > Qwen2.5-32B-Instruct-FP8-A16 - okwinds 在 ModelScope 开源的模型。# 通义千问 Qwen2.5-32B-Instruct-FP8-A16 高精量化校准 okwinds/Qwen2.5-32B-Instruct-FP8-A16 是 ModelScope 魔搭社区上的 32.8B 参数text-generation模型，采用 qwen 许可，基于 Qwen/Qwen2.5-32B-Instruct 构建，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: okwinds/Qwen2.5-32B-Instruct-FP8-A16 - **License**: qwen - **Tasks**: text-generation - **Parameters**: 32.8B - **Base model**: Qwen/Qwen2.5-32B-Instruct - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2 - **Tags**: Chat, Instruct - **Downloads**: 592 - **Stars**: 5 - **Last updated**: 2024-12-01 Source: https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16 --- # # 通义千问 Qwen2.5-32B-Instruct-FP8-A16 高精量化校准 原模型 [qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct) ```python # download model from modelscope import snapshot_download model_dir = snapshot_download('okwinds/Qwen2.5-32B-Instruct-FP8-A16') ``` ## Qwen2.5 同系列量化模型 | **Model** | GPTQ Compression & FP8 | GGUF | **MLX** (macOS) | | ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *3B params* | - | [3B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-GGUF-V3-LOT) | [3B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-4bit)
[3B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-8bit) | | *7B params* | [7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int4-W4A16)
[7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int8-W8A16)
[7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-FP8-A16)
[Coder-7B-Instruct-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8) | [7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-GGUF-V3-LOT) | [7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-4bit)
[7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-8bit) | | *Coder 7B params* | [Coder-7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int4-W4A16)
[Coder-7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int8-W8A16)
[Coder-7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8-A16) | [Coder-7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-GGUF-V3-LOT) | [Coder-7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-4bit)
[Coder-7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-8bit) | | *14B params* | [14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16)
[14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int8-W8A16)
[14B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-FP8-A16) | [14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-GGUF-V3-LOT) | [14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-4bit)
[14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-8bit) | | *Coder 14B params* | [Coder-14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int4-W4A16)
[Coder-14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int8-W8A16) | [Coder-14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-GGUF-V3-LOT) | [Coder-14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-4bit)
[Coder-14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-8bit) | | *32B params* | [32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16)
[32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16)
[32B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16) | [32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-GGUF-V3-LOT) | [32B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-2bit)
[32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-4bit)
[32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-8bit) | | *Coder 32B params* | [Coder-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-Int4-W4A16)
[Coder-32B-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-FP8) | [Coder-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-GGUF-V3-LOT) | [Coder-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-4bit)
[Coder-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-8bit) | | *QwQ-Preview 32B params* | [QwQ-32B-Preview-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-Int4-W4A16)
[QwQ-32B-Preview-FP8](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-FP8) | [QwQ-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-GGUF-V3-LOT) | [QwQ-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-4bit)
[QwQ-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-8bit) | | *72B params* | - | [72B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT) | [72B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-2bit)
[72B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-4bit)
[72B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-8bit) | ## 模型概述 Qwen2.5-32B-Instruct-FP8-A16 是一个基于 通义千问2.5-32B-Instruct 的 FP8 量化并校准的模型。 - **模型名称:** Qwen2.5-32B-Instruct-FP8-A16 - **模型架构:** Qwen2.5 - **权重量化:** FP8 ### 量化模型 该模型通过将 [qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-32B-Instruct) 的权重量化为 FP8 数据类型而实现。 量化过程将每个参数从 16bit 减少到 8bit，将模型占用磁盘空间大小，以及推理时加载模型需要的GPU显存空间，减少到了大约为原模型的1/2。 FP8（Floating Point 8-bit）是一种8位浮点数数据格式，由Nvidia、Arm、Intel联合推出，旨在加速深度学习的训练和推理过程。 本模型的量化，采用 F8_E4M3 数据类型仅对 transformer 中的 Linear 层权重进行 static 量化，其他层均保持为原模型数据类型，采用混合精度的计算方式，尽可能减少模型量化后所带来的精度损失。 并且，在量化过程中，进行了数据校准，以进一步提升模型生成精度。 > 注意：本模型需要 compute capability > 8.0（Ampere、Ada Lovelace、Hopper 架构）的 Nvidia GPU 来支持 FP8 及**混合精度**计算。 ## 部署推理 #### 推荐使用 vLLM （ version > 0.5.0 ） Openai api 兼容模式 ```bash >>> python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --quantization fp8 --gpu-memory-utilization 0.9 --served-model-name "Qwen2.5-32B-Instruct-FP8-A16" --model "/home/gavin/llm/Qwen2.5-32B-Instruct-FP8-A16" ``` ## Benchmark _FP8 (A16) 的量化方法，对比原模型在精度上的总体表现，损失较低。_ | Benchmark | Qwen2.5-32B-Instruct | _Qwen2.5-32B-Instruct-FP8-A16_ | Recovery | |-------------------------------------| -------------------- | ------------------------------ | -------- | | **Overall** | 73.21 | 72.98 | 99.71% | | Exam | 86.33 | 86.49 | 100.19% | | Language | 54.08 | 54.20 | 100.22% | | Knowledge | 40.63 | 40.54 | 99.78% | | Understanding | 92.89 | 92.81 | 99.91% | | Coding | 85.54 | 84.95 | 99.31% | | Reasoning | 69.56 | 68.66 | 98.71% | | Instruct Follow | 83.45 | 83.18 | 99.68% | | **Exam** | - | - | - | | C-Eval | 85.87 | 86.25 | 100.44% | | MMLU | 83.47 | 83.54 | 100.08% | | CMMLU | 85.57 | 85.73 | 100.19% | | GaokaoBench | 90.43 | 90.43 | 100.00% | | **Language** | - | - | - | | WSC | 89.42 | 89.42 | 100.00% | | Tydiqa-goldp | 18.74 | 18.98 | 101.28% | | **Knowledge** | - | - | - | | TriviaQA | 60.68 | 60.48 | 99.67% | | Natual Questions | 20.58 | 20.61 | 100.15% | | **Understanding** | - | - | - | | Race-middle | 94.29 | 94.08 | 99.78% | | Race-high | 91.48 | 91.54 | 100.07% | | **Coding** | - | - | - | | Humaneval | 87.80 | 87.80 | 100.00% | | Sanitized MBPP | 83.27 | 82.10 | 98.59% | | **Reasoning** | - | - | - | | MATH | 77.66 | 77.48 | 99.77% | | GSM8K | 84.46 | 84.91 | 100.53% | | Winogrande | 83.27 | 83.66 | 100.47% | | Hellaswag | 92.29 | 92.02 | 99.71% | | BBH | 84.42 | 84.07 | 99.59% | | TheoremQA | 22.38 | 21.62 | 96.60% | | GPQA Diamond | 42.42 | 36.87 | 86.92% | | **Instruct Follow** | - | - | - | | Prompt-level-strict-accuracy | 79.48 | 79.11 | 99.53% | | Prompt-level-strict-accuracy | 85.25 | 84.77 | 99.44% | | Prompt-level-loose-accuracy | 81.89 | 81.89 | 100.00% | | Inst-level-loose-accuracy | 87.17 | 86.93 | 99.72% | _以上 benchmark 以及量化工作均在 A800 80GB 显卡上完成。_ ***并发说明*** 在以上 benchmark 基准测试中，采用 vllm 推理，观察到以下基准测试中的并发表现。 - vLLM 0.6.1.post2 - 原模型 BF16 峰值并发，在无 pending 的前提下，达到 running 70 左右。 - 量化后的本模型 FP8(A16) 峰值并发 ，在无 pending 的前提下，达到 running 130 左右。