--- title: qwq-32b-gptq-int4 canonical_url: "https://www.modelscope.cn/models/tclf90/qwq-32b-gptq-int4" md_url: "https://www.modelscope.cn/models/tclf90/qwq-32b-gptq-int4.md" repository: tclf90/qwq-32b-gptq-int4 chinese_name: "通义千问 QwQ-32B-GPTQ-Int4" last_updated: 2025-05-07 license: qwen pipeline_tag: text-generation tasks: - text-generation model_type: - qwen2 architectures: - Qwen2ForCausalLM base_model: - Qwen/QwQ-32B base_model_relation: quantized parameters: 5.7B tensor_type: - F16 - I32 library_name: - safetensors - pytorch frameworks: - Pytorch inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 87159 stars: 9 tags: - QwQ - gptq - int4 - "量化修复" - vLLM - sglang --- # qwq-32b-gptq-int4 > qwq-32b-gptq-int4 - tclf90 在 ModelScope 开源的模型。通义千问 QwQ-32B-GPTQ-Int4 基础型 Qwen/QwQ-32B tclf90/qwq-32b-gptq-int4 是 ModelScope 魔搭社区上的 5.7B 参数text-generation模型，采用 qwen 许可，基于 Qwen/QwQ-32B 构建，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: tclf90/qwq-32b-gptq-int4 - **License**: qwen - **Tasks**: text-generation - **Parameters**: 5.7B - **Base model**: Qwen/QwQ-32B - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2 - **Tags**: QwQ, gptq, int4, 量化修复, vLLM, sglang - **Downloads**: 87159 - **Stars**: 9 - **Last updated**: 2025-05-07 Source: https://www.modelscope.cn/models/tclf90/qwq-32b-gptq-int4 --- # 通义千问 QwQ-32B-GPTQ-Int4 基础型 [Qwen/QwQ-32B](https://www.modelscope.cn/models/Qwen/QwQ-32B) ### 【模型更新日期】 ``` 2025-04-24 1. add group 128v3 2. 用黑科技修掉之前 Compute 7 显卡（GPTQ格式）运行时候出现的感叹号问题 3. 删除对话模板的字符，使得模型作答的时候，能完整吐出模块 4. 提供一个无思考模式的对话模板（tokenizer_config-无思考模式.json）供参考把玩。 2025-03-06 1. add group 128 ``` ### 【依赖】 ``` vllm==0.8.4 transformers==4.51.3 ```
### 【💡新版 VLLM 注意事项💡】 #### 1. 需使用V0推理模式 启动vllm之前，先设置环境变量 ``` export VLLM_USE_V1=0 ```
### 【模型列表】 | tag | 文件大小 | 最近更新时间 | |----------|--------|--------------| | `g128v3` | `19GB` | `2025-04-24` | | `g128` | `19GB` | `2025-03-06` | ### 【模型下载】 ```python from modelscope import snapshot_download snapshot_download('tclf90/qwq-32b-gptq-int4', cache_dir="本地路径", revision='g128v3') ``` ### 【修复内容】 1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。 2. 支持1卡、2卡及4卡的`tensor-parallel-size`启动。 ### 【介绍】 Chat ## Introduction QwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini. **This repo contains the QwQ 32B model**, which has the following features: - Type: Causal Language Models - Training Stage: Pretraining & Post-training (Supervised Finetuning and Reinforcement Learning) - Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias - Number of Parameters: 32.5B - Number of Paramaters (Non-Embedding): 31.0B - Number of Layers: 64 - Number of Attention Heads (GQA): 40 for Q and 8 for KV - Context Length: Full 131,072 tokens **Note:** For the best experience, please review the [usage guidelines](#usage-guidelines) before deploying QwQ models. For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).