---
title: Qwen1.5-32B-Chat-GPTQ-Int3
canonical_url: "https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3"
md_url: "https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3.md"
repository: tclf90/Qwen1.5-32B-Chat-GPTQ-Int3
chinese_name: "通义千问1.5-32B-Chat-GPTQ-Int3-量化修复"
last_updated: 2024-08-29
license: qwen
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen2
architectures:
  - Qwen2ForCausalLM
parameters: 4.7B
tensor_type:
  - F16
  - I32
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 1077
stars: 3
tags:
  - qwen1.5
  - gptq
  - int3
  - "量化修复"
  - vLLM
  - sglang
---

# Qwen1.5-32B-Chat-GPTQ-Int3

> Qwen1.5-32B-Chat-GPTQ-Int3 - tclf90 在 ModelScope 开源的模型。通义千问1.5-32B-Chat-GPTQ-Int3-量化修复 原模型 qwen/Qwen1.5-32B-Chat

tclf90/Qwen1.5-32B-Chat-GPTQ-Int3 是 ModelScope 魔搭社区上的 4.7B 参数text-generation模型，采用 qwen 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: tclf90/Qwen1.5-32B-Chat-GPTQ-Int3
- **License**: qwen
- **Tasks**: text-generation
- **Parameters**: 4.7B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: qwen1.5, gptq, int3, 量化修复, vLLM, sglang
- **Downloads**: 1077
- **Stars**: 3
- **Last updated**: 2024-08-29

Source: https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3

---

# 通义千问1.5-32B-Chat-GPTQ-Int3-量化修复
原模型 [qwen/Qwen1.5-32B-Chat](https://www.modelscope.cn/models/qwen/Qwen1.5-32B-Chat)


### 【模型更新日期】

注：通过`snapshot_download`函数传入`revision=...`来下载指定的`tag`版本

``` 
【tag】2024-08-30
1. 优化量化算法，原有基础上继续降低约50%的量化损失
2. 该tag版本模型为 groupsize 128, 支持最多双卡负载 (tp size = 2)

【tag】v1.0.0
2024-05-24
1. 首版量化修复模型, 修复千文官方量化的无限感叹号等问题，并进一步降低量化比特
```

### 【模型大小】
`15GB`

### 【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
2. 有些推理框架的默认`top_k`与`top_p`较大，可以考虑减小对应数值，来获得更合理的模型输出。
3. 根据模型实际情况，可以支持1卡、2卡及4卡的`tensor-parallel-size`启动。


### 【介绍】

Qwen1.5 is the beta version of Qwen2, a transformer-based decoder-only language model pretrained on a large amount of data. In comparison with the previous released Qwen, the improvements include: 

* 8 model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, 32B and 72B dense models, and an MoE model of 14B with 2.7B activated;
* Significant performance improvement in human preference for chat models;
* Multilingual support of both base and chat models;
* Stable support of 32K context length for models of all sizes
* No need of `trust_remote_code`.

For more details, please refer to our [blog post](https://qwenlm.github.io/blog/qwen1.5/) and [GitHub repo](https://github.com/QwenLM/Qwen1.5).


### 【模型下载】
```python
from modelscope import snapshot_download
model_dir = snapshot_download('tclf90/模型名', cache_dir="本地路径")
```

### 【高并发RESTFul API推理】
方式1：[vllm](https://github.com/vllm-project/vllm)

方式2：[sglang](https://github.com/sgl-project/sglang)

目前推荐使用sglang进行部署，相较于vllm, sglang于A100实测，能有50%～100%的吞吐增益。