---
title: Qwen2-72B-Instruct-GPTQ-Int3
canonical_url: "https://www.modelscope.cn/models/tclf90/Qwen2-72B-Instruct-GPTQ-Int3"
md_url: "https://www.modelscope.cn/models/tclf90/Qwen2-72B-Instruct-GPTQ-Int3.md"
repository: tclf90/Qwen2-72B-Instruct-GPTQ-Int3
chinese_name: "通义千问2-72B-Instruct-GPTQ-Int3-量化修复"
last_updated: 2024-08-26
license: qwen
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen2
architectures:
  - Qwen2ForCausalLM
parameters: 11.5B
tensor_type:
  - I32
  - F16
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 879
stars: 7
tags:
  - qwen2
  - gptq
  - int3
  - "量化修复"
  - vLLM
  - sglang
---

# Qwen2-72B-Instruct-GPTQ-Int3

> Qwen2-72B-Instruct-GPTQ-Int3 - tclf90 在 ModelScope 开源的模型。通义千问2-72B-Instruct-GPTQ-Int3-量化修复 原模型 qwen/Qwen2-72B-Instruct

tclf90/Qwen2-72B-Instruct-GPTQ-Int3 是 ModelScope 魔搭社区上的 11.5B 参数text-generation模型，采用 qwen 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: tclf90/Qwen2-72B-Instruct-GPTQ-Int3
- **License**: qwen
- **Tasks**: text-generation
- **Parameters**: 11.5B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: qwen2, gptq, int3, 量化修复, vLLM, sglang
- **Downloads**: 879
- **Stars**: 7
- **Last updated**: 2024-08-26

Source: https://www.modelscope.cn/models/tclf90/Qwen2-72B-Instruct-GPTQ-Int3

---

# 通义千问2-72B-Instruct-GPTQ-Int3-量化修复
原模型 [qwen/Qwen2-72B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2-72B-Instruct)


### 【模型更新日期】
``` 
2024-08-26
优化量化算法，原有基础上继续降低量化损失约达50%

2024-06-07
首次commit
```

### 【模型大小】
`34.5GB`

### 【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
2. 有些推理框架的默认`top_k`与`top_p`较大，可以考虑减小对应数值，来获得更合理的模型输出。
3. 可以支持1卡、2卡及4卡的`tensor-parallel-size`启动。


### 【介绍】

Qwen2 is the new series of Qwen large language models. For Qwen2, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters, including a Mixture-of-Experts model. This repo contains the instruction-tuned 72B Qwen2 model.

Compared with the state-of-the-art opensource language models, including the previous released Qwen1.5, Qwen2 has generally surpassed most opensource models and demonstrated competitiveness against proprietary models across a series of benchmarks targeting for language understanding, language generation, multilingual capability, coding, mathematics, reasoning, etc.

[更多详情...](https://www.modelscope.cn/models/qwen/Qwen2-72B-Instruct/summary)


### 【模型下载】
```python
from modelscope import snapshot_download
model_dir = snapshot_download('tclf90/模型名', cache_dir="本地路径")
```

### 【高并发RESTFul API推理】
方式1：[vllm](https://github.com/vllm-project/vllm)

方式2：[sglang](https://github.com/sgl-project/sglang)

目前推荐使用sglang进行部署，相较于vllm, sglang于A100实测，能有50%～100%的吞吐增益。