---
title: qwen2-vl-72b-instruct-gptq-int3
canonical_url: "https://www.modelscope.cn/models/tclf90/qwen2-vl-72b-instruct-gptq-int3"
md_url: "https://www.modelscope.cn/models/tclf90/qwen2-vl-72b-instruct-gptq-int3.md"
repository: tclf90/qwen2-vl-72b-instruct-gptq-int3
chinese_name: "通义千问2-VL-72B-Instruct-GPTQ-Int3-量化修复"
last_updated: 2024-09-26
license: qwen
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen2_vl
architectures:
  - Qwen2VLForConditionalGeneration
parameters: 12.2B
tensor_type:
  - F16
  - I32
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb/vlm"
  - "lmdeploy 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 221
stars: 2
tags:
  - qwen2
  - gptq
  - int3
  - "量化修复"
  - vLLM
---

# qwen2-vl-72b-instruct-gptq-int3

> qwen2-vl-72b-instruct-gptq-int3 - tclf90 在 ModelScope 开源的模型。通义千问2-VL-72B-Instruct-GPTQ-Int3-量化修复 原模型 Qwen/Qwen2-VL-72B-Instruct

tclf90/qwen2-vl-72b-instruct-gptq-int3 是 ModelScope 魔搭社区上的 12.2B 参数text-generation模型，采用 qwen 许可，可用 deploy_task text/emb/vlm、lmdeploy 0.9.1、sglang 0.5.2 部署。

- **Repository**: tclf90/qwen2-vl-72b-instruct-gptq-int3
- **License**: qwen
- **Tasks**: text-generation
- **Parameters**: 12.2B
- **Inference backends**: deploy_task text/emb/vlm, lmdeploy 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: qwen2, gptq, int3, 量化修复, vLLM
- **Downloads**: 221
- **Stars**: 2
- **Last updated**: 2024-09-26

Source: https://www.modelscope.cn/models/tclf90/qwen2-vl-72b-instruct-gptq-int3

---

# 通义千问2-VL-72B-Instruct-GPTQ-Int3-量化修复
原模型 [Qwen/Qwen2-VL-72B-Instruct](https://www.modelscope.cn/models/qwen/qwen2-vl-72b-instruct)


### 【模型更新日期】

注：通过`snapshot_download`函数传入`revision=...`来下载指定的`tag`版本

``` 
2024-09-26
1. add group 32
```

### 【模型列表】

| tag    | 文件大小   | 最近更新时间       |
|--------|--------|--------------|
| `g32`  | `36GB` | `2024-09-26` |


### 【模型下载】
```python
from modelscope import snapshot_download
snapshot_download('tclf90/...', cache_dir="本地路径", revision='g32')
```

### 【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
2. 相同技术移植至AWQ。
3. 有些推理框架的默认`top_k`与`top_p`较大，可以考虑减小对应数值，来获得更合理的模型输出。
4. 根据模型实际情况，可以支持2卡、4卡及8卡的`tensor-parallel-size`启动。

### 【高并发RESTFul API推理】
方式1：[vllm](https://github.com/vllm-project/vllm)

截止 `2024-09-26`为止，vllm启动qwen2 vl模型需要依次进行下述操作：

1. 安装/升级python相关module
```
pip install -U vllm
pip install -U --force git+https://github.com/huggingface/transformers
pip install qwen-vl-utils
```

2. 编辑，于 [.../vllm/model_executor/models/qwen2.py#L175](https://github.com/fyabc/vllm/blob/6f3116c9dad0537b2858af703938aa9bf6c25bcf/vllm/model_executor/models/qwen2.py#L175) 前面加一行插入下述代码
```
if rope_scaling and rope_scaling.get('type','default') == 'default': 
    rope_scaling['type'] = 'mrope'
```

随后即可用vllm常规启动qwen2 vl模型。

### 【API推理测试】
下述为简易的两个API调用示例，具体参考qwen2 vl官方说明。
```python
import requests
from openai import OpenAI

# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://0.0.0.0:8080/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

url = f'{openai_api_base}/models'
response = requests.get(url=url)
# print(f'status {response.status_code}')
data = response.json()
# print(json.dumps(data, indent=2, ensure_ascii=False))
model_name = data['data'][0]['id']
print(model_name)

chat_response = client.chat.completions.create(
    model=model_name,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"
                    },
                },
                {"type": "text", "text": "图中写了什么字?"},
            ],
        },
    ],
)
print("Chat response:", chat_response)

chat_response = client.chat.completions.create(
    model=model_name,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
                    },
                },
                {"type": "text", "text": "描述一下这张图"},
            ],
        },
    ],
)
print("Chat response:", chat_response)
```



### 【介绍】
* **Naive Dynamic Resolution**: Unlike before, Qwen2-VL can handle arbitrary image resolutions, mapping them into a dynamic number of visual tokens, offering a more human-like visual processing experience.

<p align="center">
    <img src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/Qwen2-VL/qwen2_vl.jpg" width="80%"/>
<p>

* **Multimodal Rotary Position Embedding (M-ROPE)**: Decomposes positional embedding into parts to capture 1D textual, 2D visual, and 3D video positional information, enhancing its multimodal processing capabilities.

<p align="center">
    <img src="http://qianwen-res.oss-accelerate-overseas.aliyuncs.com/Qwen2-VL/mrope.png" width="80%"/>
<p>

We have three models with 2, 8 and 72 billion parameters. This repo contains the instruction-tuned 72B Qwen2-VL model. For more information, visit our [Blog](https://qwenlm.github.io/blog/qwen2-vl/) and [GitHub](https://github.com/QwenLM/Qwen2-VL).
