---
title: qvq-72b-preview-gptq-int4
canonical_url: "https://www.modelscope.cn/models/tclf90/qvq-72b-preview-gptq-int4"
md_url: "https://www.modelscope.cn/models/tclf90/qvq-72b-preview-gptq-int4.md"
repository: tclf90/qvq-72b-preview-gptq-int4
chinese_name: "通义千问QVQ-72B-Preview-GPTQ-Int4"
last_updated: 2024-12-31
license: qwen
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen2_vl
architectures:
  - Qwen2VLForConditionalGeneration
parameters: 12.6B
tensor_type:
  - I32
  - F16
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb/vlm"
  - "lmdeploy 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 619
stars: 3
tags:
  - qwen2.5
  - gptq
  - int4
  - "量化修复"
  - vLLM
---

# qvq-72b-preview-gptq-int4

> qvq-72b-preview-gptq-int4 - tclf90 在 ModelScope 开源的模型。通义千问QVQ-72B-Preview-GPTQ-Int4 原始模型 Qwen/QVQ-72B-Preview

tclf90/qvq-72b-preview-gptq-int4 是 ModelScope 魔搭社区上的 12.6B 参数text-generation模型，采用 qwen 许可，可用 deploy_task text/emb/vlm、lmdeploy 0.9.1、sglang 0.5.2 部署。

- **Repository**: tclf90/qvq-72b-preview-gptq-int4
- **License**: qwen
- **Tasks**: text-generation
- **Parameters**: 12.6B
- **Inference backends**: deploy_task text/emb/vlm, lmdeploy 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: qwen2.5, gptq, int4, 量化修复, vLLM
- **Downloads**: 619
- **Stars**: 3
- **Last updated**: 2024-12-31

Source: https://www.modelscope.cn/models/tclf90/qvq-72b-preview-gptq-int4

---

# 通义千问QVQ-72B-Preview-GPTQ-Int4
原始模型 [Qwen/QVQ-72B-Preview](https://www.modelscope.cn/models/Qwen/QVQ-72B-Preview)


### 【模型更新日期】

注：通过`snapshot_download`函数传入`revision=...`来下载指定的`tag`版本

``` 
2024-12-31
1. add group 128 (tag g128)
```

### 【模型列表】

| tag     | 文件大小   | 最近更新时间       |
|---------|--------|--------------|
| `g128`  | `41GB` | `2024-12-31` |

<i>
如果启动时候遇到 gptq_marlin 报错，需要强行指定 --quantization gptq。
</i>

```python
from modelscope import snapshot_download
snapshot_download('tclf90/qvq-72b-preview-gptq-int4', cache_dir="本地路径", revision='g128')
```

### 【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
2. 根据模型实际情况，可以支持1卡、2卡及4卡的`tensor-parallel-size`启动。


### 【介绍】
**QVQ-72B-Preview** is an experimental research model developed by the Qwen team, focusing on enhancing visual reasoning capabilities.

## Performance

|                  | **QVQ-72B-Preview** | o1-2024-12-17 | gpt-4o-2024-05-13 | Claude3.5 Sonnet-20241022 | Qwen2VL-72B |
|------------------|---------------------|---------------|-------------------|---------------------------|-------------|
| MMMU(val)        | 70.3                | 77.3          | 69.1              | 70.4                      | 64.5        |
| MathVista(mini)  | 71.4                | 71.0          | 63.8              | 65.3                      | 70.5        |
| MathVision(full) | 35.9                | –             | 30.4              | 35.6                      | 25.9        |
| OlympiadBench    | 20.4                | –             | 25.9              | –                         | 11.2        |


**QVQ-72B-Preview** has achieved remarkable performance on various benchmarks. It scored a remarkable 70.3% on the Multimodal Massive Multi-task Understanding (MMMU) benchmark, showcasing QVQ's powerful ability in multidisciplinary understanding and reasoning. Furthermore, the significant improvements on MathVision highlight the model's progress in mathematical reasoning tasks. OlympiadBench also demonstrates the model's enhanced ability to tackle challenging problems.

***But It's Not All Perfect:  Acknowledging the Limitations***

While **QVQ-72B-Preview** exhibits promising performance that surpasses expectations, it’s important to acknowledge several limitations:

1. **Language Mixing and Code-Switching:** The model might occasionally mix different languages or unexpectedly switch between them, potentially affecting the clarity of its responses.
2. **Recursive Reasoning Loops:**  There's a risk of the model getting caught in recursive reasoning loops, leading to lengthy responses that may not even arrive at a final answer.
3. **Safety and Ethical Considerations:** Robust safety measures are needed to ensure reliable and safe performance. Users should exercise caution when deploying this model.
4. **Performance and Benchmark Limitations:** Despite the improvements in visual reasoning, QVQ doesn’t entirely replace the capabilities of Qwen2-VL-72B. During multi-step visual reasoning, the model might gradually lose focus on the image content, leading to hallucinations. Moreover, QVQ doesn’t show significant improvement over Qwen2-VL-72B in basic recognition tasks like identifying people, animals, or plants.

Note: Currently, the model only supports single-round dialogues and image outputs. It does not support video inputs.

### 【模型下载】
```python
from modelscope import snapshot_download
snapshot_download('tclf90/模型名', cache_dir="本地路径", revision='...tag...')
```

### 【高并发RESTFul API推理】
方式1：[vllm](https://github.com/vllm-project/vllm)