---
title: rombos-llm-v2.6-qwen-14b-gptq-int4
canonical_url: "https://www.modelscope.cn/models/tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4"
md_url: "https://www.modelscope.cn/models/tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4.md"
repository: tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4
chinese_name: "Rombos-LLM-V2.6-Qwen-14B-GPTQ-Int4-量化修复"
last_updated: 2024-10-30
license: qwen
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen2
architectures:
  - Qwen2ForCausalLM
parameters: 3.3B
tensor_type:
  - I32
  - F16
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 96
stars: 2
tags:
  - qwen2.5
  - gptq
  - int4
  - "量化修复"
  - vLLM
  - sglang
---

# rombos-llm-v2.6-qwen-14b-gptq-int4

> rombos-llm-v2.6-qwen-14b-gptq-int4 - tclf90 在 ModelScope 开源的模型。Rombos-LLM-V2.6-Qwen-14B-GPTQ-Int4-量化修复 HF原始模型 rombodawg/Rombos-LLM-V2.6-Qwen-14b

tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4 是 ModelScope 魔搭社区上的 3.3B 参数text-generation模型，采用 qwen 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4
- **License**: qwen
- **Tasks**: text-generation
- **Parameters**: 3.3B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: qwen2.5, gptq, int4, 量化修复, vLLM, sglang
- **Downloads**: 96
- **Stars**: 2
- **Last updated**: 2024-10-30

Source: https://www.modelscope.cn/models/tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4

---

# Rombos-LLM-V2.6-Qwen-14B-GPTQ-Int4-量化修复
HF原始模型 [rombodawg/Rombos-LLM-V2.6-Qwen-14b](https://huggingface.co/rombodawg/Rombos-LLM-V2.6-Qwen-14b)


### 【模型更新日期】

注：通过`snapshot_download`函数传入`revision=...`来下载指定的`tag`版本

``` 
2024-10-30
1. add group 128 非对称量化 (tag g128-asym)

2024-10-28
1. add group 32 (tag g32)
1. add group 128 (tag g128)
```

### 【模型列表】

| tag         | 文件大小    | 最近更新时间       |
|-------------|---------|--------------|
| `g128-asym` | `9.4GB` | `2024-10-30` |
| `g32`       | `11GB`  | `2024-10-28` |

```python
from modelscope import snapshot_download
snapshot_download('tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4', cache_dir="本地路径", revision='g128-asym')
snapshot_download('tclf90/rombos-llm-v2.6-qwen-14b-gptq-int4', cache_dir="本地路径", revision='g32')
```

### 【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
2. 有些推理框架的默认`top_k`与`top_p`较大，可以考虑减小对应数值，来获得更合理的模型输出。
3. 根据模型实际情况，可以支持1卡、2卡及4卡的`tensor-parallel-size`启动。


### 【介绍】
Rombos-LLM-V2.6-Qwen-14b is the upgraded version of "rombodawg/Rombos-LLM-V2.5-Qwen-14b". The magic I performed to make this model better than it already was is only known to the Deepest state, dankest memers and God himself, so dont ask 😉. But it does perform a decent bit better than version 2.5 from my hand testing. Benchmarks will come later.

Check out the Continuous Finetuning method that I apply to all my models bellow:

- https://docs.google.com/document/d/1OjbjU5AOz4Ftn9xHQrX3oFQGhQ6RDUuXQipnQ9gn6tU/edit?usp=sharing


### 【模型下载】
```python
from modelscope import snapshot_download
snapshot_download('tclf90/模型名', cache_dir="本地路径", revision='...tag...')
```

### 【高并发RESTFul API推理】
方式1：[vllm](https://github.com/vllm-project/vllm)

方式2：[sglang](https://github.com/sgl-project/sglang)
