---
title: AquilaChat2-34B-Int4-GPTQ
canonical_url: "https://www.modelscope.cn/models/BAAI/AquilaChat2-34B-Int4-GPTQ"
md_url: "https://www.modelscope.cn/models/BAAI/AquilaChat2-34B-Int4-GPTQ.md"
repository: BAAI/AquilaChat2-34B-Int4-GPTQ
last_updated: 2023-11-15
license: other
model_type:
  - llama
architectures:
  - LlamaForCausalLM
parameters: 5.6B
tensor_type:
  - I32
  - F16
  - BF16
library_name:
  - transformer
  - safetensors
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 409
stars: 1
---

# AquilaChat2-34B-Int4-GPTQ

> AquilaChat2-34B-Int4-GPTQ - BAAI 在 ModelScope 开源的模型。We opensource gptq format of Int4-quantized AquilaChat2-34B model, which can be used for quick downloading and usage.

BAAI/AquilaChat2-34B-Int4-GPTQ 是 ModelScope 魔搭社区上的 5.6B 参数机器学习模型，采用 other 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: BAAI/AquilaChat2-34B-Int4-GPTQ
- **License**: other
- **Parameters**: 5.6B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Downloads**: 409
- **Stars**: 1
- **Last updated**: 2023-11-15

Source: https://www.modelscope.cn/models/BAAI/AquilaChat2-34B-Int4-GPTQ

---

<p align="center">
  <a href="https://github.com/FlagAI-Open/Aquila2" target="_blank">Github</a> • <a href="https://github.com/FlagAI-Open/Aquila2/blob/main/assets/wechat-qrcode.jpg" target="_blank">WeChat</a> <br>
</p>

We opensource gptq format of Int4-quantized  **AquilaChat2-34B** model, which can be used for quick downloading and usage.

我们开源了用gptq做4比特量化的**AquilaChat2-34B** 模型，可以更快的下载和使用。


## Quick Start  快速上手 AquilaChat2-34B-Int4-GPTQ


### 1. Environment setup


Follow the instructions in https://github.com/PanQiWei/AutoGPTQ/tree/main#quick-installation to install Auto-GPTQ

按照https://github.com/PanQiWei/AutoGPTQ/tree/main#quick-installation里的指示安装Auto-GPTQ

### 2. Inference 模型推理

```Python
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM


# pretrained_model_dir = "/share/project/ldwang/checkpoints/Aquila-33b-knowledge6-341000-sft-v0.9.16/iter_0004000_hf"
model_dir = "./checkpoints/Aquilachat34b-4bit" # 模型路径
device="cuda:0"

tokenizer = AutoTokenizer.from_pretrained(model_dir, use_fast=True,trust_remote_code=True)
model = AutoGPTQForCausalLM.from_quantized(model_dir, inject_fused_attention=False, low_cpu_mem_usage=True, device=device)


model.eval()
import time 
texts = ["请给出10个要到北京旅游的理由。",
         "写一个林黛玉倒拔垂杨柳的故事",
         "write a poet about moon"]
from predict import predict
start_time = time.time()
for text in texts:
    out = predict(model, text, tokenizer=tokenizer, max_gen_len=200, top_p=0.95,
                seed=1234, topk=200, temperature=1.0, sft=True, device=device,
                model_name="AquilaChat2-34B")
print(out)
print(f"Elapsed time model loading: {time.time()-start_time} seconds")
```



## License

Aquila2 series open-source model is licensed under [ BAAI Aquila Model Licence Agreement](https://huggingface.co/BAAI/AquilaChat2-34B/blob/main/BAAI-Aquila-Model-License%20-Agreement.pdf)