---
title: Llama-2-7b-AQLM-2Bit-2x8-hf
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/Llama-2-7b-AQLM-2Bit-2x8-hf"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/Llama-2-7b-AQLM-2Bit-2x8-hf.md"
repository: AI-ModelScope/Llama-2-7b-AQLM-2Bit-2x8-hf
last_updated: 2024-03-11
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - llama
architectures:
  - LlamaForCausalLM
parameters: 1.9B
tensor_type:
  - I8
  - F16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 4981
stars: 0
---

# Llama-2-7b-AQLM-2Bit-2x8-hf

> Llama-2-7b-AQLM-2Bit-2x8-hf - AI-ModelScope 在 ModelScope 开源的模型。Official AQLM quantization of meta-llama/Llama-2-7b-hf.

AI-ModelScope/Llama-2-7b-AQLM-2Bit-2x8-hf 是 ModelScope 魔搭社区上的 1.9B 参数text-generation模型，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: AI-ModelScope/Llama-2-7b-AQLM-2Bit-2x8-hf
- **Tasks**: text-generation
- **Parameters**: 1.9B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Downloads**: 4981
- **Stars**: 0
- **Last updated**: 2024-03-11

Source: https://www.modelscope.cn/models/AI-ModelScope/Llama-2-7b-AQLM-2Bit-2x8-hf

---

Official [AQLM](https://arxiv.org/abs/2401.06118) quantization of `meta-llama/Llama-2-7b-hf`.

For this quantization, we used 2 codebooks of 8 bits.

Selected evaluation results for this and other models: 

| Model      | AQLM scheme | WikiText 2 PPL | Model size, Gb | Hub link                                                                 |
|------------|-------------|----------------|----------------|--------------------------------------------------------------------------|
| Llama-2-7b | 1x16        | 5.92          | 2.4            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf) |
| Llama-2-7b (THIS) | 2x8         | 6.69          | 2.2            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf)  |
| Llama-2-7b | 8x8         | 6.61          | 2.2            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf)  |
| Llama-2-13b| 1x16        | 5.22           | 4.1            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf)|
| Llama-2-70b| 1x16        | 3.83           | 18.8           | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf)|
| Llama-2-70b| 2x8         | 4.21           | 18.2           | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf) |
| Mixtral-8x7b| 1x16       | 3.35           | 12.6            | [Link](https://huggingface.co/ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf)|
| Mixtral-8x7b-Instruct| 1x16       | -           | 12.6            | [Link](https://huggingface.co/ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf)|

**UPD** (20.02.2024).
We applied global finetuning on top of quantized model and improved results compared to first revision.

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the [official GitHub repo](https://github.com/Vahe1994/AQLM).