---
title: Meta-Llama-3-8B-Instruct-GPTQ
canonical_url: "https://www.modelscope.cn/models/LLM-Research/Meta-Llama-3-8B-Instruct-GPTQ"
md_url: "https://www.modelscope.cn/models/LLM-Research/Meta-Llama-3-8B-Instruct-GPTQ.md"
repository: LLM-Research/Meta-Llama-3-8B-Instruct-GPTQ
last_updated: 2024-04-21
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - llama
architectures:
  - LlamaForCausalLM
parameters: 2.0B
tensor_type:
  - F16
  - I32
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 16073
stars: 1
tags:
  - finetuned
  - quantized
  - 4-bit
  - gptq
  - transformers
  - safetensors
  - llama
  - text-generation
  - facebook
  - meta
  - pytorch
  - llama-3
  - conversational
  - en
  - "license:other"
  - autotrain_compatible
  - endpoints_compatible
  - has_space
  - text-generation-inference
  - "region:us"
---

# Meta-Llama-3-8B-Instruct-GPTQ

> Meta-Llama-3-8B-Instruct-GPTQ - LLM-Research 在 ModelScope 开源的模型。Description MaziyarPanahi/Meta-Llama-3-8B-Instruct-GPTQ is a quantized (GPTQ) version of meta-llama/Meta-Llama-3-8B-Instruct

LLM-Research/Meta-Llama-3-8B-Instruct-GPTQ 是 ModelScope 魔搭社区上的 2.0B 参数text-generation模型，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: LLM-Research/Meta-Llama-3-8B-Instruct-GPTQ
- **Tasks**: text-generation
- **Parameters**: 2.0B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: finetuned, quantized, 4-bit, gptq, transformers, safetensors, llama, text-generation, facebook, meta, pytorch, llama-3, conversational, en, license:other, autotrain_compatible, endpoints_compatible, has_space, text-generation-inference, region:us
- **Downloads**: 16073
- **Stars**: 1
- **Last updated**: 2024-04-21

Source: https://www.modelscope.cn/models/LLM-Research/Meta-Llama-3-8B-Instruct-GPTQ

---

# Description
[MaziyarPanahi/Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/MaziyarPanahi/Meta-Llama-3-8B-Instruct-GPTQ) is a quantized (GPTQ) version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)

## How to use
### Install the necessary packages

```
pip install --upgrade accelerate auto-gptq transformers
```

### Example Python code


```python
from transformers import AutoTokenizer, pipeline
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import torch

model_id = "MaziyarPanahi/Meta-Llama-3-8B-Instruct-GPTQ"

quantize_config = BaseQuantizeConfig(
        bits=4,
        group_size=128,
        desc_act=False
    )

model = AutoGPTQForCausalLM.from_quantized(
        model_id,
        use_safetensors=True,
        device="cuda:0",
        quantize_config=quantize_config)

tokenizer = AutoTokenizer.from_pretrained(model_id)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
    repetition_penalty=1.1
)

outputs = pipe("What is a large language model?")
print(outputs[0]["generated_text"])
```