--- title: openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 canonical_url: "https://www.modelscope.cn/models/tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3" md_url: "https://www.modelscope.cn/models/tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3.md" repository: tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 chinese_name: "openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3-量化修复" last_updated: 2024-05-26 license: other pipeline_tag: text-generation tasks: - text-generation model_type: - llama architectures: - LlamaForCausalLM parameters: 9.2B tensor_type: - I32 - F16 library_name: - pytorch - transformer - safetensors frameworks: - Pytorch inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 102 stars: 1 tags: - openbuddy - gptq - int3 - "量化修复" - vLLM --- # openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 > openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 - tclf90 在 ModelScope 开源的模型。openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3-量化修复 原模型 OpenBuddy/openbuddy-llama3-70b-v21.1-8k tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 是 ModelScope 魔搭社区上的 9.2B 参数text-generation模型，采用 other 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 - **License**: other - **Tasks**: text-generation - **Parameters**: 9.2B - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2 - **Tags**: openbuddy, gptq, int3, 量化修复, vLLM - **Downloads**: 102 - **Stars**: 1 - **Last updated**: 2024-05-26 Source: https://www.modelscope.cn/models/tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3 --- # openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3-量化修复 原模型 [OpenBuddy/openbuddy-llama3-70b-v21.1-8k](https://www.modelscope.cn/models/OpenBuddy/openbuddy-llama3-70b-v21.1-8k) ### 【模型更新日期】 ``` 2024-05-25 ``` ### 【模型大小】 `31.5GB` ### 【量化修复】 调优了现有 `AWQ` 与 `GPTQ` 量化算法的量化策略。带有`量化修复`标签的`Int3`模型，可以比肩默认`AWQ`与`GPTQ`算法的`Int8`模型的能力。 1. 量化修复可以极大减少模型的`1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等量化损失造成的模型不可用的情况。 2. 调优后的量化模型，`AWQ`与`GPTQ`模型在能力上没有表现出明显区别。同时考虑到`GPTQ`的`vLLM`引擎的并发推理效率最好，所以不再制作`AWQ`模型。 3. 调优后的量化模型，`int4`与`int3`模型在能力上没有表现出明显区别，所以也不再制作`int4`模型。 ### 【同期量化修复模型】 | 模型名称 | 磁盘大小（GB） | |---------------------------------------------------------------------------------------------------------------------------------|----------| | [零一万物-1.5-6B-Chat-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Yi-1.5-6B-Chat-GPTQ-Int3) | 3.3 | | [零一万物-1.5-9B-Chat-16K-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Yi-1.5-9B-Chat-16K-GPTQ-Int3) | 4.4 | | [零一万物-1.5-34B-Chat-16K-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Yi-1.5-34B-Chat-16K-GPTQ-Int3) | 15.1 | | [通义千问1.5-7B-Chat-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Qwen1.5-7B-Chat-GPTQ-Int3) | 5.1 | | [通义千问1.5-14B-Chat-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Qwen1.5-14B-Chat-GPTQ-Int3) | 8.1 | | [通义千问1.5-32B-Chat-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3) | 15.4 | | [通义千问1.5-72B-Chat-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Qwen1.5-72B-Chat-GPTQ-Int3) | 32.5 | | [通义千问1.5-110B-Chat-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/Qwen1.5-110B-Chat-GPTQ-Int3) | 47.9 | | [openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3-量化修复](https://www.modelscope.cn/models/tclf90/openbuddy-llama3-70b-v21.1-8k-GPTQ-Int3) | 31.5 | ### 【模型下载】 ```python from modelscope import snapshot_download model_dir = snapshot_download('tclf90/模型名', cache_dir="本地路径") ``` ### 【[vLLM](https://github.com/vllm-project/vllm)推理（目前仅限Linux）】 #### 1. Python 简易调试 ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams max_model_len, tp_size = 4000, 1 model_name = "本地路径/tclf90/模型名称" # 例："./my_models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3" model_name = model_name.replace('.', '___') tokenizer = AutoTokenizer.from_pretrained(model_name) llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True, enforce_eager=True) sampling_params = SamplingParams(temperature=0.7, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id]) messages_list = [ [{"role": "user", "content": "你是谁"}], [{"role": "user", "content": "介绍一下你自己"}], [{"role": "user", "content": "用python写一个快排函数"}], ] prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list] outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params) generated_text = [output.outputs[0].text for output in outputs] print(generated_text) ``` #### 2. 类ChatGPT RESTFul API Server ``` >>> python -m vllm.entrypoints.openai.api_server --model tclf90/模型名称 ``` ### 【Transformer推理】 ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig model_name = "本地路径/tclf90/模型名称" # 例："./my_models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3" model_name = model_name.replace('.', '___') tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda() model.generation_config = GenerationConfig.from_pretrained(model_name) model.generation_config.pad_token_id = model.generation_config.eos_token_id messages = [ {"role": "user", "content": "你好你是谁"} ] input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") outputs = model.generate(input_tensor.to(model.device), max_new_tokens=100) result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True) print(result) ```