---
title: Qwen2.5-72B-Instruct-GGUF-V3-LOT
canonical_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT"
md_url: "https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT.md"
repository: okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT
chinese_name: "通义千问Qwen2.5-72B-Instruct-GGUF-V3-LOT高精量化"
last_updated: 2025-01-05
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
library_name:
  - gguf
language:
  - en
  - zh
downloads: 772
stars: 4
tags:
  - GGUF
  - Chat
  - Instruct
  - gguf
---

# Qwen2.5-72B-Instruct-GGUF-V3-LOT

> Qwen2.5-72B-Instruct-GGUF-V3-LOT - okwinds 在 ModelScope 开源的模型。通义千问Qwen2.5-72B-Instruct-GGUF-V3-LOT

okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT 是 ModelScope 魔搭社区上的text-generation模型，采用 apache-2.0 许可。

- **Repository**: okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT
- **License**: apache-2.0
- **Tasks**: text-generation
- **Tags**: GGUF, Chat, Instruct, gguf
- **Downloads**: 772
- **Stars**: 4
- **Last updated**: 2025-01-05

Source: https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT

---

# 通义千问Qwen2.5-72B-Instruct-GGUF-V3-LOT

原模型 [qwen/Qwen2.5-72B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-72B-Instruct)

官方GGUF仓库 [Qwen/Qwen2.5-72B-Instruct-GGUF](https://www.modelscope.cn/models/qwen/qwen2.5-72B-instruct-gguf)

## Qwen2.5 同系列模型

#### Finetuned Model

- **Model tree** </br>
	&nbsp;Base model - [Qwen/Qwen2.5-32B](https://www.modelscope.cn/models/Qwen/Qwen2.5-32B) </br>
		&nbsp; <span style="font-weight: bold;">╰</span>─  Finetuned - [Qwen/Qwen2.5-32B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen2.5-32B-Instruct) </br>
		&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span style="font-weight: bold;"> ├</span>─  Finetuned -  [CompassJudger-1](https://www.modelscope.cn/collections/CompassJudger-1-Int-and-FP8---Mixed-Precision-85d756858db345) </br>
		&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span style="font-weight: bold;">└</span>─  Finetuned -  [Rombos-LLM-Qwen2.5](https://www.modelscope.cn/collections/Rombos-LLM-Qwen25-4dc8690ad7f541) </br>

#### 同系量化

| **Model**    | GPTQ Compression & FP8                                                                                                                                                                                                                                                                                                                                                                                                     | GGUF                                                                                         | **MLX** (macOS)                                                                                                                                                                                                                                                              |
| ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *3B params*  | -                                                                                                                                                                                                                                                                                                                                                                                                                          | [3B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-GGUF-V3-LOT)   | [3B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-4bit)<br/>[3B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-8bit)                                                                                                |
| *7B params*  | [7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int4-W4A16)<br/>[7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int8-W8A16)<br/>[7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-FP8-A16) <br/>[Coder-7B-Instruct-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8) | [7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-GGUF-V3-LOT)   | [7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-4bit)<br/>[7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-8bit)                                                                                                |
| *Coder 7B params* | [Coder-7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int4-W4A16)<br/>[Coder-7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int8-W8A16) <br/>[Coder-7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8-A16)       | [Coder-7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-GGUF-V3-LOT) | [Coder-7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-4bit)<br/>[Coder-7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-8bit)   |
| *14B params* | [14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16)<br/>[14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int8-W8A16)<br/>[14B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-FP8-A16)                                                                                                                                         | [14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-GGUF-V3-LOT) | [14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-4bit)<br/>[14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-8bit)                                                                                            |
| *Coder 14B params* | [Coder-14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int4-W4A16)<br/>[Coder-14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int8-W8A16)                                                                                                                                        | [Coder-14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-GGUF-V3-LOT) | [Coder-14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-4bit)<br/>[Coder-14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-8bit)  |
| *32B params* | [32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16)<br/>[32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16)<br/>[32B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16)                                                                                                                                                                                                                                        | [32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-GGUF-V3-LOT) | [32B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-2bit)<br/>[32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-4bit)<br/>[32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-8bit) |
| *Coder 32B params* | [Coder-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-Int4-W4A16)<br/>[Coder-32B-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-FP8)                                                                                                                                                                                                                                        | [Coder-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-GGUF-V3-LOT) | [Coder-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-4bit)<br/>[Coder-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-8bit) |
| *QwQ-Preview 32B params* | [QwQ-32B-Preview-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-Int4-W4A16)<br/>[QwQ-32B-Preview-FP8](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-FP8)    | [QwQ-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-GGUF-V3-LOT) | [QwQ-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-4bit)<br/>[QwQ-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-8bit) |
| *72B params* | -                                                                                                                                                                                                                                                                                                                                                                                                                          | [72B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT) | [72B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-2bit)<br/>[72B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-4bit)<br/>[72B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-8bit) |

## Quickstart

建议参考千问官方的[llama.cpp](https://github.com/ggerganov/llama.cpp)文档

#### 手动下载

由于克隆整个存储库可能效率低下，因此你可以手动下载所需的 GGUF 文件或使用 modelscope：

1. 安装
   
   ```shell
   pip install -U modelscope
   ```

2. 下载
   
   ```shell
   modelscope download --model=okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT --include "qwen2.5-72B-it-Q2_K-LOT.gguf" --local_dir .
   ```
#### <span style="color:#ff6600">使用 *Ollama*</span>

如果机器上安装了 [Ollama](https://ollama.com/)，还可以通过 [Ollama](https://ollama.com/) 拉取仓库中的模型。如下示例：

- 使用仓库的默认地址会拉取 Q4_K_M 精度的 GGUF 模型

```shell
# 仅拉取模型
ollama pull modelscope.cn/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT

# 拉取模型以后立即运行、推理模型
ollama run modelscope.cn/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT
```

- 拉取指定精度的模型

```shell
# 拉取 Q8_0 精度的 GGUF 模型
ollama pull modelscope.cn/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT:Q2_K
```

## 模型简介

通义千问Qwen2.5-72B-Instruct-GGUF-V3-LOT 是一个基于 通义千问2.5-72B-Instruct 的 GGUF 格式量化模型。

本仓库的GGUF格式模型与官方的不同之处是：

1. 模型权重文件不分片。下载单文件即可用。

2. 模型在量化过程中<span style="color: red;">保留了 output tensor 的激活值</span>，以提升量化后模型的精度表现。所以相比官方量化的 GGUF 格式模型，本仓库的同量化等级的权重文件，会略大一点，当然，理论上精度也会高一点。

本仓库包含如下量化等级：q2_K、q3_K_M、q4_K_M。

<br/>

<br/>
_______________________________

#### _作者在此 👇🏻 扫一扫_

<img src="https://www.modelscope.cn/models/okwinds/GPT-2/resolve/master/qrcode_for_jcl_258.jpg" />