---
title: GLM-5-INT4-Pack8
canonical_url: "https://www.modelscope.cn/models/Chranos/GLM-5-INT4-Pack8"
md_url: "https://www.modelscope.cn/models/Chranos/GLM-5-INT4-Pack8.md"
repository: Chranos/GLM-5-INT4-Pack8
last_updated: 2026-03-05
model_type:
  - deepseek_v32
architectures:
  - DeepseekV32ForCausalLM
parameters: 386.6B
tensor_type:
  - I8
  - BF16
  - F32
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
downloads: 42
stars: 2
---

# GLM-5-INT4-Pack8

> GLM-5-INT4-Pack8 - Chranos 在 ModelScope 开源的模型。基于ZhipuAI/GLM-5的INT4-Pack8量化版本，经过天垓150 16卡并行测试。

- **Repository**: Chranos/GLM-5-INT4-Pack8
- **Parameters**: 386.6B
- **Downloads**: 42
- **Stars**: 2
- **Last updated**: 2026-03-05

Source: https://www.modelscope.cn/models/Chranos/GLM-5-INT4-Pack8

---

# GLM-5-INT4-Pack8

## 模型介绍

GLM-5-INT4-Pack8 是 GLM-5 模型的 INT4 量化版本，采用 Pack8 量化技术，在保持模型性能的同时显著降低了显存占用和推理延迟。该模型基于 vLLM 推理框架，提供 OpenAI 兼容的 API 服务，经过天垓150 16卡并行测试。

## 模型特性

- **量化方案**: INT4 Pack8 量化，支持 W4A8 优化
- **推理框架**: vLLM（支持流水线并行和张量并行）

## 环境要求

- **推理框架**: vLLM > 0.11.2+corex.4.5.0.rc.1.20260209
- **Python 版本**: 3.8+
- **GPU**: 支持多卡并行部署
- **显存**: 建议单卡 24GB 以上

## 快速开始

### 启动服务

```bash
export VLLM_W8A8_MOE_USE_W4A8=1
export VLLM_MLA_DISABLE=0

vllm serve <model_path> \
  --trust-remote-code \
  --pipeline-parallel-size 2 \
  --tensor-parallel-size 8 \
  --max-model-len 16384 \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.90
```

### API 调用示例

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model_path>",
    "messages": [
      {
        "role": "user",
        "content": "介绍一下李白"
      }
    ],
    "chat_template_kwargs": {
      "thinking": true
    },
    "max_tokens": 512
  }'
```

### Python 调用示例

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="<model_path>",
    messages=[
        {"role": "user", "content": "介绍一下李白"}
    ],
    extra_body={
        "chat_template_kwargs": {
            "thinking": True
        }
    },
    max_tokens=512
)

print(response.choices[0].message.content)
```
