---
title: GLM-4.1V-9B-Thinking-8bit
canonical_url: "https://www.modelscope.cn/models/mlx-community/GLM-4.1V-9B-Thinking-8bit"
md_url: "https://www.modelscope.cn/models/mlx-community/GLM-4.1V-9B-Thinking-8bit.md"
repository: mlx-community/GLM-4.1V-9B-Thinking-8bit
last_updated: 2026-08-01
license: mit
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - glm4v
architectures:
  - Glm4vForConditionalGeneration
base_model:
  - zai-org/GLM-4.1V-9B-Thinking
base_model_relation: quantized
parameters: 3.5B
tensor_type:
  - BF16
  - U32
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
  - zh
downloads: 25
stars: 0
tags:
  - reasoning
  - thinking
  - vision-language
  - multimodal
  - mlx
---

# GLM-4.1V-9B-Thinking-8bit

> GLM-4.1V-9B-Thinking-8bit - mlx-community 在 ModelScope 开源的模型。mlx-community/GLM-4.1V-9B-Thinking-8bit

mlx-community/GLM-4.1V-9B-Thinking-8bit 是 ModelScope 魔搭社区上的 3.5B 参数image-text-to-text模型，采用 mit 许可，基于 zai-org/GLM-4.1V-9B-Thinking 构建。

- **Repository**: mlx-community/GLM-4.1V-9B-Thinking-8bit
- **License**: mit
- **Tasks**: image-text-to-text
- **Parameters**: 3.5B
- **Base model**: zai-org/GLM-4.1V-9B-Thinking
- **Tags**: reasoning, thinking, vision-language, multimodal, mlx
- **Downloads**: 25
- **Stars**: 0
- **Last updated**: 2026-08-01

Source: https://www.modelscope.cn/models/mlx-community/GLM-4.1V-9B-Thinking-8bit

---

# mlx-community/GLM-4.1V-9B-Thinking-8bit

[zai-org/GLM-4.1V-9B-Thinking](https://huggingface.co/zai-org/GLM-4.1V-9B-Thinking)
converted to MLX and quantized to **8-bit**, for inference on Apple Silicon.

GLM-4.1V-9B-Thinking is a reasoning vision-language model that emits an explicit
`<think>` block before its answer. **The vision path is preserved** in this
conversion.

Existing MLX conversions of this model are 4-bit and 5-bit; this fills the 8-bit
gap.

## Quantization

| | |
|---|---|
| Requested bits | 8 |
| Group size | 64 |
| Mode | affine |
| **Effective bits per weight** | **9.15** |
| On-disk size | 11 GB |
| Shards | 3 |

Effective bits exceed the requested value because `mlx-vlm` quantizes only the
language model and leaves the **vision tower in bf16** by design — 181 vision
tensors, none of them quantized. The vision encoder is a small share of the
weights but disproportionately sensitive to quantization error.

```
language_model : 242 tensors quantized (8-bit, group size 64)
vision_tower   : 181 tensors, bf16    <- unquantized
```

## Fidelity vs the original weights

Measured against the **bf16 source**, tensor by tensor, over all 242 quantized
tensors (9,399,435,264 parameters). No prompts or sampling — a direct measurement
of how much numerical information the quantization discarded, exact and
hardware-independent.

| Metric | 8-bit |
|---|---|
| Relative L2 error | **0.73%** |
| Cosine similarity | **0.999973** |
| Signal-to-quantization-noise | **42.68 dB** |
| Worst single-element error | 0.005859 |

Highest-error tensors — `v_proj` / `k_proj` and the LM head are the most
sensitive:

```
rel_l2=0.00829  snr= 41.63 dB  language_model.lm_head
rel_l2=0.00815  snr= 41.77 dB  language_model.model.layers.15.self_attn.k_proj
rel_l2=0.00811  snr= 41.82 dB  language_model.model.layers.26.self_attn.v_proj
rel_l2=0.00776  snr= 42.20 dB  language_model.model.layers.32.self_attn.v_proj
rel_l2=0.00774  snr= 42.22 dB  language_model.model.layers.29.self_attn.v_proj
```

42.68 dB is the same figure measured for other 8-bit affine conversions in this
series (e.g. [MagenticBrain-8bit](https://huggingface.co/mlx-community/MagenticBrain-8bit)),
which is what 8-bit at group size 64 should produce.

## Throughput

M2 Pro / 32 GB, 64 generated tokens, greedy.

| Variant | Decode tok/s | Prompt tok/s | Peak RAM |
|---|---|---|---|
| 8-bit | 15.8 | 110.2 | 11.89 GB |

Numbers do not transfer across chips. Note this model emits a `<think>` block, so
end-to-end latency per answer is higher than the token rate alone suggests.

## Why fidelity is reported at the weight level

Other conversions in this series report perplexity ratio, top-1 agreement and KL
divergence against the bf16 source. Those were **run** for this model and are
**deliberately not reported**, because the measurement turned out to be invalid
here — and that is worth stating rather than quietly omitting.

Teacher-forced on five plain-text passages, the numbers came out as perplexity
ratio 3.99, top-1 agreement 0.568, KL 1.71 nats/token. Those would be alarming for
an 8-bit quantization. But the same run also produced BLEU 83.92, chrF 87.67 and
ROUGE-1 0.897 against bf16, and the sampled generations were **byte-identical**.
Both cannot be true.

Breaking the aggregate down by passage explains it:

| Passage | Tokens | bf16 PPL |
|---|---|---|
| prose | 26 | 4.86 |
| **Python code** | **82** | **467.07** |
| legal | 33 | 16.96 |
| biology | 34 | 5.71 |
| meta | 24 | 10.92 |

Aggregate perplexity is `exp(sum_nll / sum_tokens)`, so the code block — 41% of
the tokens, at a perplexity of 467 for the **unquantized** source — dominates
everything. The bf16 reference spans **96x** across these passages. Where the
reference distribution is that flat, quantization noise flips the argmax easily,
so top-1 agreement and KL understate fidelity rather than measuring it.

The weight-level comparison above has no such dependency on prompt choice, so it
is what this card reports. For a model of this class a meaningful behavioural
benchmark would need multimodal reasoning inputs, not plain prose.

## What was not measured

No standard benchmarks: no MMMU, MathVista, MMBench, or any reasoning eval. No
judged quality. The vision path was verified to load and run, not scored on a
dataset. **Evaluate on your own tasks before relying on this in production.**

## Usage

```bash
pip install mlx-vlm
```

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/GLM-4.1V-9B-Thinking-8bit")

prompt = apply_chat_template(
    processor, model.config,
    "What is shown in this image? Reason step by step.",
    num_images=1,
)
out = generate(model, processor, prompt, image=["image.png"], max_tokens=512)
print(out.text)
```

Give it room in `max_tokens` — the model spends tokens inside `<think>` before
answering. Text-only works too: pass `num_images=0` and omit `image`.

Note that stock `mlx-lm` loads the **text path only**; use `mlx-vlm` for image
input.

## Credits

All credit for the model belongs to Zhipu AI / Tsinghua KEG (Z.ai). This is a
format conversion and quantization; no training or fine-tuning was performed.
Licensed MIT, as the original. See the
[original card](https://huggingface.co/zai-org/GLM-4.1V-9B-Thinking) for intended
use and limitations.