---
title: Astrea-R8-Chat-9B-OptiQ-4bit
canonical_url: "https://www.modelscope.cn/models/mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit"
md_url: "https://www.modelscope.cn/models/mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit.md"
repository: mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit
last_updated: 2026-10-23
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - qwen3_5
architectures:
  - Qwen3_5ForConditionalGeneration
base_model:
  - HemmingwayAI/Astrea-R8-Chat-9B
base_model_relation: quantized
parameters: 2.4B
tensor_type:
  - BF16
  - U32
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 18
stars: 0
tags:
  - mlx
  - optiq
  - quantized
  - mixed-precision
  - 4bit
  - 8bit
  - qwen3.5
  - chat
  - creative-writing
  - vision-language-model
  - image-text-to-text
  - apple-silicon
---

# Astrea-R8-Chat-9B-OptiQ-4bit

> Astrea-R8-Chat-9B-OptiQ-4bit - mlx-community 在 ModelScope 开源的模型。mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit

mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit 是 ModelScope 魔搭社区上的 2.4B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 HemmingwayAI/Astrea-R8-Chat-9B 构建。

- **Repository**: mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Parameters**: 2.4B
- **Base model**: HemmingwayAI/Astrea-R8-Chat-9B
- **Tags**: mlx, optiq, quantized, mixed-precision, 4bit, 8bit, qwen3.5, chat, creative-writing, vision-language-model, image-text-to-text, apple-silicon
- **Downloads**: 18
- **Stars**: 0
- **Last updated**: 2026-10-23

Source: https://www.modelscope.cn/models/mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit

---

# mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit

> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)

**Altworld's creative-writing model, quantized for a Mac.** A mixed-precision MLX quant of [HemmingwayAI/Astrea-R8-Chat-9B](https://huggingface.co/HemmingwayAI/Astrea-R8-Chat-9B), a Qwen3.5-9B fine-tune built for chat and fiction. 8.0 GB on disk, down from 18.8 GB bf16. See the [original card](https://huggingface.co/HemmingwayAI/Astrea-R8-Chat-9B) for what the model is trained to do and how Altworld evaluates it.

OptiQ quantizes the language tower to mixed 4/8-bit and keeps the vision tower at bf16 in a sidecar, so one checkpoint does both text and images.

## What it is

| Property | Value |
|---|---|
| Base | HemmingwayAI/Astrea-R8-Chat-9B (Qwen3.5-9B fine-tune) |
| Language tower | per-layer 4/8-bit: 116 layers at 4-bit, 134 at 8-bit (250 total) |
| Vision tower | bf16, kept in `optiq_vision.safetensors` (333 tensors) |
| On disk | 7.10 GB language + 0.91 GB vision |
| Group size | 64 |
| Context | 262,144 tokens |

Following `llama.cpp`'s naming for mixed quants, the "4bit" label denotes the predominant precision, not the weighted average.

## How the bit-widths were chosen

The per-layer allocation was not measured on this model. It was transferred from [mlx-community/Qwen3.5-9B-OptiQ-4bit](https://huggingface.co/mlx-community/Qwen3.5-9B-OptiQ-4bit), whose allocation came from a KL-divergence sensitivity sweep over a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix). The two language towers share an architecture exactly, and all **250 tensors matched with none unmatched**.

## Checked after conversion

Run through this quant and its parent, Qwen3.5-9B-OptiQ-4bit, on Apple Silicon with greedy decoding and thinking off:

| Prompt | Astrea | Parent |
|---|---|---|
| Capital of France, one word | Paris | Paris |
| 120 km in 1.5 h, speed in km/h | 80 | 80 |
| `is_palindrome(s)` ignoring case and non-letters | correct function | correct function |
| What makes a good short-story opening, two sentences | coherent answer in its own wording | coherent answer |

Image input: shown a red rectangle, a blue circle and the number 42, it answered "The image contains a red square, a blue circle, and the number 42 written in black."

## Serve it with n-gram drafting

Measured on a 24 GB M4, batch 1, greedy, median of three runs each:

| Workload | Plain | `--ngram-draft 16` |
|---|---|---|
| Edit (rename a parameter across a file and return it) | 16.6 tok/s | **44.9 tok/s (2.7x)** |
| Code generation | 16.7 tok/s | 18.0 tok/s (1.08x) |
| Prose from a 1,024-token prompt | 16.6 tok/s | 16.2 tok/s (0.98x) |

```bash
pip install mlx-optiq
optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16
```

This checkpoint has no MTP head, so `--mtp` does not apply.

## Use it

```bash
# chat, images, and every app that talks to optiq serve
optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16
optiq lab --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit
```

Text only, with stock mlx-lm:

```python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit")
messages = [{"role": "user", "content": "Open a short story set on a night ferry."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=400))
```