---
title: Inkling-mlx-4bit
canonical_url: "https://www.modelscope.cn/models/mlx-community/Inkling-mlx-4bit"
md_url: "https://www.modelscope.cn/models/mlx-community/Inkling-mlx-4bit.md"
repository: mlx-community/Inkling-mlx-4bit
last_updated: 2026-07-26
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - inkling_mm_model
architectures:
  - InklingForConditionalGeneration
base_model:
  - thinkingmachines/Inkling
base_model_relation: finetune
parameters: 157.4B
tensor_type:
  - F32
  - BF16
  - U32
library_name:
  - transformer
  - mlx
  - safetensors
downloads: 48
stars: 0
tags:
  - mlx
  - inkling
  - moe
  - multimodal
  - text-generation
  - image-text-to-text
---

# Inkling-mlx-4bit

> Inkling-mlx-4bit - mlx-community 在 ModelScope 开源的模型。Inkling-mlx-4bit (4-bit, omni / multimodal, self-contained MLX)

mlx-community/Inkling-mlx-4bit 是 ModelScope 魔搭社区上的 157.4B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 thinkingmachines/Inkling 构建。

- **Repository**: mlx-community/Inkling-mlx-4bit
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Parameters**: 157.4B
- **Base model**: thinkingmachines/Inkling
- **Tags**: mlx, inkling, moe, multimodal, text-generation, image-text-to-text
- **Downloads**: 48
- **Stars**: 0
- **Last updated**: 2026-07-26

Source: https://www.modelscope.cn/models/mlx-community/Inkling-mlx-4bit

---

# Inkling-mlx-4bit (4-bit, omni / multimodal, self-contained MLX)

An **MLX 4-bit** build of Thinking Machines' **Inkling** (975B-total / 41B-active MoE),
quantized from the BF16 checkpoint for **Apple Silicon**. **Omni**: keeps the text decoder
**plus the vision (HMLP) and audio (dMel) towers**. **Self-contained**. This bundles the loader
package (`inkling_mlx/`), so it runs with just `mlx` + `mlx-lm` + `transformers`.

The higher-fidelity sibling of [`mlx-community/Inkling-mlx-2bit`](https://huggingface.co/mlx-community/Inkling-mlx-2bit):
~**548 GB** (vs ~315 GB at 2-bit), trading size for sharper text and multimodal quality.

## Quantization (recipe: `experts_only`)

- **4-bit** affine group quantization (group size 64) on the **routed experts** and the
  **vision / audio matmuls** this is the bulk of the weights.
- **Kept in bf16 (protected):** attention, token / output embeddings, RMSNorms, the router
  gate, the per-layer short-convolutions, and the relative-position bias.

## Usage

```python
# pip install mlx mlx-lm transformers
# download this repo (it already includes the inkling_mlx/ loader)
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

model, config = load("./Inkling-mlx-4bit")
tok = AutoTokenizer.from_pretrained("./Inkling-mlx-4bit", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
```

## Running on a single Mac with SSD expert-offload

At ~548 GB this doesn't fit resident on one Mac but an MoE only fires **6 of 256** experts
per token. So keep the always-needed weights in RAM (attention, shared experts, embeddings,
norms, router, vision / audio towers) and **page the routed experts from SSD on demand**,
letting the OS page cache hold the hot ones. This runs the omni build on a single Mac Studio.

## Sample text decoding (4-bit, greedy)

  | Prompt | Generated continuation | Notes |
  |---|---|---|
  | `The capital of France is` | ` Paris. The capital of Italy is Rome. The capital of Spain is Madrid. The capital of Russia is Moscow.` | ~0.33 tok/s, SSD expert-offload |

Tool: **[github.com/huckiyang/mlx-moe-offload](https://github.com/huckiyang/mlx-moe-offload)**
(MIT) — a drop-in over mlx-lm's `SwitchGLU`.

```bash
pip install mlx mlx-lm scipy pillow
git clone https://github.com/huckiyang/mlx-moe-offload && cd mlx-moe-offload && pip install -e .

# 1) one-time: repack the stacked experts into a per-expert SSD store
python -m mlx_moe_offload.repack --build /path/Inkling-mlx-4bit --out /path/Inkling-mlx-4bit-offload
OFF=/path/Inkling-mlx-4bit-offload

# 2) generate — text, image, or audio (omni)
python examples/inkling_omni.py --offload-dir $OFF --prompt "The capital of France is"
python examples/inkling_omni.py --offload-dir $OFF --image cat.png --prompt "What is in this image?"
python examples/inkling_omni.py --offload-dir $OFF --audio q.wav   --prompt "Transcribe in English:"
```

Decode speed is gated by the routing cache hit-rate and SSD bandwidth, please check `store.stats()`.

## Attribution

- mlx-moe-offload from [huckiyang/mlx-moe-offload](https://github.com/huckiyang/mlx-moe-offload)
- Weights converted from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).
- The bundled `inkling_mlx/` loader is vendored from
  [PipeNetwork/inkling-mlx](https://github.com/PipeNetwork/inkling-mlx) (**Apache-2.0**,
  Copyright 2026 PipeNetwork) and [David](https://github.com/davidrhodus).

`LICENSE` + `THIRD_PARTY_NOTICES.md` are included in this repo.