--- title: Inkling-mlx-2bit canonical_url: "https://www.modelscope.cn/models/mlx-community/Inkling-mlx-2bit" md_url: "https://www.modelscope.cn/models/mlx-community/Inkling-mlx-2bit.md" repository: mlx-community/Inkling-mlx-2bit last_updated: 2026-07-25 license: apache-2.0 pipeline_tag: image-text-to-text tasks: - image-text-to-text model_type: - inkling_mm_model architectures: - InklingForConditionalGeneration base_model: - thinkingmachines/Inkling base_model_relation: finetune parameters: 98.9B tensor_type: - BF16 - U32 - F32 library_name: - transformer - mlx - safetensors downloads: 252 stars: 0 tags: - mlx - inkling - moe - multimodal - text-generation - image-text-to-text --- # Inkling-mlx-2bit > Inkling-mlx-2bit - mlx-community 在 ModelScope 开源的模型。Inkling-mlx-2bit (2-bit, omni / multimodal, self-contained MLX) mlx-community/Inkling-mlx-2bit 是 ModelScope 魔搭社区上的 98.9B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 thinkingmachines/Inkling 构建。 - **Repository**: mlx-community/Inkling-mlx-2bit - **License**: apache-2.0 - **Tasks**: image-text-to-text - **Parameters**: 98.9B - **Base model**: thinkingmachines/Inkling - **Tags**: mlx, inkling, moe, multimodal, text-generation, image-text-to-text - **Downloads**: 252 - **Stars**: 0 - **Last updated**: 2026-07-25 Source: https://www.modelscope.cn/models/mlx-community/Inkling-mlx-2bit --- # Inkling-mlx-2bit (2-bit, omni / multimodal, self-contained MLX) An **MLX 2-bit** build of Thinking Machines' **Inkling** (975B-total / 41B-active MoE), quantized from the BF16 checkpoint. **Omni**: keeps the text decoder **plus the vision (HMLP) and audio (dMel) towers**. **Self-contained**. We put this repo bundles the loader package (`inkling_mlx/`), so it runs with just `mlx` + `mlx-lm` + `transformers`. > Community build for large-model-on-Apple-Silicon experiments. ~**315 GB** in 2-bit -> > doesn't fit a single 192 GB Mac *resident*, so run it via multi-Mac (distributed MLX) or > **SSD expert-offload** (see below — one 192 GB Mac, or even a 64 GB Mac slowly). ## Quantization (recipe: experts_only) - **2-bit** (affine int4-style group quant, group size 64) on the **routed experts** and the **vision/audio matmuls** (the bulk of the weights). - **Kept bf16 (protected):** attention, token/output embeddings, RMSNorms, the router gate, the per-layer short-convolutions, and the relative-position bias. ## Usage (self-contained) ```python # pip install mlx mlx-lm transformers ; download this repo (it includes inkling_mlx/) from inkling_mlx.load import load from inkling_mlx.generate import greedy_generate from transformers import AutoTokenizer model, config = load("./Inkling-mlx-2bit") tok = AutoTokenizer.from_pretrained("./Inkling-mlx-2bit", trust_remote_code=True) ids = tok("The capital of France is")["input_ids"] print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64))) ``` ## Run on a single Mac with SSD expert-offload - See image of SSD expert-offload in: [github.com/huckiyang/mlx-moe-offload](https://github.com/huckiyang/mlx-moe-offload) ~315 GB doesn't fit a 192 GB Mac resident but an MoE only fires **6 of 256** experts per token, so you can keep the always-needed weights in RAM (attention, shared experts, embeddings, norms, router, vision/audio towers ≈ 35 GB) and **page the routed experts from SSD on demand**, letting the OS page cache hold the hot ones. This makes the 2-bit omni build runnable on one 192 GB Mac Studio (and even generates on a 64 GB Mac, slowly). Tool: **[github.com/huckiyang/mlx-moe-offload](https://github.com/huckiyang/mlx-moe-offload)** (MIT) — a drop-in over mlx-lm's `SwitchGLU`. It also answers the mlx-lm feature request [ml-explore/mlx-lm#1438](https://github.com/ml-explore/mlx-lm/issues/1438). ```bash pip install mlx mlx-lm scipy pillow git clone https://github.com/huckiyang/mlx-moe-offload && cd mlx-moe-offload && pip install -e . # 1) one-time: repack the stacked experts into a per-expert SSD store # (copies the bundled inkling_mlx/ loader in, so the offload dir is self-contained) python -m mlx_moe_offload.repack --build /path/Inkling-mlx-2bit --out /path/Inkling-mlx-2bit-offload OFF=/path/Inkling-mlx-2bit-offload # 2) generate — text, image, or audio (omni) python examples/inkling_omni.py --offload-dir $OFF --prompt "The capital of France is" python examples/inkling_omni.py --offload-dir $OFF --image cat.png --prompt "What is in this image?" python examples/inkling_omni.py --offload-dir $OFF --audio q.wav --prompt "Transcribe in English:" ``` The prompt is prefilled in small `--prefill-chunk` blocks so only a bounded set of experts is live at once (a lazy full-prompt prefill would page ~all experts per layer and OOM). Decode speed is gated by the routing hit-rate + SSD bandwidth — check `store.stats()`. At 2-bit, text is solid (*"…capital of Japan is Tokyo"*) while image/audio are recognizable but degraded by quantization; a higher-bit build sharpens multimodal quality. ## Sample decoding (on-device, 2-bit + SSD expert-offload, M4 Max 64 GB) Run entirely on a single 64 GB Mac with the routed experts paged from SSD: | Modality | Input | Output | |---|---|---| | Text | `The capital of Japan is` | `Tokyo, but it is not` | | Audio | [`inkling_mlx/test-inkling.wav`](inkling_mlx/test-inkling.wav) — *"I love sushi and hamburger very much."* | `I love sushi and hamburger very much` ✅ | The audio example transcribes clean English speech **verbatim** — the dMel audio tower + text backbone recover the sentence exactly, even at 2-bit. (Harder / accented clips can drift toward phonetically-similar wordings at 2-bit; a higher-bit build tightens this.) Throughput is SSD-bound: on this 64 GB Mac (cold cache, expert LRU off) decode runs roughly 0.03–0.4 tok/s, with most of the wall-clock spent in a one-time prefill as the selected experts stream from disk. A 192 GB Mac Studio keeps about half the experts resident in the page cache and should reach the ~6–15 tok/s range. ## Attribution - Weights converted from `thinkingmachines/Inkling`. - We made a revised bundled `inkling_mlx/` loader is vendored from [PipeNetwork/inkling-mlx](https://github.com/PipeNetwork/inkling-mlx) (**Apache-2.0**, Copyright 2026 PipeNetwork) and [David](https://github.com/davidrhodus). LICENSE + THIRD_PARTY_NOTICES.md are included.