--- title: Inkling-Small-mlx-4bit canonical_url: "https://www.modelscope.cn/models/mlx-community/Inkling-Small-mlx-4bit" md_url: "https://www.modelscope.cn/models/mlx-community/Inkling-Small-mlx-4bit.md" repository: mlx-community/Inkling-Small-mlx-4bit last_updated: 2026-07-31 license: apache-2.0 pipeline_tag: image-text-to-text tasks: - image-text-to-text model_type: - inkling_mm_model architectures: - InklingForConditionalGeneration base_model: - thinkingmachines/Inkling-Small base_model_relation: finetune parameters: 44.2B tensor_type: - BF16 - U32 - F32 library_name: - transformer - mlx - safetensors - pytorch frameworks: - pytorch downloads: 11139 stars: 0 tags: - mlx - inkling - moe - multimodal - text-generation - image-text-to-text - apple-silicon --- # Inkling-Small-mlx-4bit > Inkling-Small-mlx-4bit - mlx-community 在 ModelScope 开源的模型。Inkling-Small-mlx-4bit mlx-community/Inkling-Small-mlx-4bit 是 ModelScope 魔搭社区上的 44.2B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 thinkingmachines/Inkling-Small 构建。 - **Repository**: mlx-community/Inkling-Small-mlx-4bit - **License**: apache-2.0 - **Tasks**: image-text-to-text - **Parameters**: 44.2B - **Base model**: thinkingmachines/Inkling-Small - **Tags**: mlx, inkling, moe, multimodal, text-generation, image-text-to-text, apple-silicon - **Downloads**: 11139 - **Stars**: 0 - **Last updated**: 2026-07-31 Source: https://www.modelscope.cn/models/mlx-community/Inkling-Small-mlx-4bit --- # Inkling-Small-mlx-4bit An **MLX 4bit** build of [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small) — 264 B total parameters, **~12 B active**; 42 layers, 256 routed experts (top-6) + 2 shared; natively multimodal (image + audio in, text out). Sized to run **fully resident on a 192 GB Mac** — no expert-offload, no expert pruning, no layer streaming. The whole model sits in unified memory and decodes at conversational speed. > Why this model is the interesting one for Apple Silicon: speed follows the **active** > parameter count, not how hard you compress. Inkling-Small activates ~12 B params per token, > so this tier reads roughly 10.8 GB per token. The 975 B Inkling, by contrast, needs SSD > expert-offload even at 2-bit and manages ~0.2–0.4 tok/s. ## Tiers Pick the tier that fits your machine. Sizes are on-disk and **measured from the built artifacts**; peak/speed columns are filled in only where a benchmark has actually run. | tier | target Mac | recipe | on-disk | peak | load | prefill tok/s | |---|---|---|---|---|---|---| | **4bit** ← **this repo** | 192 GB | experts 4-bit, non-expert 8-bit | 153.5 GB | — | — | — | | [3bit](https://huggingface.co/mlx-community/Inkling-Small-mlx-3bit) | 128 GB | experts 3-bit, non-expert 8-bit | 120.9 GB | — | — | — | | [2bit](https://huggingface.co/mlx-community/Inkling-Small-mlx-2bit) | 96 GB | experts 2-bit, non-expert 8-bit | 88.4 GB | — | — | — | > **Speeds are not yet measured.** On-disk sizes above are real (taken from the built artifacts), but load/peak/tok-s columns stay empty until `serving/bench_mlx.py` has run on the target hardware — we would rather ship a blank column than a guess. Run it yourself with the command below and please open a discussion with your numbers. ## Why MLX on Apple Silicon MLX is Apple's array framework for Apple Silicon. For a big sparse MoE like this one, the practical differences from a CPU/GPU-split runtime are: | | what it means here | |---|---| | **Unified memory** | The GPU reads the same DRAM as the CPU, so a 153 GB build needs 153 GB of *system* memory — not VRAM plus a host copy. There is no PCIe transfer per layer, which is what makes a >100 GB model practical on a desktop at all. | | **Lazy, mmap'd loading** | `mx.load` memory-maps safetensors, so weights page in on demand instead of being read and copied up front. | | **Native quantized matmul** | Quantized weights are multiplied in their packed form (`quantized_matmul`, and `gather_qmm` for MoE expert gathers) rather than dequantized to fp16 first — so low-bit tiers save bandwidth at *runtime*, not just on disk. | | **Per-module precision** | `nn.quantize(..., class_predicate=...)` lets one checkpoint mix bit-widths per tensor class, which is exactly how these tiers keep attention and the router high-precision while the experts go low. | | **Small dependency surface** | `pip install mlx mlx-lm` and a single Python model file. No compile step, no separate server binary. | Honest limits: MLX's ecosystem is younger than llama.cpp's, it has no importance-matrix ("i-quant") style calibrated quantization yet, and it runs on Apple Silicon only. If you want a GGUF build instead, one exists at [`unsloth/Inkling-Small-GGUF`](https://huggingface.co/unsloth/Inkling-Small-GGUF). ## Memory notes `peak` (once measured) is peak unified-memory use, not file size — leave headroom for the KV cache and the OS. Two things matter on a machine near its limit: * **Raise the Metal wired limit.** It defaults to ~75% of RAM, which is below this tier's footprint on a 192 GB machine: ```bash sudo sysctl iogpu.wired_limit_mb=180000 # ~180 GB on a 192 GB Mac; scale to your RAM ``` * **macOS will kill a process that stays too large for too long** (jetsam), even while memory pressure looks fine. If long generations die silently, move down a tier. ## Quantization recipe `experts_only`: the routed experts (and the vision/audio matmuls) carry the tier's bit-width, while **attention, token/output embeddings, RMSNorms, the router gate, the per-layer short-convolutions and the relative-position bias stay high-precision (8-bit)**. That asymmetry is the point, and it is nearly free: non-expert weights are only **5.9 B of the 264 B params (~6 GB at 8-bit)**, yet they are read **in full on every token**, while just 6 of 256 experts are. Bits spent there cost ~2% of the file and protect the paths every token depends on. Expert precision degrades *gracefully*; attention and router precision degrade *globally*. ## Usage ```python # pip install mlx mlx-lm transformers (+ scipy for audio, pillow for images) # this repo bundles the inkling_mlx/ loader, so there is nothing else to install from inkling_mlx.load import load from inkling_mlx.generate import greedy_generate from transformers import AutoTokenizer path = "./Inkling-Small-mlx-4bit" model, config = load(path) tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True) ids = tok("The capital of France is")["input_ids"] print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64))) ``` Images are cut into 40 px patches (one soft-token each) and audio is d-mel encoded at 20 tokens/s; both go through `InklingProcessor`. See the source repo for a multimodal runner. ### Benchmark it yourself ```bash python serving/bench_mlx.py --model ./Inkling-Small-mlx-4bit --tier 4bit ``` Reports load time, prefill tok/s, decode tok/s at several context lengths, and peak memory. ## Attribution - Base model: [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small), Apache-2.0. - The bundled `inkling_mlx/` loader is vendored from [PipeNetwork/inkling-mlx](https://github.com/PipeNetwork/inkling-mlx), Apache-2.0. - Quantized with `mlx`; see the recipe above for exactly which tensors were touched.