--- title: Ornith-1.0-9B-OptiQ-6bit canonical_url: "https://www.modelscope.cn/models/mlx-community/Ornith-1.0-9B-OptiQ-6bit" md_url: "https://www.modelscope.cn/models/mlx-community/Ornith-1.0-9B-OptiQ-6bit.md" repository: mlx-community/Ornith-1.0-9B-OptiQ-6bit last_updated: 2026-08-07 license: mit pipeline_tag: image-text-to-text tasks: - image-text-to-text model_type: - qwen3_5 architectures: - Qwen3_5ForConditionalGeneration base_model: - ornith-ai/Ornith-1.0-9B base_model_relation: quantized parameters: 2.2B tensor_type: - BF16 - U32 library_name: - mlx - safetensors - pytorch frameworks: - pytorch downloads: 47 stars: 0 tags: - mlx - quantized - mixed-precision - 6bit - 8bit - optiq - apple-silicon - image-text-to-text - vision-language - qwen3.5 --- # Ornith-1.0-9B-OptiQ-6bit > Ornith-1.0-9B-OptiQ-6bit - mlx-community 在 ModelScope 开源的模型。mlx-community/Ornith-1.0-9B-OptiQ-6bit mlx-community/Ornith-1.0-9B-OptiQ-6bit 是 ModelScope 魔搭社区上的 2.2B 参数image-text-to-text模型，采用 mit 许可，基于 ornith-ai/Ornith-1.0-9B 构建。 - **Repository**: mlx-community/Ornith-1.0-9B-OptiQ-6bit - **License**: mit - **Tasks**: image-text-to-text - **Parameters**: 2.2B - **Base model**: ornith-ai/Ornith-1.0-9B - **Tags**: mlx, quantized, mixed-precision, 6bit, 8bit, optiq, apple-silicon, image-text-to-text, vision-language, qwen3.5 - **Downloads**: 47 - **Stars**: 0 - **Last updated**: 2026-08-07 Source: https://www.modelscope.cn/models/mlx-community/Ornith-1.0-9B-OptiQ-6bit --- # mlx-community/Ornith-1.0-9B-OptiQ-6bit A 6-bit mixed-precision MLX quant of [ornith-ai/Ornith-1.0-9B](https://huggingface.co/ornith-ai/Ornith-1.0-9B), built on the Qwen3.5-9B architecture. Sensitive layers are kept at 8-bit and robust ones at 6-bit. 18.8 GB of bf16 weights become **9.06 GB**, of which 8.15 GB is the quantized language tower and 0.91 GB the bf16 vision tower. Built at the request of [a discussion on the 4-bit card](https://huggingface.co/mlx-community/Ornith-1.0-9B-OptiQ-4bit/discussions/1) asking for a quality-first variant. If you are choosing between the two, this one costs 1.0 GB more on disk and puts every layer at least two bits above the 4-bit variant's floor. Sizes here are decimal GB, matching what the Hub sidebar reports. **Image input works.** The vision tower is kept at bf16 in a sidecar, so this quant takes images as well as text. ## Quantization details | Property | Value | |---|---| | Predominant precision | 6-bit | | Layers at 8-bit (sensitive) | 117 | | Layers at 6-bit (robust) | 133 | | Total quantized layers | 250 | | Weighted mean bits per weight, quantized tensors | 6.626 | | Effective bpw, language tower on disk | 7.284 | | Effective bpw, whole artifact incl. vision | 8.099 | | Group size | 64 | | Mode | affine | | Quantized params | 8,952,741,888 | | Vision tower | bf16, 333 tensors, in `optiq/optiq_vision.safetensors` | | Size on disk | 9.06 GB, from an 18.8 GB bf16 base | Following the naming convention `llama.cpp` uses for Q4_K_M and similar mixed quants, the "6-bit" label is the predominant precision, not the weighted average. The three bpw figures differ for two separate reasons, both worth being explicit about. 6.626 is the weighted mean over the quantized tensors themselves. 7.284 adds what the packed format costs on disk: per-group scales and biases, plus 177 bf16 tensors (norms and biases, 1.06 M params) that are never quantized. 8.099 adds the bf16 vision tower, which is not quantized at all, by design, as in every OptiQ VLM. Verified: 250 quantized tensors, `U32 elements * (32 / bits)` summing to exactly 8,952,741,888 params. ### How the bit-widths were chosen By **structural rules** (`--method static`), not by measurement. mlx-optiq assigns bits from architectural position: embeddings and output head, first and last block, attention over dense MLP. Same method as [Qwen3.5-122B-A10B-OptiQ-2bit](https://huggingface.co/mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit) and [Mistral-Small-4-119B-2603-OptiQ-2bit](https://huggingface.co/mlx-community/Mistral-Small-4-119B-2603-OptiQ-2bit). This differs from the [4-bit variant](https://huggingface.co/mlx-community/Ornith-1.0-9B-OptiQ-4bit), which transferred a KL-divergence sensitivity map from Qwen3.5-9B. Neither is a sensitivity sweep measured on Ornith's own weights. We started one and abandoned it, for a reason worth stating: At candidate bits 6 and 8, the measured per-layer KL gap between the two is **3-5%**, against roughly an order of magnitude for 4 vs 8. Worse, the sweep on 32 GB of RAM has to run in `uniform_4bit` reference mode (mlx-optiq requires the bf16 checkpoint to fit in 70% of available RAM, so 18.8 GB of weights needs ~26.9 GB free), so KL is measured against a 4-bit baseline that is *worse than both candidates*. That works when one candidate is the baseline and you are measuring the gain from upgrading; at 6 vs 8 the difference sinks into calibration noise. An 8-hour sweep would have produced a layer ranking driven by noise, and a card claiming it was measured. Structural rules with an honest label are the better artifact. If you want the measurement, it needs a machine that fits the bf16 reference. ## Fidelity Weight-level, measured by dequantizing every quantized tensor out of this artifact and comparing against the bf16 source tensor by tensor. No prompts, no sampling, hardware-independent. | Group | Tensors | Params | rel L2 | SNR | |---|---|---|---|---| | 8-bit layers | 117 | 3.50 B | 0.742% | 42.59 dB | | 6-bit layers | 133 | 5.45 B | 2.266% | 32.89 dB | | **All** | **250** | **8.95 B** | **1.771%** | **35.04 dB** | Worst single-element error across all 250 tensors: 0.0117. The 8-bit figure is the sanity check. Affine 8-bit at group size 64 lands at 42.6-42.7 dB across every architecture we have measured, and 42.59 dB here says the conversion is sound. The 0.742% relative error also matches the 0.7% the 4-bit card reports for its own 8-bit layers. ## Throughput M-series, 32 GB unified memory, `mlx-lm` text path: | Metric | Value | |---|---| | Decode | 23.1 tok/s | | Prompt | 12.2 tok/s | | Time to first token | 1.62 s | | Peak memory, text | 8.30 GB | | Peak memory, image + text | 20.93 GB | | Load time | 4.1 s | The image path's 20.93 GB peak is on a 760x240 screenshot. Larger images cost more. Text-only fits comfortably in 16 GB; image work on 16 GB will be tight. ## Verification Run on the finished artifact, after conversion: - **Arithmetic.** `17*24` gives 408, with correct intermediate steps. - **Instruction following.** "List exactly three primary colors, one per line, no other text" gives three lines, no extra text. - **Code.** Correct Python string reversal. - **Multilingual.** Spanish prompt answered in Spanish, correctly. - **Image.** A synthetic UI screenshot with two buttons and a total: names "SubmitOrder", "Cancel" and "Total: 42.50 USD", all three correct. - **Vision sidecar.** Checked tensor by tensor against the sidecar published in the 4-bit repo: 333/333 tensors, identical names, identical shapes, all BF16. ### What was not measured - **No task benchmarks.** No MMLU, GSM8K, or any harness score. Nothing here claims this quant scores within N% of the source on any benchmark. - **No perplexity or top-1 agreement against bf16.** The bf16 control does not fit alongside this model in 32 GB, and our aggregate-perplexity harness has a known defect where one out-of-domain passage dominates the total. Weight-level fidelity above is exact and prompt-independent; it answers a different question than behavioural quality, and does not substitute for it. - **No sensitivity measurement on Ornith's own weights.** See above. - **No comparison against the 4-bit variant on the failure the requester reported** (skipped instructions, shallower reasoning). Higher bit-widths and a 2.3x lower reconstruction error make it plausible this helps, but we did not measure that specific behaviour, and the fidelity table is not evidence for it. - **Long context untested.** All checks were short prompts. ## Usage ### Text The sidecar lives in an `optiq/` subfolder, so a stock `*.safetensors` glob ignores it and `mlx-lm` sees a clean language model. ```bash pip install mlx-lm ``` ```python from mlx_lm import load, generate model, tokenizer = load("mlx-community/Ornith-1.0-9B-OptiQ-6bit") prompt = tokenizer.apply_chat_template( [{"role": "user", "content": "Explain the difference between TCP and UDP."}], add_generation_prompt=True, tokenize=False) print(generate(model, tokenizer, prompt=prompt, max_tokens=512)) ``` This is a reasoning model: it thinks inside `...` before answering, so give it enough `max_tokens` to finish. ### Images Image input needs [`mlx-optiq`](https://mlx-optiq.com/), which loads the bf16 vision sidecar and feeds the merged embeddings to the quantized language tower. **Use 0.4.17 or newer.** Versions up to 0.2.18 look for the sidecar only in the repo root, so they will not find it in `optiq/`, and their vision-tower prefix list has no `visual.` entry, which this architecture needs. Note that a bare `pip install mlx-optiq` may resolve to an older version depending on your other pins, so check `optiq.__version__`. ```bash pip install "mlx-optiq>=0.4.17" ``` ```python from PIL import Image from optiq.runtime.engine import OptiqEngine engine = OptiqEngine("mlx-community/Ornith-1.0-9B-OptiQ-6bit") answer = engine.generate("What is in this image?", images=[Image.open("photo.jpg")], max_tokens=512) print(answer.text) ``` The base model ships no MTP head, so this quant has no speculative-decoding sidecar. `OptiqEngine` prints `MTP weights not found` on load and falls back to autoregressive decoding; that line is expected, not an error. ## Sibling variants Weight sizes, read from the Hub API (`.safetensors` only, decimal GB): | Repo | Bits | Weights | Vision | |---|---|---|---| | [Ornith-1.0-9B-4bit](https://huggingface.co/mlx-community/Ornith-1.0-9B-4bit) | uniform 4 | 5.95 GB | no | | [Ornith-1.0-9B-OptiQ-4bit](https://huggingface.co/mlx-community/Ornith-1.0-9B-OptiQ-4bit) | mixed 4/8, 5.209 bpw | 8.01 GB | yes | | [Ornith-1.0-9B-6bit](https://huggingface.co/mlx-community/Ornith-1.0-9B-6bit) | uniform 6 | 8.19 GB | no | | **this one** | mixed 6/8, 6.626 bpw | 9.06 GB | yes | | [Ornith-1.0-9B-8bit](https://huggingface.co/mlx-community/Ornith-1.0-9B-8bit) | uniform 8 | 10.43 GB | no | | [Ornith-1.0-9B-bf16](https://huggingface.co/mlx-community/Ornith-1.0-9B-bf16) | none | 18.82 GB | no | The two OptiQ rows include an identical 0.91 GB bf16 vision tower, so on the language tower alone this quant is 8.15 GB against the 4-bit's 7.10 GB. That 1.05 GB is the whole cost of moving the 4-bit floor up to 6. The uniform variants do not carry a vision sidecar. If you need image input, use one of the two OptiQ quants. Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as [the original](https://huggingface.co/ornith-ai/Ornith-1.0-9B).