--- title: Agents-A1-OptiQ-4bit canonical_url: "https://www.modelscope.cn/models/mlx-community/Agents-A1-OptiQ-4bit" md_url: "https://www.modelscope.cn/models/mlx-community/Agents-A1-OptiQ-4bit.md" repository: mlx-community/Agents-A1-OptiQ-4bit last_updated: 2026-09-15 license: apache-2.0 pipeline_tag: image-text-to-text tasks: - image-text-to-text model_type: - qwen3_5_moe architectures: - Qwen3_5MoeForConditionalGeneration base_model: - InternScience/Agents-A1 base_model_relation: quantized parameters: 6.5B tensor_type: - U32 - BF16 library_name: - mlx - safetensors - pytorch frameworks: - pytorch downloads: 88 stars: 0 tags: - mlx - quantized - mixed-precision - 4bit - 8bit - optiq - apple-silicon - image-text-to-text - vision-language - moe - vlm - vision - agentic --- # Agents-A1-OptiQ-4bit > Agents-A1-OptiQ-4bit - mlx-community 在 ModelScope 开源的模型。mlx-community/Agents-A1-OptiQ-4bit mlx-community/Agents-A1-OptiQ-4bit 是 ModelScope 魔搭社区上的 6.5B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 InternScience/Agents-A1 构建。 - **Repository**: mlx-community/Agents-A1-OptiQ-4bit - **License**: apache-2.0 - **Tasks**: image-text-to-text - **Parameters**: 6.5B - **Base model**: InternScience/Agents-A1 - **Tags**: mlx, quantized, mixed-precision, 4bit, 8bit, optiq, apple-silicon, image-text-to-text, vision-language, moe, vlm, vision, agentic - **Downloads**: 88 - **Stars**: 0 - **Last updated**: 2026-09-15 Source: https://www.modelscope.cn/models/mlx-community/Agents-A1-OptiQ-4bit --- # mlx-community/Agents-A1-OptiQ-4bit > **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/) A 4-bit mixed-precision MLX quant of [InternScience/Agents-A1](https://huggingface.co/InternScience/Agents-A1), an agentic reasoning model built on the Qwen3.5-35B-A3B Mixture-of-Experts architecture (256 experts, 8 active per token). Sensitive layers are kept at 8-bit and robust ones at 4-bit. **Image input works.** The vision tower is kept at bf16 in a sidecar, so this quant takes images as well as text. 65 GB of bf16 weights become **22 GB**. ## Running it on a 24 GB Mac At 22 GB this does not fit comfortably in a 24 GB Mac's Metal working set, and loading it resident makes decoding painfully slow. Serve it with SSD expert streaming instead, which reads only the active experts per token: ```bash optiq serve --model mlx-community/Agents-A1-OptiQ-4bit --stream-experts ``` That brings resident memory down to **4.58 GB**. Streaming is the default (`auto`) in `optiq serve`, so it engages by itself when a MoE will not fit; the flag above just makes it explicit. On a 32 GB+ Mac the model fits resident and streaming is unnecessary. ## Quantization details | Property | Value | |---|---| | Predominant precision | 4-bit | | Layers at 8-bit (sensitive) | 397 | | Layers at 4-bit (robust) | 113 | | Total quantized layers | 510 | | Achieved bits per weight | 4.513 | | Group size | 64 | | Experts | 256 per layer, 8 active per token | | Vision tower | bf16, 333 tensors, in `optiq/optiq_vision.safetensors` | | Size on disk | 22 GB (language 21 GB, vision sidecar 0.9 GB), from a 65 GB bf16 base | We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is the predominant precision, not the weighted average. The base model ships no MTP head, so this quant has no speculative-decoding sidecar. ### How the bit-widths were chosen The per-layer allocation is transferred from [mlx-community/Qwen3.5-35B-A3B-OptiQ-4bit](https://huggingface.co/mlx-community/Qwen3.5-35B-A3B-OptiQ-4bit), where it was derived by a KL-divergence sensitivity sweep against the bf16 reference on a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix). Agents-A1 is built on `Qwen/Qwen3.5-35B-A3B` and its architecture is unchanged (every field of the text config matches), so all 510 quantizable layers map across exactly and the allocation lands at the same 4.513 bits per weight when recomputed against Agents-A1's own tensors. These are **measured** bit-widths, not a static rule-of-thumb recipe. But they were measured on the base model, not on this fine-tune. Fine-tuning shifts weights, so Agents-A1's own per-layer sensitivities could differ somewhat from the base's. Which layers are fragile is mostly a property of the architecture, so the transfer is sound, but it is a transfer and you should know that. Only the language tower is quantized. The vision tower stays at bf16, which is how every OptiQ VLM ships. ## Usage ### Text ```bash pip install mlx-optiq optiq serve --model mlx-community/Agents-A1-OptiQ-4bit --stream-experts ``` Then point any OpenAI-compatible client at `http://127.0.0.1:8080/v1`. The sidecar lives in an `optiq/` subfolder, so a stock `*.safetensors` glob ignores it and `mlx-lm` sees a clean language model: ```python from mlx_lm import load, generate model, tokenizer = load("mlx-community/Agents-A1-OptiQ-4bit") response = generate(model, tokenizer, prompt="Explain MoE routing.", max_tokens=512) ``` Note that `mlx_lm.load` holds the whole model resident, which is slow on a 24 GB Mac. Prefer `optiq serve --stream-experts` there. This is a reasoning model: it thinks before answering, so give it enough `max_tokens` to finish. ### Images Send an image through the OpenAI-compatible endpoint: ```python import base64, io, requests from PIL import Image buf = io.BytesIO(); Image.open("photo.jpg").save(buf, format="PNG") uri = "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode() requests.post("http://127.0.0.1:8080/v1/chat/completions", json={ "model": "a1", "max_tokens": 256, "messages": [{"role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": uri}}]}]}) ``` ## Verification Text, arithmetic reasoning, and image understanding were all exercised on the finished artifact before release, through expert streaming on a 24 GB M4. The quantization was also checked numerically: dequantizing individual experts out of the artifact and comparing them against the corresponding experts in the bf16 checkpoint gives 0.7% mean relative error on the 8-bit layers and 9.8% on the 4-bit layers, which is what each bit-width should cost. No task benchmarks were run on this quant; for measured quality numbers on the base architecture, see the [Qwen3.5-35B-A3B OptiQ card](https://huggingface.co/mlx-community/Qwen3.5-35B-A3B-OptiQ-4bit). Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as [the original](https://huggingface.co/InternScience/Agents-A1).