--- title: Laguna-S-2.1-oQ2e-fast canonical_url: "https://www.modelscope.cn/models/mlx-community/Laguna-S-2.1-oQ2e-fast" md_url: "https://www.modelscope.cn/models/mlx-community/Laguna-S-2.1-oQ2e-fast.md" repository: mlx-community/Laguna-S-2.1-oQ2e-fast last_updated: 2026-07-24 license: openmdw-1.1 pipeline_tag: text-generation tasks: - text-generation model_type: - laguna architectures: - LagunaForCausalLM base_model: - poolside/Laguna-S-2.1 base_model_relation: quantized parameters: 9.6B tensor_type: - U32 - BF16 library_name: - mlx - safetensors - pytorch frameworks: - pytorch downloads: 33 stars: 0 tags: - mlx - oq - quantized - moe - laguna --- # Laguna-S-2.1-oQ2e-fast > Laguna-S-2.1-oQ2e-fast - mlx-community 在 ModelScope 开源的模型。Laguna-S-2.1-oQ2e-fast mlx-community/Laguna-S-2.1-oQ2e-fast 是 ModelScope 魔搭社区上的 9.6B 参数text-generation模型，采用 openmdw-1.1 许可，基于 poolside/Laguna-S-2.1 构建。 - **Repository**: mlx-community/Laguna-S-2.1-oQ2e-fast - **License**: openmdw-1.1 - **Tasks**: text-generation - **Parameters**: 9.6B - **Base model**: poolside/Laguna-S-2.1 - **Tags**: mlx, oq, quantized, moe, laguna - **Downloads**: 33 - **Stars**: 0 - **Last updated**: 2026-07-24 Source: https://www.modelscope.cn/models/mlx-community/Laguna-S-2.1-oQ2e-fast --- # Laguna-S-2.1-oQ2e-fast Speed-tuned variant of [Laguna-S-2.1-oQ2e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e), the calibrated 2-bit MLX quantization of [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) (118B total, 8B activated per token). Same oQ level 2 enhanced, same imatrix, same experts — the one thing I changed is the budget: instead of letting oQ spend freely on the dense spine, I capped it at `target_bpw=2.54`, and oQ reallocated the attention on its own. **2.60 bits/weight effective**, 35 GB on disk. For Apple Silicon. - **35 GB** on disk, down from 235 GB BF16 and 36 GB for the oQ2e it derives from - 48 layers, 47 of them MoE with 256 routed experts + 1 shared, top-10 (L0 is a dense MLP); interleaved attention (12 global with YaRN to 1M context, 36 sliding-window 512) - Peak memory in my tests: **33.2 GB** at 1k context, 36.3 GB at 64k — fits a 48 GB Mac - **78.8 tok/s** at 1k context against 61.5 for the oQ2e, at the same mmlu_pro within noise - Converted and tested on a **Macbook Pro M5 Max 128GB 40 GPU** ## What changed against the oQ2e Only the bit budget. oQ derives bit-width per tensor from an imatrix-calibrated sensitivity pass; at level 2 enhanced it lands on 2-bit experts and puts **every** non-expert tensor — attention, embeddings, `lm_head`, routers, shared expert, 386 in total — at 8 bits. With 8B parameters active per token, that's a defensible allocation: the dense spine is read in full on every token while only 10 of 256 experts are touched, so protecting it is cheap in disk and expensive in bandwidth. Passing `target_bpw=2.54` (with `hard_cap_bpw=3.0`) tells the allocator to hit a lower budget, and it picks where to take the loss. It kept the shared expert, embeddings and `lm_head` at 8 bits and spent the rest on attention selectively: most projections drop to 3 bits, but q/k/v of the 12 global-attention layers stay at 6–8. Those are the layers that carry the long-context path; the 36 sliding-window layers only see 512 tokens. | | oQ2e | this build | |---|---|---| | routed experts | 2b gs128 | 2b gs128 (byte-identical) | | shared expert, embeddings, `lm_head` | 8b | 8b | | attention q/k/v | 8b | 3b ×81, 4b ×27, 6b ×14, 8b ×22 | | attention o/g | 8b | 3b ×54, 4b ×32, 8b ×10 | ## Requirements mlx-lm doesn't support the `laguna` architecture yet — there's an open PR: [mlx-lm#1223](https://github.com/ml-explore/mlx-lm/pull/1223). Until it lands, use **mlx-vlm** (0.6.3+), which implements laguna as a text-only model: ```bash uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ2e-fast --prompt "..." ``` oMLX serves it directly from **0.5.3** on — it vendors the PR and patches it into mlx-lm at import, so no model setting is needed. On older builds that predate the patch, discovery lands on mlx-lm and fails with `Model type laguna not supported`; set `model_type_override: "vlm"` and refresh discovery (`omlx restart`). ## How it was built `omlx.oq.quantize_oq_streaming` at `oq_level=2`, `enhanced=True`, `group_size=128`, dtype bfloat16, with `target_bpw=2.54` and `hard_cap_bpw=3.0`. Sensitivity was measured against a uniform 4-bit proxy on disk — the 235 GB FP model doesn't fit in 128 GB of RAM — reusing the same cached imatrix as the whole oQ ladder, so the only difference against the oQ2e is the budget. Output is standard MLX affine quantization: no custom kernels or runtime required. Unlike the upstream config, `generation_config.json` here ships `repetition_penalty: 1.05`: at this bit-width the model can fall into verbatim repetition loops in long-form generation, and this is the mildest setting that reliably broke them in my tests. ## Why it's faster Generation is memory-bandwidth bound, and what matters is bytes read per token, not bytes on disk. The dense spine is read in full on every token; of the experts, only 10 of 256 are touched. The oQ2e reads 5.21 GB per token, this build reads 3.72 GB. The speedup lands at 28%, not the 40% the byte count alone predicts, because MLX's low-bit kernels move fewer bytes per second than its 8-bit ones — part of what the narrower attention saves in traffic comes back as slower matmuls. ## Conversion check Smoke-tested with `mlx_vlm.generate`: coherent on a math prompt (`17 * 24` decomposed by the distributive property, then verified a second way, 408, no repetition loop) at 76 tok/s. It also completed a 300-question mmlu_pro run with no truncations or malformed answers. ## Performance Measured with oMLX's benchmark harness on a **Macbook Pro M5 Max 128GB 40 GPU**, single request, 128 generated tokens: | prompt | gen tok/s | prefill tok/s | TTFT ms | peak GB | |---|---|---|---|---| | 1k | 78.8 | 1204.8 | 851 | 33.21 | | 4k | 74.4 | 1147.7 | 3570 | 33.35 | | 8k | 72.9 | 1050.7 | 7798 | 33.52 | | 16k | 67.9 | 961.3 | 17044 | 33.91 | | 32k | 60.8 | 891.3 | 36766 | 34.66 | | 64k | 48.6 | 794.1 | 82535 | 36.27 | Continuous batching at 1k prompt / 128 generated: | batch | tg tok/s | speedup | TTFT ms | E2E s | |---|---|---|---|---| | 1 | 78.8 | 1.00x | 851 | 2.48 | | 2 | 106.5 | 1.35x | 1791 | 4.19 | | 4 | 145.6 | 1.85x | 3039 | 6.62 | | 8 | 175.4 | 2.23x | 4412 | 11.49 | ## Benchmarks & Variants mmlu_pro, mathqa and winogrande, n=300 seeded samples each, thinking off, identical questions across every variant. The bf16 row is the hosted API, measured the same way. Standard error at this n is around 2.5 points, so the gap to the oQ2e is not a real difference; the gap to oQ4e and up is. ![Accuracy vs bits per weight, three benchmarks, n=300](ladder.png) | Variant | Size | bpw | gen tok/s (1k → 64k) | mmlu_pro | mathqa | winogrande | |---|---|---|---|---|---|---| | [**Laguna-S-2.1-oQ2e-fast**](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e-fast) (this repo) | 35 GB | 2.60 | 78.8 → 48.6 | 0.700 | 0.850 | 0.713 | | [Laguna-S-2.1-oQ2e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e) | 36 GB | 2.70 | 61.5 → 38.8 | 0.703 | 0.840 | 0.707 | | [Laguna-S-2.1-oQ3e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ3e-fast) | 49 GB | 3.56 | 77.2 → 48.4 | 0.750 | 0.887 | 0.760 | | [Laguna-S-2.1-oQ3e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ3e) | 49 GB | 3.59 | 67.5 → 40.1 | 0.750 | 0.880 | 0.760 | | [Laguna-S-2.1-oQ4e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e-fast) | 63 GB | 4.54 | 69.3 → 45.5 | 0.787 | 0.873 | 0.777 | | [Laguna-S-2.1-oQ4e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e) | 64 GB | 4.60 | 55.9 → 39.8 | 0.757 | 0.887 | 0.777 | | [Laguna-S-2.1-oQ5e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ5e) | 78 GB | 5.30 | 57.5 → 38.1 | 0.773 | 0.883 | 0.797 | | [Laguna-S-2.1-oQ6e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ6e) | 92 GB | 6.27 | 53.0 → 32.9 | 0.763 | 0.873 | 0.777 | | [Laguna S 2.1 (API, bf16)](https://openrouter.ai/poolside/laguna-s-2.1) | — | 16 | — | 0.773 | 0.880 | 0.810 | Treat this as a rough sighting, not a verdict. Three benchmarks at n=300 cover a narrow slice of what the model does — no long-context work, no agentic loops, no real code — and at this sample size most of the ladder above 3.6 bpw sits inside the error bars. I ran them to size the drop between levels, not to rank the variants against each other. Test the one you're considering on your own workload before trusting any of it. Pick this one over the oQ2e if you want the throughput — at this n the two are indistinguishable on accuracy. Pick oQ4e or above if accuracy matters more than fitting in 48 GB. ## Usage ```bash # mlx-vlm — plain mlx-lm doesn't support the laguna architecture uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ2e-fast \ --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300 # oMLX — discovers the model from the HF cache omlx serve ``` ## License [OpenMDW-1.1](https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md), inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use.