--- title: Laguna-S-2.1-oQ4e-fast canonical_url: "https://www.modelscope.cn/models/mlx-community/Laguna-S-2.1-oQ4e-fast" md_url: "https://www.modelscope.cn/models/mlx-community/Laguna-S-2.1-oQ4e-fast.md" repository: mlx-community/Laguna-S-2.1-oQ4e-fast last_updated: 2026-07-24 license: openmdw-1.1 pipeline_tag: text-generation tasks: - text-generation model_type: - laguna architectures: - LagunaForCausalLM base_model: - poolside/Laguna-S-2.1 base_model_relation: quantized parameters: 16.8B tensor_type: - U32 - BF16 library_name: - mlx - safetensors - pytorch frameworks: - pytorch downloads: 40 stars: 0 tags: - mlx - oq - quantized - moe - laguna --- # Laguna-S-2.1-oQ4e-fast > Laguna-S-2.1-oQ4e-fast - mlx-community 在 ModelScope 开源的模型。Laguna-S-2.1-oQ4e-fast mlx-community/Laguna-S-2.1-oQ4e-fast 是 ModelScope 魔搭社区上的 16.8B 参数text-generation模型，采用 openmdw-1.1 许可，基于 poolside/Laguna-S-2.1 构建。 - **Repository**: mlx-community/Laguna-S-2.1-oQ4e-fast - **License**: openmdw-1.1 - **Tasks**: text-generation - **Parameters**: 16.8B - **Base model**: poolside/Laguna-S-2.1 - **Tags**: mlx, oq, quantized, moe, laguna - **Downloads**: 40 - **Stars**: 0 - **Last updated**: 2026-07-24 Source: https://www.modelscope.cn/models/mlx-community/Laguna-S-2.1-oQ4e-fast --- # Laguna-S-2.1-oQ4e-fast Speed-tuned variant of [Laguna-S-2.1-oQ4e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e), the calibrated 4-bit MLX quantization of [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) (118B total, 8B activated per token). Same oQ level 4 enhanced, same 4-bit experts — the one thing I changed is the budget: instead of letting oQ spend freely on the attention, I capped it at `target_bpw=4.00` / `hard_cap_bpw=4.55`, and oQ dropped the sliding-window attention to the 4-bit base on its own while keeping the global-attention path protected. **4.54 bits/weight effective**, 63 GB on disk. For Apple Silicon. - **63 GB** on disk — essentially the same as the oQ4e it derives from; the fast trick trades bandwidth, not size, and its disk saving shrinks as the base bit-width rises - 48 layers, 47 of them MoE with 256 routed experts + 1 shared, top-10 (L0 is a dense MLP); interleaved attention (12 global with YaRN to 1M context, 36 sliding-window 512) - Peak memory in my tests: **59.8 GB** at 1k context, 62.9 GB at 64k — fits a 64 GB Mac - **69.3 tok/s** at 1k context against 55.9 for the oQ4e, at the same accuracy within noise - Converted and tested on a **Macbook Pro M5 Max 128GB 40 GPU** ## What changed against the oQ4e Only the bit budget. oQ derives bit-width per tensor from an imatrix-calibrated sensitivity pass; at level 4 enhanced it lands on 4-bit experts and spreads the attention across 5/8 bits. Attention is ~4% of the file but ~43% of the bytes read per token — the dense spine is read in full on every token while only 10 of 256 experts are touched — so it's the cheapest place on disk to buy back bandwidth. Passing `target_bpw=4.00` (with `hard_cap_bpw=4.55`) tells the allocator to hit a lower budget, and it picks where to take the loss. It kept the shared expert, embeddings and `lm_head` at 8 bits and dropped the sliding-window attention to the 4-bit base, holding the global-attention q/k/v at 6–8. Those are the layers that carry the long-context path; the 36 sliding-window layers only see 512 tokens. | | oQ4e | this build | |---|---|---| | routed experts | 4b gs128 | 4b gs128 (re-calibrated) | | shared expert, embeddings, `lm_head` | 8b | 8b | | attention q/k/v | 5b ×60, 8b ×84 | 4b ×107, 5b ×1, 6b ×14, 8b ×22 | | attention o/g | 5b ×39, 8b ×57 | 4b ×66, 5b ×1, 6b ×19, 8b ×10 | ## Requirements mlx-lm doesn't support the `laguna` architecture yet — there's an open PR: [mlx-lm#1223](https://github.com/ml-explore/mlx-lm/pull/1223). Until it lands, use **mlx-vlm** (0.6.3+), which implements laguna as a text-only model: ```bash uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ4e-fast --prompt "..." ``` oMLX serves it directly from **0.5.3** on — it vendors the PR and patches it into mlx-lm at import, so no model setting is needed. On older builds that predate the patch, discovery lands on mlx-lm and fails with `Model type laguna not supported`; set `model_type_override: "vlm"` and refresh discovery (`omlx restart`). ## How it was built `omlx.oq.quantize_oq_streaming` at `oq_level=4`, `enhanced=True`, `group_size=128`, dtype bfloat16, with `target_bpw=4.00` and `hard_cap_bpw=4.55`. The ladder's cached imatrix was no longer on disk, so I re-collected one (128 samples, same `oqe_code_multilingual` calibration); the experts are the same 4-bit as the oQ4e, re-calibrated, and the accuracy landed on top of it regardless (table below). Sensitivity was measured against a uniform 4-bit proxy on disk — the 235 GB FP model doesn't fit in 128 GB of RAM. Output is standard MLX affine quantization: no custom kernels or runtime required. ## Why it's faster Generation is memory-bandwidth bound, and what matters is bytes read per token, not bytes on disk. The oQ4e reads 5.62 GB per token, this build reads 4.91 — the difference is the sliding-window attention dropping from a 5/8 mix to the 4-bit base. The byte count alone predicts ~14%; measured is **24% at 1k, easing to 14% at 64k** as the protected global-attention path (kept at 6–8 bits for the long-context route) takes over the per-token work. Disk and peak memory barely move — this is a strictly-faster oQ4e at the same size and accuracy. ## Conversion check Smoke-tested after conversion with `mlx_vlm.generate`: coherent — solved `17 * 24 = 408`, broke it down by the distributive property and verified the result a second way with the standard algorithm, no repetition loop, at 67 tok/s. ## Performance Measured with oMLX's benchmark harness on a **Macbook Pro M5 Max 128GB 40 GPU**, single request, 128 generated tokens: | prompt | gen tok/s | prefill tok/s | TTFT ms | peak GB | |---|---|---|---|---| | 1k | 69.3 | 1145.7 | 895 | 59.82 | | 4k | 68.0 | 1103.0 | 3715 | 59.97 | | 8k | 65.9 | 978.6 | 8372 | 60.13 | | 16k | 62.4 | 893.7 | 18334 | 60.52 | | 32k | 55.2 | 836.3 | 39184 | 61.27 | | 64k | 45.5 | 749.6 | 87428 | 62.89 | Continuous batching at 1k prompt / 128 generated: | batch | tg tok/s | speedup | TTFT ms | E2E s | |---|---|---|---|---| | 1 | 69.3 | 1.00x | 895 | 2.75 | | 2 | 93.8 | 1.35x | 1855 | 4.59 | | 4 | 124.3 | 1.79x | 3127 | 7.32 | | 8 | 153.1 | 2.21x | 4764 | 12.80 | ## Benchmarks & Variants mmlu_pro, mathqa and winogrande, n=300 seeded samples each, thinking off. This build was measured on the oMLX LM engine directly (greedy, mathqa/winogrande at 1024 max tokens so reasoning isn't truncated). Standard error at this n is around 2.5 points, so the oQ4e-fast and the oQ4e are indistinguishable here. ![Accuracy vs bits per weight, three benchmarks, n=300](ladder.png) | Variant | Size | bpw | gen tok/s (1k → 64k) | mmlu_pro | mathqa | winogrande | |---|---|---|---|---|---|---| | [Laguna-S-2.1-oQ2e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e-fast) | 35 GB | 2.60 | 78.8 → 48.6 | 0.700 | 0.850 | 0.713 | | [Laguna-S-2.1-oQ2e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e) | 36 GB | 2.70 | 61.5 → 38.8 | 0.703 | 0.840 | 0.707 | | [Laguna-S-2.1-oQ3e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ3e-fast) | 49 GB | 3.56 | 77.2 → 48.4 | 0.750 | 0.887 | 0.760 | | [Laguna-S-2.1-oQ3e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ3e) | 49 GB | 3.59 | 67.5 → 40.1 | 0.750 | 0.880 | 0.760 | | [**Laguna-S-2.1-oQ4e-fast**](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e-fast) (this repo) | 63 GB | 4.54 | 69.3 → 45.5 | 0.787 | 0.873 | 0.777 | | [Laguna-S-2.1-oQ4e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e) | 64 GB | 4.60 | 55.9 → 39.8 | 0.757 | 0.887 | 0.777 | | [Laguna-S-2.1-oQ5e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ5e) | 78 GB | 5.30 | 57.5 → 38.1 | 0.773 | 0.883 | 0.797 | | [Laguna-S-2.1-oQ6e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ6e) | 92 GB | 6.27 | 53.0 → 32.9 | 0.763 | 0.873 | 0.777 | | [Laguna S 2.1 (API, bf16)](https://openrouter.ai/poolside/laguna-s-2.1) | — | 16 | — | 0.773 | 0.880 | 0.810 | Treat this as a rough sighting, not a verdict. Three benchmarks at n=300 cover a narrow slice of what the model does — no long-context work, no agentic loops, no real code — and at this sample size most of the ladder above 3.6 bpw sits inside the error bars. I ran them to size the drop between levels, not to rank the variants against each other. Test the one you're considering on your own workload before trusting any of it. The oQ4e-fast is the odd one on the ladder: it's faster than the oQ3e (69.3 vs 67.5 at 1k) despite being 14 GB larger, because it sheds attention the smaller build keeps. Pick it over the oQ4e whenever you'd have reached for the oQ4e — it's the same size, the same accuracy, and 14–24% faster. Pick the oQ3e-fast instead if 49 GB and 3-bit experts are enough for your workload and you want the highest throughput. ## Usage ```bash # mlx-vlm — plain mlx-lm doesn't support the laguna architecture uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ4e-fast \ --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300 # oMLX — discovers the model from the HF cache omlx serve ``` ## License [OpenMDW-1.1](https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md), inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use.