--- title: NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit canonical_url: "https://www.modelscope.cn/models/mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit" md_url: "https://www.modelscope.cn/models/mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit.md" repository: mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit last_updated: 2026-09-14 license: other pipeline_tag: text-generation tasks: - text-generation model_type: - nemotron_h architectures: - NemotronHForCausalLM base_model: - nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 base_model_relation: quantized parameters: 833.0M tensor_type: - BF16 - U32 library_name: - mlx - safetensors - pytorch frameworks: - pytorch language: - en downloads: 33 stars: 0 tags: - mlx - quantized - mixed-precision - 4bit - 8bit - optiq - apple-silicon - text-generation - nemotron - mamba - hybrid - nvidia --- # NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit > NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit - mlx-community 在 ModelScope 开源的模型。mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit 是 ModelScope 魔搭社区上的 833.0M 参数text-generation模型，采用 other 许可，基于 nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 构建。 - **Repository**: mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit - **License**: other - **Tasks**: text-generation - **Parameters**: 833.0M - **Base model**: nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 - **Tags**: mlx, quantized, mixed-precision, 4bit, 8bit, optiq, apple-silicon, text-generation, nemotron, mamba, hybrid, nvidia - **Downloads**: 33 - **Stars**: 0 - **Last updated**: 2026-09-14 Source: https://www.modelscope.cn/models/mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit --- # mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit > **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/) A 4-bit mixed-precision MLX quant of [mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16](https://huggingface.co/mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16) produced by [mlx-optiq](https://mlx-optiq.com/), the sensitivity-aware quantization toolkit for Apple Silicon. **+1.6 pp on GSM8K over stock uniform 4-bit** at the same per-weight precision target. Nemotron 3 Nano is a hybrid Mamba2 + attention model. Of the 42 backbone blocks, 4 are full attention and the rest are Mamba2 SSM or MLP. OptiQ measures each linear's KL-divergence sensitivity against a bf16 reference forward pass and assigns 4-bit or 8-bit per-layer based on that signal. Sensitive layers go to 8-bit; robust ones stay at 4-bit. ## Quantization details | Property | Value | |---|---| | Predominant precision | 4-bit | | Layers at 8-bit (sensitive) | 46 | | Layers at 4-bit (robust) | 47 | | Total quantized layers | 93 | | Achieved BPW | 5.43 | | Group size | 64 | | Calibration mix | [six-domain mix](https://mlx-optiq.com/blog/calibration-mix) (40 samples) | | Reference for sensitivity | bf16 (gold standard) | | Bundled KV-cache recipe | `kv_config.json`, 3 attn layers @ 4-bit, 1 @ 8-bit (5.0 avg KV bits) | We follow the same naming convention `llama.cpp` uses for Q4_K_M-style mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. ## Usage Load it with `mlx-lm` (the custom NemotronH modeling files ship in the repo and are picked up automatically): ```bash pip install mlx-lm ``` ```python from mlx_lm import load, generate model, tokenizer = load("mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit") response = generate( model, tokenizer, prompt="Explain why hybrid Mamba+attention models scale to long contexts.", max_tokens=300, ) ``` For mixed-precision KV-cache serving and sensitivity-aware LoRA fine-tuning, install [`mlx-optiq`](https://mlx-optiq.com/): ```bash pip install mlx-optiq # Serve with the bundled KV-cache recipe optiq serve --model mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit \ --kv-config kv_config.json ``` ## Benchmarks Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit: | Metric | OptiQ | Uniform 4-bit | Δ | |---|---:|---:|---:| | MMLU (5-shot, 1000 samples) | **64.0%** | 63.3% | +0.7 | | GSM8K (1000 samples, 3-shot CoT) | **81.5%** | 79.9% | +1.6 | | IFEval (full set, strict) | **56.2%** | 56.0% | +0.2 | | BFCL-V3 simple (200 calls) | **94.0%** | 94.0% | +0.0 | | HumanEval (164 problems, pass@1) | **77.4%** | 80.5% | -3.1 | | HashHop (long-context retrieval) | **27.0%** | 25.0% | +2.0 | | **Capability Score** (mean of 6) | **66.68** | 66.44 | **+0.24** | | KL vs bf16 reference (mean / p95) | 0.0353 / 0.1136 |, |, | | On-disk size | 2.94 GB | 2.13 GB | +0.81 | Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. On this Mamba-attention hybrid the mixed allocation runs a little richer than on a pure transformer, Mamba2 SSM blocks carry only two linears each, so the sensitivity probe upgrades more of them to 8-bit, which is why the disk delta is larger than on the Qwen / Gemma families. See the [eval-framework writeup](https://mlx-optiq.com/blog/eval-framework) for the full methodology. ## Links - **Project website:** [mlx-optiq.com](https://mlx-optiq.com/) - **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/) - **Calibration mix:** [mlx-optiq.com/blog/calibration-mix](https://mlx-optiq.com/blog/calibration-mix) - **Eval framework:** [mlx-optiq.com/blog/eval-framework](https://mlx-optiq.com/blog/eval-framework) - **Base model:** [mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16](https://huggingface.co/mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16) ## Base model This is a quantized derivative of NVIDIA Nemotron 3 Nano 4B. See the [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) for terms, the quant is distributed under the same license as the base. ## Quantize your own This quant was produced by [mlx-optiq](https://mlx-optiq.com). Point it at any Hugging Face model to get the same sensitivity-aware mixed precision: ```bash pip install mlx-optiq optiq convert --target-bpw 5.0 --candidate-bits 4,8 optiq lab # full local workbench: chat, compare, quantize, fine-tune ```