---
title: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit
canonical_url: "https://www.modelscope.cn/models/mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit"
md_url: "https://www.modelscope.cn/models/mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit.md"
repository: mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit
last_updated: 2026-09-15
license: other
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - nemotron_h
architectures:
  - NemotronHForCausalLM
base_model:
  - nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
base_model_relation: quantized
parameters: 6.0B
tensor_type:
  - BF16
  - U32
  - F32
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 36
stars: 0
tags:
  - mlx
  - optiq
  - quantized
  - 4bit
  - mixed-precision
  - nemotron
  - nemotron-h
  - mamba
  - moe
  - apple-silicon
---

# NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

> NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit - mlx-community 在 ModelScope 开源的模型。mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit 是 ModelScope 魔搭社区上的 6.0B 参数text-generation模型，采用 other 许可，基于 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 构建。

- **Repository**: mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit
- **License**: other
- **Tasks**: text-generation
- **Parameters**: 6.0B
- **Base model**: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- **Tags**: mlx, optiq, quantized, 4bit, mixed-precision, nemotron, nemotron-h, mamba, moe, apple-silicon
- **Downloads**: 36
- **Stars**: 0
- **Last updated**: 2026-09-15

Source: https://www.modelscope.cn/models/mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

---

# mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)

**NVIDIA's Mamba-attention hybrid MoE, running locally on a Mac.** This is an OptiQ mixed-precision quant of [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16). 22.8 GB on disk, down from 65.8 GB.

Lightning is a `nemotron_h` hybrid: Mamba2 state-space blocks with a handful of attention layers, and a 128-expert MoE with 6 experts active per token. The routed experts carry most of the weight, so that is where the bits come from.

## What it is

| Property | Value |
|---|---|
| Base | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (~30B total, ~3B active) |
| Method | OptiQ mixed-precision, sensitivity-driven (uniform-4-bit reference) |
| Precision | per-layer 4/8-bit: 36 layers at 4-bit, 127 at 8-bit |
| Experts | 46 fused expert tensors: 36 at 4-bit, 10 at 8-bit |
| On disk | 22.8 GB |
| Speculative decoding | MTP head preserved in `optiq/mtp.safetensors` |

All 164 quantizable projections were measured. Every 4-bit assignment in the model is an expert tensor: the 128-expert stack is where the size is, and the sweep found it is also where the precision is least needed.

Following `llama.cpp`'s naming for mixed quants, the "4bit" label denotes the family, not the weighted average.

## Capability Score

Six-metric mean (the standard OptiQ text eval). Strong code generation; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers would predict.

| Metric | Score |
|---|---:|
| MMLU (5-shot, 969 samples) | 78.9% |
| GSM8K (1000 samples) | 85.5% |
| IFEval (full set, strict) | 68.4% |
| BFCL-V3 simple (200 calls) | 83.0% |
| HumanEval (164 problems, pass@1) | 87.8% |
| HashHop (long-context retrieval) | 40.0% |
| **Capability Score** (mean of 6) | **73.94** |

## Run it

```bash
pip install "mlx-optiq>=0.4.20"
```

```python
import optiq  # registers the nemotron_h MoE + MTP sidecar
from mlx_lm import load, generate

model, tok = load("mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit")
msgs = [{"role": "user", "content": "Explain why the sky is blue."}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt=prompt, max_tokens=600))
```

It is a reasoning model and emits a `<think>` block before its answer, so give it room and read the text after `</think>`.

## Serve it on a smaller Mac

At 22.8 GB this does not fit resident on a 24 GB machine. It is a 128-expert MoE, so stream the experts from SSD instead:

```bash
optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-experts
```

Measured on a 26 GB M4 Pro: 4.2 GB resident, 6.0 GB load peak, against a resident load that runs out of memory. The same policy is available from the library API via `OptiqEngine(path, stream_experts="auto")`, and from `optiq eval --stream-experts` for benchmarking.

## Links

- **Project website:** [mlx-optiq.com](https://mlx-optiq.com/)
- **Family guide:** [mlx-optiq.com/docs/nemotron3](https://mlx-optiq.com/docs/nemotron3)
- **All OptiQ quants:** [mlx-optiq.com/models](https://mlx-optiq.com/models)
- **Base model:** [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16)