---
title: gemma-4-12B-it-OptiQ-4bit
canonical_url: "https://www.modelscope.cn/models/mlx-community/gemma-4-12B-it-OptiQ-4bit"
md_url: "https://www.modelscope.cn/models/mlx-community/gemma-4-12B-it-OptiQ-4bit.md"
repository: mlx-community/gemma-4-12B-it-OptiQ-4bit
last_updated: 2026-10-18
license: gemma
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - gemma4_unified
architectures:
  - Gemma4UnifiedForConditionalGeneration
base_model:
  - google/gemma-4-12B-it
base_model_relation: quantized
parameters: 2.5B
tensor_type:
  - BF16
  - U32
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 230
stars: 1
tags:
  - mlx
  - quantized
  - mixed-precision
  - 4bit
  - 8bit
  - optiq
  - apple-silicon
  - text-generation
  - gemma-4
---

# gemma-4-12B-it-OptiQ-4bit

> gemma-4-12B-it-OptiQ-4bit - mlx-community 在 ModelScope 开源的模型。mlx-community/gemma-4-12B-it-OptiQ-4bit

mlx-community/gemma-4-12B-it-OptiQ-4bit 是 ModelScope 魔搭社区上的 2.5B 参数text-generation模型，采用 gemma 许可，基于 google/gemma-4-12B-it 构建。

- **Repository**: mlx-community/gemma-4-12B-it-OptiQ-4bit
- **License**: gemma
- **Tasks**: text-generation
- **Parameters**: 2.5B
- **Base model**: google/gemma-4-12B-it
- **Tags**: mlx, quantized, mixed-precision, 4bit, 8bit, optiq, apple-silicon, text-generation, gemma-4
- **Downloads**: 230
- **Stars**: 1
- **Last updated**: 2026-10-18

Source: https://www.modelscope.cn/models/mlx-community/gemma-4-12B-it-OptiQ-4bit

---

# mlx-community/gemma-4-12B-it-OptiQ-4bit

> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)

A 4-bit mixed-precision MLX quant produced by [mlx-optiq](https://mlx-optiq.com/), the sensitivity-aware quantization toolkit for Apple Silicon. Scores +6.40 over stock uniform 4-bit on the six-metric Capability Score, the second-largest mixed-precision gain in the Gemma-4 lineup.

A 4-bit mixed-precision MLX quant of [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it), the unified (text+vision+audio) Gemma-4. This artifact is the **text-inference path**: the language tower is quantized and the vision/audio towers are dropped during conversion. Per-layer bit-widths come from a KL-divergence sensitivity pass on a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix) (prose, reasoning, code, agent, tool-call, constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit.

## Quantization details

| Property | Value |
|---|---|
| Predominant precision | 4-bit |
| Layers at 8-bit (sensitive) | 156 |
| Layers at 4-bit (robust) | 172 |
| Total quantized layers | 328 |
| Average bits per weight | 5.22 |
| Group size | 64 |
| Calibration mix | [six-domain mix](https://mlx-optiq.com/blog/calibration-mix) (40 samples × 6 domains) |
| Reference for sensitivity | uniform-4-bit (bf16 base too large to fit in RAM; auto-resolved) |

We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform 4-bit on the Capability Score at a comparable disk size.

## Usage

This is the **unified** Gemma-4 12B (`model_type: gemma4_unified`), so it needs **mlx-lm from main** and **`import optiq`** to register the model type. The unified text tower is not in the 0.31.3 PyPI release of mlx-lm, and the main build also reports version `0.31.3`, so install from git rather than pinning a version:

```bash
pip install -U mlx-optiq "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git"
```

```python
import optiq  # registers the gemma4_unified model type
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/gemma-4-12B-it-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain quantum computing in simple terms.",
    max_tokens=200,
)
```

Gemma-4-12B is a reasoning model with a thinking channel. For direct (non-thinking) answers on tasks like extraction or classification, pass `chat_template_kwargs={"enable_thinking": False}` when applying the chat template.

For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters, sandboxed Python execution for agent workflows), install [`mlx-optiq`](https://mlx-optiq.com/):

```bash
pip install mlx-optiq
```

See the [Gemma-4 family guide](https://mlx-optiq.com/docs/gemma-4) on [mlx-optiq.com](https://mlx-optiq.com/) for sampling defaults, training recipes, and family-specific caveats.

## Benchmarks

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit, both at the full sample counts:

| Metric | OptiQ | Uniform 4-bit | Δ |
|---|---:|---:|---:|
| MMLU (5-shot, 1000 samples) | **42.6%** | 34.4% | +8.3 |
| GSM8K (1000 samples, 3-shot CoT) | **93.4%** | 90.1% | +3.3 |
| IFEval (full set, strict) | **73.9%** | 71.2% | +2.8 |
| BFCL-V3 simple (200 calls) | 71.0% | 71.5% | −0.5 |
| HumanEval (164 problems, pass@1) | **88.4%** | 76.8% | +11.6 |
| HashHop (long-context retrieval) | **40.0%** | 27.0% | +13.0 |
| **Capability Score** (mean of 6) | **68.23** | 61.83 | **+6.40** |
| KL vs uniform-4-bit reference (mean / p95) | 1.52 / 4.85 |, |, |
| On-disk size | 8.3 GB | 6.3 GB | +2.0 |

Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. BFCL-V3 is within its ±6.3pp confidence interval (a tie); the other five benchmarks all improve, with the largest gains on long-context retrieval and code. MMLU is scored cloze-style (answer-letter log-likelihood), which underscores reasoning models that prefer to think before answering; the strong GSM8K and HumanEval results are the better quality signal. See the [eval-framework writeup](https://mlx-optiq.com/blog/eval-framework) for the full methodology.

## Links

- **Project website:** [mlx-optiq.com](https://mlx-optiq.com/)
- **Gemma-4 family guide:** [mlx-optiq.com/docs/gemma-4](https://mlx-optiq.com/docs/gemma-4)
- **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/)
- **Calibration mix:** [mlx-optiq.com/blog/calibration-mix](https://mlx-optiq.com/blog/calibration-mix)
- **Eval framework:** [mlx-optiq.com/blog/eval-framework](https://mlx-optiq.com/blog/eval-framework)
- **Base model:** [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it)


## Quantize your own

This quant was produced by [mlx-optiq](https://mlx-optiq.com). Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

```bash
pip install mlx-optiq
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8
optiq lab   # full local workbench: chat, compare, quantize, fine-tune
```

## License

Gemma license (inherits from base model). See https://ai.google.dev/gemma/terms for the terms of use.