---
title: Qwen3.8-27B-DFlash2-NVFP4-RTNcal
canonical_url: "https://www.modelscope.cn/models/manateelazycat/Qwen3.8-27B-DFlash2-NVFP4-RTNcal"
md_url: "https://www.modelscope.cn/models/manateelazycat/Qwen3.8-27B-DFlash2-NVFP4-RTNcal.md"
repository: manateelazycat/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
last_updated: 2026-09-20
license: apache-2.0
model_type:
  - qwen3
architectures:
  - DFlash2DraftModel
base_model:
  - incoai/Qwen3.8-27B-DFlash2
base_model_relation: quantized
parameters: 1.1B
tensor_type:
  - F32
  - U8
  - BF16
  - F8_E4M3
library_name:
  - safetensors
downloads: 183
stars: 0
tags:
  - qwen3.8
  - dflash2
  - speculative-decoding
  - draft-model
  - nvfp4
  - modelopt
  - blackwell
  - rtx-5090
---

# Qwen3.8-27B-DFlash2-NVFP4-RTNcal

> Qwen3.8-27B-DFlash2-NVFP4-RTNcal - manateelazycat 在 ModelScope 开源的模型。Qwen3.8-27B NVFP4 snapshot, revision bd7a934213c47a9e7ef69eef36bb3325f47fd1f1. Original model card and Apache-2.0 license metadata preserved. Source is recorded in Git history.

manateelazycat/Qwen3.8-27B-DFlash2-NVFP4-RTNcal 是 ModelScope 魔搭社区上的 1.1B 参数机器学习模型，采用 apache-2.0 许可，基于 incoai/Qwen3.8-27B-DFlash2 构建。

- **Repository**: manateelazycat/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
- **License**: apache-2.0
- **Parameters**: 1.1B
- **Base model**: incoai/Qwen3.8-27B-DFlash2
- **Tags**: qwen3.8, dflash2, speculative-decoding, draft-model, nvfp4, modelopt, blackwell, rtx-5090
- **Downloads**: 183
- **Stars**: 0
- **Last updated**: 2026-09-20

Source: https://www.modelscope.cn/models/manateelazycat/Qwen3.8-27B-DFlash2-NVFP4-RTNcal

---

# Qwen3.8-27B-DFlash2-NVFP4-RTNcal

Calibrated **NVFP4 (W4A4)** quantization of the [DFlash 2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2) block-diffusion
draft model for **Qwen3.8-27B**, packaged in the NVIDIA ModelOpt layout that SGLang loads natively.

Built for a single **RTX 5090 (32 GB)** serving Qwen3.8-27B in NVFP4 with SGLang: the draft shrinks from **3.53 GB (BF16) to 1.37 GB**
of VRAM, which turns into **+44 % KV-cache context (90K → 130K tokens)** at the same decode speed and the same acceptance rate
as the BF16 draft. Output quality is unchanged by construction: the target model verifies every drafted token.

| Draft | VRAM | KV context* | Decode (end-to-end) | Accepted / 8 |
|---|---|---|---|---|
| BF16 (upstream) | 3.53 GB | 90K | 215 tok/s | 3.71 |
| **NVFP4, calibrated (this repo)** | **1.37 GB** | **130K** | **228 tok/s** | **3.60** |
| NVFP4, round-to-nearest, *uncalibrated* | 1.37 GB | 130K | 210 tok/s | 3.26 |

\* Target `gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090`, FP8 KV cache, `--mem-fraction-static 0.90`, HiCache on.
Benchmark: 5 mixed prompts (code, math, French prose, JSON, essay) × 600 generated tokens, 3 runs each, temperature 0.7.
Run-to-run noise on the acceptance length is about ±0.2, so BF16 and this checkpoint are at parity.

## Long-context agentic benchmark

Coding-agent session on the SGLang source tree (system prompt + 6 tool schemas, tool results = real source files), thinking on,
streaming; 3 turns per context size (cold analysis, follow-up on the cached prefix, tool call with `tools`), 2 passes.
TTFT is identical (prefill is the target's work, ~4K tok/s; cached prefix → 0.04–0.2 s in both), so only decode is shown.

| Context | Decode BF16 → **NVFP4 cal.** (turn 1, thinking) | Decode BF16 → **NVFP4 cal.** (turn 2) | Accepted/8 BF16 → **NVFP4 cal.** |
|---|---|---|---|
| 8K | 166 → **200** tok/s | 202 → 183 | 3.12 → 3.03 |
| 32K | 168 → **196** | 217 → **235** | 3.18 → 3.32 |
| 64K | 174 → **180** | 203 → **229** | 3.44 → 3.19 |
| 85K | 157 → **166** | 178 → **218** | 3.18 → 3.21 |
| 110K | *exceeds BF16 context* → **207** | — → **226** | — → **3.63** |

Tool calls were emitted correctly in 10/10 tool turns with this draft (6/8 with BF16, the misses being sampling noise at T=0.7).

## What is quantized

- **NVFP4 (E2M1, group size 16, FP8-E4M3 block scales, FP32 per-tensor global scale)** — all 35 linear projections of the
  5 draft layers: `self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj`. Fused shards (`q/k/v`, `gate/up`) share their
  `weight_scale_2` and `input_scale`, as ModelOpt requires.
- **BF16 (unchanged)** — `fc` (target-feature projection), `attention_conv.*` / `mlp_conv.*` (dynamic depthwise convolutions),
  `candidate_selector.*` (codebooks + hidden projection), all RMSNorm weights. These are plain `nn.Linear`/parameters in
  SGLang's `DFlash2DraftModel` and cannot be quantized there.
- Activations are quantized at runtime by SGLang to NVFP4 with the static per-tensor `input_scale` stored here (W4A4).

## Calibration — the part that matters

Round-to-nearest NVFP4 without calibration loses ~13 % acceptance (3.26 vs 3.71). The whole gap is recovered by calibrating the
**activation scales** on the draft's real inputs:

- **On-policy data**: 460 conversations generated by the target model itself through SGLang (thinking enabled on half of them),
  6 domains — English multi-turn chat (ultrachat), French (Wikipedia-based tasks + free prompts), code (glaive-code-assistant +
  bash/SQL/TS/Rust/Go/CUDA prompts), tool calling (glaive-function-calling-v2 with tool schemas), math (GSM8K, MATH-500),
  structured output (JSON/YAML/Markdown). 281,649 generated tokens.
- **Capture**: forward pre-hooks on the 20 quantizable linears of the BF16 draft while it served those conversations
  (prefill and decode, eager mode), ~621K input rows per layer. Per-tensor `input_scale = amax / (6 · 448)`.
- **Weights**: round-to-nearest on the NVFP4 grid; global scale `amax / (6 · 448)`, per-group FP8 scales.

Things that were tried and made **no measurable difference** on this draft (all within ±0.2 noise): GPTQ with full Hessians,
keeping `q/k/v` in BF16, SmoothQuant folding (`o_proj`, `gate_up`, `down` only — the `qkv` fold is not function-preserving in
SGLang because context K/V are materialized from `hidden_norm(fc(features))` without the per-layer `input_layernorm`).

## Usage with SGLang

Requires SGLang built from `main` at or after commit `ecbadf0b` (adds `DFlash2DraftModel`), CUDA 13, a Blackwell GPU (sm120)
for the NVFP4 GEMMs, and the draft quantization flag:

```bash
sglang serve \
  --trust-remote-code \
  --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --served-model-name qwen3.8-27b \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.90 \
  --attention-backend flashinfer \
  --max-running-requests 1 --cuda-graph-max-bs 1 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 5.61 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <this repo> \
  --speculative-draft-model-quantization modelopt_fp4 \
  --speculative-num-draft-tokens 8 \
  --chunked-prefill-size 1024 \
  --cuda-graph-bs-prefill 64 128 256 512 1024 \
  --enable-memory-saver \
  --enable-hierarchical-cache --hicache-ratio 2 --hicache-write-policy write_through \
  --host 0.0.0.0 --port 30000
```

Notes:
- `--speculative-draft-model-quantization modelopt_fp4` is required; SGLang does not requantize custom draft models on the fly.
- `--speculative-num-draft-tokens 8` is the DFlash 2 block size and cannot be raised.
- The `ignore` list in `config.json` (flat ModelOpt format) is what SGLang reads to leave `fc` & co. in BF16; keep it if you edit the config.
- `--cuda-graph-bs-prefill 64 128 256 512 1024` and `MAX_JOBS=2` keep host-RAM usage sane on a 64 GB box during graph capture / JIT.
- KV cache in FP4 (`--kv-cache-dtype nvfp4`) does not work with speculative decoding on this SGLang build (hybrid GDN model).

## Files

- `model.safetensors` — 1,550,153,248 bytes (sha256 `2228b9b2…`), 186 tensors (35 uint8 packed FP4 weights, 35 FP8 block scales, 70 FP32 scales, 46 BF16).
- `hf_quant_config.json`, `config.json` (`quantization_config` with `ignore`), tokenizer-free (uses the target's tokenizer), `README.incoai-original.md` (upstream card).

## Reproduce

Scripts (SGLang side): `dflash_calib_hook.py` (activation capture, temporary patch of `sglang/srt/models/dflash.py`),
`calib/build_prompts.py` + `calib/run_calib.py` (on-policy calibration run), `quantize-dflash2-gptq-nvfp4.py --rtn`
(export). Base weights: `incoai/Qwen3.8-27B-DFlash2` (BF16). Quantized on 2026-08-29.

## Changelog

- **2026-08-30** — `config.json`: `quantization_config.producer` is now ModelOpt's dict form (was a plain string, which crashed
  vLLM's draft `ModelConfig` construction with `AttributeError: 'str' object has no attribute 'get'`; SGLang ignored the key).
  Weights unchanged (same `model.safetensors`, sha256 `2228b9b2…`). Thanks to @joelafrite for the report and the vLLM numbers.
- **2026-08-29** — initial release.

## License

Apache-2.0, same as the base DFlash 2 draft and Qwen3.8-27B.