---
title: Qwen3.8-Flash-Next-NInfer-Offload
canonical_url: "https://www.modelscope.cn/models/mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload"
md_url: "https://www.modelscope.cn/models/mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload.md"
repository: mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload
chinese_name: "Qwen3.8-Flash-Next NInfer-Offload 模型"
last_updated: 2026-10-02
license: other
pipeline_tag: text-generation
tasks:
  - text-generation
base_model:
  - Qwen/Qwen3.8-Flash-Next
base_model_relation: finetune
library_name:
  - other
frameworks:
  - other
language:
  - en
  - zh
downloads: 1
stars: 1
tags:
  - ninfer
  - mtp
  - ninfer-offload
  - moe
  - cpu-offload
  - quantized
---

# Qwen3.8-Flash-Next-NInfer-Offload

> Qwen3.8-Flash-Next-NInfer-Offload - mymodel3861 在 ModelScope 开源的模型。Qwen3.8-Flash-Next · NInfer-Offload

mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload 是 ModelScope 魔搭社区上的text-generation模型，采用 other 许可，基于 Qwen/Qwen3.8-Flash-Next 构建。

- **Repository**: mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload
- **License**: other
- **Tasks**: text-generation
- **Base model**: Qwen/Qwen3.8-Flash-Next
- **Tags**: ninfer, mtp, ninfer-offload, moe, cpu-offload, quantized
- **Downloads**: 1
- **Stars**: 1
- **Last updated**: 2026-10-02

Source: https://www.modelscope.cn/models/mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload

---

# Qwen3.8-Flash-Next · NInfer-Offload

English | [中文](#中文说明)

[Qwen/Qwen3.8-Flash-Next](https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next) converted to the
`.ninfer` v3 format for **[NInfer-Offload](https://github.com/1872183316/ninfer-offload)**, which runs
this 125B-total / 6B-active MoE model on a 16 GB consumer GPU by keeping the routed experts in
system RAM. It includes the model's MTP predictor for speculative decoding.

**This file only works with NInfer-Offload.** It is not a GGUF or safetensors checkpoint and cannot be
loaded by llama.cpp, vLLM, transformers or upstream NInfer.

## Files

The 110 GB artifact is stored as NInfer's native multi-file set (at most 50 GB per file). Keep all
files in one directory and pass only the entry file `qwen3_8_flash_next.ninfer` to NInfer; it reads
the `.part-000N` continuation files itself. No reassembly is needed.

| File | Bytes | SHA256 |
|---|---|---|
| `qwen3_8_flash_next.ninfer` (entry) | 49,999,998,976 | `9bb83f01…9bc529` |
| `qwen3_8_flash_next.ninfer.part-0001` | 49,999,998,976 | `8a115623…02ad05` |
| `qwen3_8_flash_next.ninfer.part-0002` | 9,593,264,896 | `a32e3c12…fff472` |
| `qwen3_8_flash_next.ninfer.conversion.json` | conversion report (format of every tensor) | |
| `SHA256SUMS` | full checksums | |

**Requires NInfer-Offload v0.1.1 or newer; MTP needs v0.3.0 or newer.** Older versions load the
file and ignore the MTP predictor. v0.1.0 cannot map the PLE table across container files.

Updated 2026-10-02: the MTP predictor was added (1.5 GB); the text model is unchanged. The previous
text-only files are replaced.

```bash
modelscope download --model mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload --local_dir flashnext
cd flashnext && sha256sum -c SHA256SUMS
```

## Run

Download the prebuilt Linux binaries from the
[GitHub release](https://github.com/1872183316/ninfer-offload/releases) (RTX 40-series), or build
from source. The simplest way downloads this model, plans GPU/CPU placement and opens a chat:

```bash
./ninfer-run Qwen/Qwen3.8-Flash-Next
```

Or run it directly (`ninfer-serve` takes the same options and serves an OpenAI/Anthropic API):

```bash
./ninfer qwen3_8_flash_next.ninfer --prompt "Hello" --no-thinking \
  --max-context 8192 --kv-capacity 8192 --spec mtp --draft-tokens 1 \
  --moe-offload --moe-threads 12 --moe-gpu-experts 64 --moe-expert-stats stats/stats_flash_next.txt
```

`--moe-gpu-experts 64` keeps the 64 most frequently routed experts of each layer on the GPU (about
14.3 GiB VRAM); use 0–32 on GPUs with less free memory. `--moe-threads` is usually the number of
physical CPU cores.

## Requirements

- NVIDIA RTX 40-series (sm_89, prebuilt binaries) or RTX 5090 (sm_120a, build from source).
  RTX 30-series and older are not supported.
- x86-64 CPU with AVX2, FMA, F16C and BMI2.
- About 80 GB of RAM (69 GB of routed experts are held in host memory); decode speed scales with
  memory bandwidth.
- Linux. Only one machine has been tested (below).

## Quantization

| Tensors | Format |
|---|---|
| Routed experts gate/up | 4-bit, group 64, FP16 scales |
| Routed experts down | 5-bit, group 64 |
| PLE n-gram embedding table | 5-bit |
| Token embedding | 8-bit |
| Output head | 6-bit |
| Other projections (attention, GatedDeltaNet, shared expert, hyper-connections) | 8-bit, group 32 |
| Router, shared-expert score, GatedDeltaNet a/b, norms | BF16 |

Routed experts average 4.58 bits/weight (69.2 GB). The exact format of every tensor is listed in
`qwen3_8_flash_next.ninfer.conversion.json`. The artifact was produced with
`python -m tools.convert --recipe qwen3_8_flash_next --components text,mtp` from the official BF16
checkpoint; the MTP predictor uses the same formats.

## Validation and speed

- On a 59-token English paragraph, NInfer's mean NLL is 1.44330 (PPL 4.2346) versus 1.44474
  (PPL 4.2407) from an independent FP32 reference over the same stored weights
  (`tools/validate/qwen4exp_reference.py`). In a teacher-forced greedy check, 15 of 16 tokens
  matched the reference argmax; the mismatch was a near-tie (0.29 nats).
- No downstream benchmark or comparison with the BF16 model has been run.
- Measured on a Xeon E5-2673 v3 (12 cores, ~21 GB/s memory read), 94 GB DDR3, RTX 4060 Ti 16 GB
  (PCIe 3.0 x8), greedy decode, BF16 KV, 4 prompts x 2 repetitions: 10.0–10.2 tok/s with 0 GPU
  experts per layer (5.2 GB VRAM), 12.5–14.1 tok/s with 64; prefill 22–35 tok/s.
- With MTP and one draft per round (`ninfer-serve`, 64 GPU experts) the four prompts decode at
  14.7–16.6 tok/s (3–22% faster than without MTP); two or three drafts help code (up to +39%) but
  slow down free-form prose. Draft acceptance ranged from 59% (essay) to 95% (code edit).

## Limitations

- Contexts above 2051 tokens use the model's QSA token selection and need `--kv-dtype bf16` (the
  default) and text-only input.
- Greedy output with MTP is not bit-identical to greedy output without it.

## License

The model weights are a derivative of Qwen3.8-Flash-Next and are distributed under the
**Qwen Community License 1.0**; the full text, including its copyright notice, is in `LICENSE`.
Note its conditions on products above 100M monthly active users or US$20M monthly revenue, and its
separate-license requirement for commercial Model-as-a-Service and AI Work Assistant businesses.
NInfer-Offload itself is Apache-2.0.

---

## 中文说明

这是 [Qwen/Qwen3.8-Flash-Next](https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next) 转换成的
`.ninfer` v3 模型文件，供 **[NInfer-Offload](https://github.com/1872183316/ninfer-offload)** 使用。
NInfer-Offload 把路由专家放在内存里，让这个总参数 125B、激活 6B 的 MoE 模型能在 16 GB 显卡上运行。
文件包含模型自带的 MTP 预测层，可用于投机解码。

**本文件只能用 NInfer-Offload 运行**，不是 GGUF 或 safetensors，llama.cpp、vLLM、transformers
和上游 NInfer 都无法加载。

### 下载

模型共 110 GB，以 NInfer 原生的多文件格式存放（每个文件不超过 50 GB）：入口文件
`qwen3_8_flash_next.ninfer` 加续卷 `.part-0001`、`.part-0002`。所有文件放在同一目录，运行时只传入口文件，
NInfer 会自动读取续卷，**不需要合并**。

**需要 NInfer-Offload v0.1.1 或更新版本；使用 MTP 需要 v0.3.0 或更新版本。** 旧版本也能加载本文件，
只是不使用 MTP。v0.1.0 无法跨文件映射 PLE 表。

2026-10-02 更新：加入 MTP 预测层（1.5 GB），文本模型部分不变，原来的纯文本文件已被替换。

```bash
modelscope download --model mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload --local_dir flashnext
cd flashnext && sha256sum -c SHA256SUMS
```

### 运行

从 [GitHub Release](https://github.com/1872183316/ninfer-offload/releases) 下载预编译的 Linux 程序
（RTX 40 系），或自行编译。最简单的方式是一条命令：自动下载本模型、规划显卡/CPU 分工并进入对话：

```bash
./ninfer-run Qwen/Qwen3.8-Flash-Next
```

也可以直接运行（`ninfer-serve` 参数相同，提供 OpenAI/Anthropic 接口）：

```bash
./ninfer qwen3_8_flash_next.ninfer --prompt "你好" --no-thinking \
  --max-context 8192 --kv-capacity 8192 --spec mtp --draft-tokens 1 \
  --moe-offload --moe-threads 12 --moe-gpu-experts 64 --moe-expert-stats stats/stats_flash_next.txt
```

`--moe-gpu-experts 64` 表示每层把最常用的 64 个专家放到显卡上（约需 14.3 GiB 显存），显存较少时用
0～32。`--moe-threads` 一般设为 CPU 物理核心数。

### 运行要求

- NVIDIA RTX 40 系（sm_89，有预编译程序）或 RTX 5090（sm_120a，需自行编译）。不支持 RTX 30 系及更早。
- 支持 AVX2、FMA、F16C、BMI2 的 x86-64 CPU。
- 约 80 GB 内存（其中 69 GB 是路由专家）；生成速度随内存带宽提高。
- Linux。目前只在下面这一台机器上测试过。

### 量化格式

路由专家 gate/up 为 4 位，down 为 5 位（分组 64，FP16 缩放），平均 4.58 位/权重，共 69.2 GB；
PLE n-gram 表 5 位；词嵌入 8 位；输出头 6 位；其他投影 8 位；路由器、归一化等为 BF16。
每个张量的格式见 `qwen3_8_flash_next.ninfer.conversion.json`。MTP 预测层使用同样的格式。

### 验证与速度

- 59 个 token 的英文段落上，NInfer 平均 NLL 为 1.44330（PPL 4.2346），独立 FP32 参考实现为
  1.44474（PPL 4.2407）。逐 token 贪心对比 16 个中 15 个一致，唯一不同处两个候选几乎并列（相差 0.29 nats）。
- 尚未做下游评测，也没有和 BF16 原版对比质量。
- 测试机：Xeon E5-2673 v3（12 核，内存实测读带宽约 21 GB/s）、94 GB DDR3、RTX 4060 Ti 16 GB
  （PCIe 3.0 x8）。贪心解码、BF16 KV、4 个提示各跑 2 次：每层 0 个 GPU 专家时 10.0–10.2 tok/s
  （显存 5.2 GB），64 个时 12.5–14.1 tok/s；预填充 22–35 tok/s。
- 开启 MTP、每轮 1 个草稿（`ninfer-serve`，64 个 GPU 专家）时，4 个提示词解码 14.7–16.6 tok/s，
  比不开 MTP 快 3%～22%；2～3 个草稿对代码更快（最多 +39%），写散文会变慢。草稿猜对率从 59%（作文）
  到 95%（改代码）。

### 限制

- 上下文超过 2051 个 token 时使用模型自带的 QSA token 选择，需要 `--kv-dtype bf16`（默认）且只支持纯文本输入。
- 开启 MTP 后的贪心输出与不开时不完全一致。

### 许可证

模型权重是 Qwen3.8-Flash-Next 的衍生作品，按 **Qwen Community License 1.0** 分发，完整条款（含版权声明）
见 `LICENSE`。请注意其中对月活超过 1 亿或月收入超过 2000 万美元产品的要求，以及商业模型服务（MaaS）
和 AI 工作助手业务需另行取得授权的规定。NInfer-Offload 程序本身采用 Apache-2.0。