--- title: Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 canonical_url: "https://www.modelscope.cn/models/manateelazycat/Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659" md_url: "https://www.modelscope.cn/models/manateelazycat/Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659.md" repository: manateelazycat/Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 last_updated: 2026-09-21 license: other pipeline_tag: image-text-to-text tasks: - image-text-to-text model_type: - qwen4_exp architectures: - Qwen4ExpForConditionalGeneration base_model: - windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 base_model_relation: quantized parameters: 119.6B tensor_type: - F32 - I64 - F8_E4M3 - BF16 - U8 library_name: - safetensors downloads: 249 stars: 0 tags: - nvfp4 - fp4 - fp8 - modelopt - vllm - abliterated - uncensored - qwen4_exp - qwen3.8 - flash-next - moe - dgx-spark --- # Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 > Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 - manateelazycat 在 ModelScope 开源的模型。Qwen3.8-Flash-Next NVFP4 PLE-FP8 snapshot, revision 2065365912e46b205c64a70ac5b85b4869674d31. Upstream README and LICENSE are preserved without modification. manateelazycat/Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 是 ModelScope 魔搭社区上的 119.6B 参数image-text-to-text模型，采用 other 许可，基于 windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 构建。 - **Repository**: manateelazycat/Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 - **License**: other - **Tasks**: image-text-to-text - **Parameters**: 119.6B - **Base model**: windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 - **Tags**: nvfp4, fp4, fp8, modelopt, vllm, abliterated, uncensored, qwen4_exp, qwen3.8, flash-next, moe, dgx-spark - **Downloads**: 249 - **Stars**: 0 - **Last updated**: 2026-09-21 Source: https://www.modelscope.cn/models/manateelazycat/Qwen3.8-Flash-Next-NVFP4-PLEFP8-20653659 --- # Qwen3.8-Flash-Next-Abliterated-NVFP4-plefp8 [windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4](https://huggingface.co/windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4) with its 51.2B PLE n-gram embedding table **stored in FP8 instead of BF16**, and its PLE tensors **re-laid-out into the shard form vLLM's loader expects**. **173.6 GiB → 125.9 GiB.** Nothing else is touched: the NVFP4 body, the abliteration, the vision tower and the MTP draft head are all copied through byte-for-byte. On 2× DGX Spark (TP2) that is 88.63 → 64.79 GiB per node, and the freed memory goes almost entirely to KV cache. ## Why the table dominates the file size Qwen3.8-Flash-Next is 125B of body **plus a 51.2B N-gram (PLE) embedding table** living in a single layer. The table is a hashed lookup, not a matmul, so every NVFP4 quantizer excludes it from the 4-bit pass. What differs between published NVFP4 builds is therefore mostly *what dtype the excluded table is stored in*: | checkpoint | PLE table | body | total | |---|---|---|---| | [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) | FP8 | FP8 | 172.8 GiB | | [Inferact/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/Inferact/Qwen3.8-Flash-Next-NVFP4) | BF16 | NVFP4 | 170.2 GiB | | [RadixArk/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4) | FP8 | NVFP4 | 125.9 GiB | | upstream of this repo | BF16 | NVFP4 | 173.6 GiB | | **this repo** | **FP8** | NVFP4 | **125.9 GiB** | NVFP4 shrinks the experts in every 4-bit build; a BF16 table hands the gain back. ## What changed, exactly **1. PLE table split and renamed.** Upstream ships the table as one consolidated tensor: ``` model.language_model.layers.1.ple.ple_embedding.shard_.weight BF16 [320001536, 160] 95.37 GiB ``` vLLM's `ple_layer.load_weights` only recognises `ngram_embedding.shard_.weight`. That name has neither the `ngram_embedding.` component nor a shard index, so it falls through to `AutoWeightsLoader`, which has no parameter to bind it to. Here it is split into `split_ngram_parts = 128` shards of 2,500,012 rows (320,001,536 / 128) named the way the loader expects — the same shape and naming Qwen's own FP8 build uses. **2. Cast BF16 → `float8_e4m3fn`** with a single global per-tensor scale, written as `...ngram_embedding.weight_scale`, bf16, shape `(1,)` — again matching Qwen's FP8 build. The computed scale is `0.00019931793213`, which is *bit-identical* to the scale in Qwen's own FP8 checkpoint (abliteration barely moves the table's dynamic range). Round-trip vs the source table: **cosine 0.9997, max relative error ~3.5%**. **3. `layer_types` normalised.** Upstream's config (emitted by transformers 5.16.0.dev0) labels the 12 QSA layers `qwen_sparse_attention`. vLLM rejects that outright: ``` ValidationError: `layer_types` entries must be in (...) but got [... 'qwen_sparse_attention' ...] ``` The 12 layer positions are identical to Qwen's published config (`[3, 7, 11 … 47]`), so they are renamed to `full_attention`. **This fix is needed to serve the upstream checkpoint too** — it is not specific to the FP8 repack. ## Does it need a patched vLLM? **Yes — one small patch.** vLLM selects its FP8 PLE embedding method only when the *global* quantization config is `Fp8Config`: ```python def _get_ple_embedding_quant_method(quant_config, prefix): if not isinstance(quant_config, Fp8Config): return None ``` This checkpoint's global config is `ModelOptNvFp4Config` (the body is NVFP4) with `*.ple.*` in `exclude_modules`, so that returns `None`, the embedding is built in BF16, and loading the `F8_E4M3` shards fails. The patch decides from the *on-disk tensor dtype* instead, which leaves BF16-table checkpoints on the unquantized path unchanged. The patch is included as `patches/fix-qwen38fn-nvfp4-ple-fp8.sh` (idempotent, self-verifying) and is also carried in [eugr/spark-vllm-docker](https://github.com/eugr/spark-vllm-docker) as `mods/fix-qwen38fn-nvfp4-ple-fp8`. Everything else — the NVFP4 body, vision tower, MTP head — loads on a stock `qwen4_exp`-capable vLLM. ## Measured: what the FP8 table costs Both builds are identical except for the table's dtype (same weights, same shard layout, same config fix), served on 2× DGX Spark, TP2, 262144 context: | | BF16 table (control) | **FP8 table (this repo)** | |---|---|---| | checkpoint | 173.59 GiB | **125.90 GiB** | | weights / node | 88.63 GiB | **64.79 GiB** | | KV cache | 9.47 GiB | **33.20 GiB** | | KV tokens | 636,230 | **2,199,175** | | max concurrency @262k | 2.43× | **8.39×** | | decode (single stream) | 44.3 / 51.7 / 58.0 tok/s | 42.8 / 53.9 / 54.9 tok/s | | MoE backend | MARLIN | MARLIN | Quality, first-token distribution over 80 harmless prompts, BF16 build as reference: | metric | value | |---|---| | **top-1 token agreement** | **100.0%** | | mean KL | 0.0018 | | median KL | 0.0006 | | p90 KL | 0.0060 | Long-generation degeneracy (20 runs each) and tool-calling (24 cases each) show no regression: `mean_distinct_3` 0.9888 → 0.9894, `max_ngram_run` 1 → 1, `json_parse_rate` 1.00 → 1.00. **The FP8 build picks the same first token as the BF16 build on every prompt tested.** For context, the NVFP4 body quantization both builds already carry is roughly an order of magnitude lossier than this cast. ## Serving (vLLM, 2× DGX Spark) ```bash vllm serve \ --tensor-parallel-size 2 \ --max-model-len 262144 \ --max-num-seqs 8 \ --gpu-memory-utilization 0.86 \ --enable-prefix-caching \ --no-enable-flashinfer-autotune \ --compilation-config '{"mode":0,"cudagraph_mode":"FULL"}' \ --tool-call-parser qwen3_coder --enable-auto-tool-choice \ --reasoning-parser qwen3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` Notes for GB10 / DGX Spark specifically: - `mode:0` disables Inductor. With compilation on, its autotuner tries to benchmark a kernel over the whole PLE table and asks for ~47.7 GiB in one allocation, which OOMs. `cudagraph_mode: FULL` still gives graph replay. - `VLLM_PLE_CPU_OFFLOAD=1` is **single-node only** (`Unsupported settings: nnodes=2`), so it cannot be used on a 2-node cluster. On unified memory it would not help with capacity anyway — host and device are the same physical pool. - `gpu_memory_utilization` ceiling is lower than it looks: CUDA's free memory tracks the OS `free` column, not `available`, so page cache eats into the budget. ## Reproducing ```bash python3 repack_ple_fp8.py \ --src \ --dst # control build (split + rename, keep BF16): python3 repack_ple_fp8.py --src --dst --keep-bf16 ``` `repack_ple_fp8.py` is included here and in [eugr/spark-vllm-docker](https://github.com/eugr/spark-vllm-docker) under `tools/ple-repack/`. It streams the table in row slices, hardlinks every untouched shard file, and rewrites the index — it never holds the 95 GiB table in memory. ## Credits - **[Qwen](https://huggingface.co/Qwen)** — Qwen3.8-Flash-Next, the base model, and the FP8 build whose PLE layout and scale convention this reproduces. - **[windowsxp811203](https://huggingface.co/windowsxp811203)** — the abliteration and the NVFP4 conversion this repo is derived from. Their `ARCHITECTURE.md` (included) is the best documentation of `qwen4_exp` internals I have found, and their `graft_nvfp4.py` / `fix_nvfp4_config.py` document real failure modes in quantizing this architecture (silently dropped vision and MTP tensors). Both are intact here: 333 vision tensors and 31 MTP tensors verified present. - **[RadixArk](https://huggingface.co/RadixArk)** — their NVFP4 build demonstrated that an FP8 PLE table is viable, which is what prompted this repack. Only the PLE table's dtype and layout differ from the upstream abliterated NVFP4 build. All credit for the model itself belongs upstream; any mistakes in the repack are mine. ## License Qwen Community License 1.0, inherited from the base model. See `LICENSE`.