---
title: Qwen3.8-27B-NVFP4-DGX-SPARK-V2
canonical_url: "https://www.modelscope.cn/models/autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2"
md_url: "https://www.modelscope.cn/models/autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2.md"
repository: autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2
last_updated: 2026-10-28
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen3_5
architectures:
  - Qwen3_5ForConditionalGeneration
parameters: 15.6B
tensor_type:
  - BF16
  - F32
  - F8_E4M3
  - U8
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
  - zh
downloads: 36
stars: 0
tags:
  - nvfp4
  - fp4
  - modelopt
  - mtp
  - speculative-decoding
  - vllm
  - blackwell
  - b200
  - dgx-spark
  - qwen3.8
---

# Qwen3.8-27B-NVFP4-DGX-SPARK-V2

> Qwen3.8-27B-NVFP4-DGX-SPARK-V2 - autotrust 在 ModelScope 开源的模型。Qwen3.8-27B-NVFP4-DGX-SPARK-V2 + 64k MTP

autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2 是 ModelScope 魔搭社区上的 15.6B 参数text-generation模型，采用 apache-2.0 许可。

- **Repository**: autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 15.6B
- **Tags**: nvfp4, fp4, modelopt, mtp, speculative-decoding, vllm, blackwell, b200, dgx-spark, qwen3.8
- **Downloads**: 36
- **Stars**: 0
- **Last updated**: 2026-10-28

Source: https://www.modelscope.cn/models/autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2

---

# Qwen3.8-27B-NVFP4-DGX-SPARK-V2 + 64k MTP

This repository combines:

- the complete **Qwen3.8-27B NVFP4** checkpoint, including its MTP tensors;
- the **64k-vocabulary MTP recipe** from `autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK`.

The 64k MTP optimization is **not baked into the safetensors weights**. It is a
runtime patch that must be loaded by the included vLLM plugin or by the offline
Python patch.

## What the 64k MTP patch does

- Restricts the draft head from 248,320 vocabulary rows to **64,026** kept rows.
- Pads the draft head to **64,064** rows for the quantized GEMM kernel.
- Gathers the draft head from the target `lm_head` and quantizes it at runtime
  to **Marlin NVFP4 W4A16**.
- Quantizes the five MTP linear projections at runtime to Marlin NVFP4.
- Leaves the target model and the verification `lm_head` untouched.

The result is a much smaller draft step, allowing larger speculative token
counts. The verified starting point is `num_speculative_tokens: 7` on B200 and
`8` on DGX Spark.

> **Important:** 64k MTP requires the included plugin. Running a plain
> `vllm serve` command with only `--speculative-config` uses native MTP, not the
> 64k/Marlin draft path.

## Requirements

- vLLM **0.31.0** (the plugin uses vLLM internals and may break on other versions)
- Blackwell GPU: B200 (`sm_100`) verified; DGX Spark / GB10 (`sm_120/121`) is the
  recipe's target platform
- Python 3.10+
- Enough memory for the 20.6 GB checkpoint plus KV cache
- PyTorch and CUDA matching the vLLM 0.31.0 build

This MTP patch is for vLLM. It is not an SGLang or TensorRT-LLM plugin.

## 1. Download the model

```bash
export MODEL_DIR="$HOME/models/Qwen3.8-27B-NVFP4-DGX-SPARK-V2"

HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2 \
  --local-dir "$MODEL_DIR"

cd "$MODEL_DIR"
```

## 2. Install vLLM and the MTP plugin

Use a clean virtual environment:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip

pip install "vllm==0.31.0"
pip install -e ./mtp64k/plugin
```

Verify that the plugin is installed:

```bash
python -c "import spark_mtp_draft; print(spark_mtp_draft.__file__)"
```

## 3. Set the 64k MTP environment

```bash
export SPARK_DRAFT_HOME="$PWD/mtp64k/scripts"
export SPARK_DRAFT_VOCAB="$PWD/mtp64k/artifacts/draft_vocab_64k.pt"
export SPARK_DRAFT_HEAD_Q=marlin
export SPARK_DRAFT_MTP_Q=marlin
```

`SPARK_DRAFT_VOCAB` is also the activation switch. If it is empty, the plugin
stays inactive. `SPARK_DRAFT_HOME` can be omitted when the plugin was installed
with `pip install -e ./mtp64k/plugin` from this repository layout, but setting
it explicitly is recommended.

## 4. Serve with vLLM

### B200

```bash
vllm serve "$MODEL_DIR" \
  --served-model-name qwen3.8-27b-nvfp4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":7,"use_local_argmax_reduction":true}' \
  --gpu-memory-utilization 0.60 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3
```

### DGX Spark / GB10

Use the same command but start with `num_speculative_tokens: 8`:

```bash
vllm serve "$MODEL_DIR" \
  --served-model-name qwen3.8-27b-nvfp4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":8,"use_local_argmax_reduction":true}' \
  --gpu-memory-utilization 0.60 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3
```

`0.60` GPU memory utilization is the safe starting point on unified-memory
Spark systems. On a discrete B200, raise it only after confirming that the
model and KV cache fit. Increase `--max-model-len` only if the deployment has
enough KV-cache memory.

### Confirm that 64k MTP is active

The server log must contain a line like:

```text
[real-draft] head rows=64064 head_q=marlin mtp_q=marlin layers=5
```

If that line is absent, the plugin did not activate. The model may still start
with native MTP, but the 64k draft vocabulary and Marlin draft head are not in
use.

## 5. Send an OpenAI-compatible request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.8-27b-nvfp4",
    "messages": [{"role": "user", "content": "Write a Python LRU cache."}],
    "temperature": 0,
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

Thinking mode is enabled by the chat template by default. Disable it per request
with `chat_template_kwargs.enable_thinking: false`.

Keep client `max_tokens` below the served `--max-model-len`; the prompt and
completion must fit in the same context.

## Offline Python usage

For the in-process vLLM API, apply the patch before creating `LLM`, and disable
vLLM V1 multiprocessing so the patch is present in the engine process:

```python
import os
import sys

os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
sys.path.insert(0, "./mtp64k/scripts")

import real_draft
real_draft.install(
    "./mtp64k/artifacts/draft_vocab_64k.pt",
    "marlin",
    "marlin",
)

from vllm import LLM, SamplingParams

llm = LLM(
    model=".",
    speculative_config={
        "method": "mtp",
        "num_speculative_tokens": 7,
        "use_local_argmax_reduction": True,
    },
    gpu_memory_utilization=0.60,
    max_model_len=32768,
    kv_cache_dtype="fp8",
)

out = llm.chat(
    [{"role": "user", "content": "Reply with exactly: OK"}],
    SamplingParams(temperature=0, max_tokens=16),
)
print(out[0].outputs[0].text)
```

Run this from the downloaded repository root.

## Tuning notes

- **B200:** start at `num_speculative_tokens: 7`.
- **DGX Spark / GB10:** start at `num_speculative_tokens: 8`.
- **Sampling temperature above ~0.7:** reduce to roughly `4–6` speculative
  tokens and re-benchmark; higher entropy lowers draft acceptance.
- **Quantization choices:** `marlin` is the verified default. The scripts also
  expose `cutlass`, `fp8`, and `bf16`; use them only after testing on your exact
  vLLM build.
- **Context length:** `32768` is the conservative starting point. Raising it
  increases KV-cache memory; `--kv-cache-dtype fp8` is recommended.
- **DGX Spark OOM:** keep `--gpu-memory-utilization 0.60`, lower
  `--max-model-len`, or reduce concurrency.

## Repository layout

```text
.
├── config.json / tokenizer* / chat_template.jinja
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model-mtp-fc.safetensors
├── model.safetensors.index.json
├── mtp64k/
│   ├── artifacts/
│   │   └── draft_vocab_64k.pt
│   ├── scripts/
│   │   ├── real_draft.py
│   │   ├── nvfp4_ops.py
│   │   └── spark_mtp_draft.py
│   ├── plugin/
│   │   ├── pyproject.toml
│   │   └── spark_mtp_draft.py
│   └── manifest.json
├── LICENSE
└── README.md
```

`mtp64k/manifest.json` contains the packaged file hashes and recommended
runtime settings.

## Troubleshooting

### The server starts but there is no `[real-draft]` line

Check all of the following:

```bash
python -c "import spark_mtp_draft; print(spark_mtp_draft.__file__)"
test -f "$SPARK_DRAFT_VOCAB" && echo "vocab exists"
```

Then restart the server with the environment from section 3. Environment
variables must be inherited by the vLLM engine process.

### MTP runs but there is no expected speedup

- Confirm `head rows=64064` appears in the log.
- Confirm `use_local_argmax_reduction: true` is present in the speculative
  config.
- Try a lower temperature and re-benchmark.
- Compare with a single stream; batching changes the relative benefit.

### CUDA driver error 803

On the B200 host used for validation, the host driver needed the system CUDA
driver library to be preloaded:

```bash
export LD_PRELOAD=/lib/x86_64-linux-gnu/libcuda.so.1
```

This is host-specific. Do not apply it blindly if your driver stack does not
need it.

## Validation

The packaged repository was served directly from its download directory with
vLLM 0.31.0 on a B200. The MTP patch loaded successfully:

```text
[real-draft] head rows=64064 head_q=marlin mtp_q=marlin layers=5
```

An OpenAI-compatible smoke request returned `OK` with `finish_reason=stop`.

## License and sources

Apache-2.0. See [`LICENSE`](LICENSE).

- 64k MTP recipe: [`autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK`](https://huggingface.co/autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK)
- Qwen3.8 model family: [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B)