--- title: Qwen3.5-397B-A17B-DFlash canonical_url: "https://www.modelscope.cn/models/modal-labs/Qwen3.5-397B-A17B-DFlash" md_url: "https://www.modelscope.cn/models/modal-labs/Qwen3.5-397B-A17B-DFlash.md" repository: modal-labs/Qwen3.5-397B-A17B-DFlash last_updated: 2026-07-18 license: apache-2.0 pipeline_tag: text-generation tasks: - text-generation model_type: - qwen3 architectures: - DFlashDraftModel base_model: - Qwen/Qwen3.5-397B-A17B base_model_relation: finetune parameters: 1.3B tensor_type: - BF16 library_name: - safetensors - pytorch frameworks: - pytorch downloads: 65 stars: 0 tags: - dflash - speculative-decoding - speculative-decoding-draft - block-diffusion - draft-model - diffusion-language-model - efficiency - qwen - qwen3 - qwen3.5 - sglang --- # Qwen3.5-397B-A17B-DFlash > Qwen3.5-397B-A17B-DFlash - modal-labs 在 ModelScope 开源的模型。Qwen3.5-397B-A17B-DFlash modal-labs/Qwen3.5-397B-A17B-DFlash 是 ModelScope 魔搭社区上的 1.3B 参数text-generation模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.5-397B-A17B 构建。 - **Repository**: modal-labs/Qwen3.5-397B-A17B-DFlash - **License**: apache-2.0 - **Tasks**: text-generation - **Parameters**: 1.3B - **Base model**: Qwen/Qwen3.5-397B-A17B - **Tags**: dflash, speculative-decoding, speculative-decoding-draft, block-diffusion, draft-model, diffusion-language-model, efficiency, qwen, qwen3, qwen3.5, sglang - **Downloads**: 65 - **Stars**: 0 - **Last updated**: 2026-07-18 Source: https://www.modelscope.cn/models/modal-labs/Qwen3.5-397B-A17B-DFlash --- # Qwen3.5-397B-A17B-DFlash [Paper](https://arxiv.org/abs/2602.06036) | [Github](https://github.com/z-lab/dflash) | [Blog](https://z-lab.ai/projects/dflash) This is a joint release collaboration between [Z-Lab](https://z-lab.ai), [Modal](https://modal.com), and [SGLang](https://github.com/sgl-project/sglang). The draft model is mirrored across the following Hugging Face repositories: - [`z-lab/Qwen3.5-397B-A17B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-397B-A17B-DFlash) - [`modal-labs/Qwen3.5-397B-A17B-DFlash`](https://huggingface.co/modal-labs/Qwen3.5-397B-A17B-DFlash) - [`lmsys/Qwen3.5-397B-A17B-DFlash`](https://huggingface.co/lmsys/Qwen3.5-397B-A17B-DFlash) This repository contains a DFlash draft model for `Qwen/Qwen3.5-397B-A17B`. It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server. DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model's output distribution. ## Quick Start This model should be used with an inference server that supports DFlash speculative decoding. An example SGLang deployment is: ```bash export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python -m sglang.launch_server \ --model-path Qwen/Qwen3.5-397B-A17B \ --trust-remote-code \ --speculative-algorithm DFLASH \ --speculative-draft-model-path modal-labs/Qwen3.5-397B-A17B-DFlash \ --speculative-dflash-block-size 8 \ --speculative-draft-attention-backend fa4 \ --attention-backend trtllm_mha \ --linear-attn-prefill-backend triton \ --linear-attn-decode-backend flashinfer \ --mamba-scheduler-strategy extra_buffer \ --tp-size 8 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 32 \ --cuda-graph-backend-prefill tc_piecewise \ --enable-flashinfer-allreduce-fusion \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` Block size `8` is the recommended default for higher-concurrency serving. Block size `16` gives longer accept lengths and the best concurrency-1 throughput in most workloads. ## Benchmark Results We benchmarked DFlash against the autoregressive baseline and Qwen3.5's built-in MTP draft path. DFlash reaches up to `4.31x` speedup at concurrency 1 and `2.77x` at concurrency 32. Across the benchmark suite, DFlash delivers higher throughput than MTP at every tested setting. ### Setup - Runtime: SGLang on 8x NVIDIA B200 GPUs, tensor parallel size 8, `bfloat16` - Backends: `trtllm_mha` target attention, `fa4` DFlash draft attention, `flashinfer` linear attention - Workloads: GSM8K, MATH500, HumanEval, MBPP, and MT-Bench with the Qwen chat template - Decoding: greedy, thinking enabled, max output length 4096 tokens - Measurement: 5 independent runs per configuration at concurrency 1 and 32 with continuous batching - Throughput: generated output tokens / wall-clock benchmark time, including prefill and scheduling - Accept length: `completion_tokens / spec_verify_ct` per generation turn, averaged across generation turns For reproduction, the evaluation scripts, Modal runner, runtime patches, and raw benchmark outputs are included under [`benchmark/`](https://huggingface.co/modal-labs/Qwen3.5-397B-A17B-DFlash/tree/main/benchmark). ### Throughput and Speedup Each cell is `output tok/s (speedup)`. Bold marks the fastest speculative configuration in each row. #### Concurrency 1 | Workload | Baseline | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 | | --- | --- | --- | --- | --- | --- | --- | --- | | gsm8k | 204.6 (1.00x) | 484.5 (2.37x) | 530.0 (2.59x) | 536.7 (2.62x) | 689.0 (3.37x) | 439.2 (2.15x) | **711.3 (3.48x)** | | math500 | 204.4 (1.00x) | 505.2 (2.47x) | 551.3 (2.70x) | 589.4 (2.88x) | 762.6 (3.73x) | 499.8 (2.44x) | **831.9 (4.07x)** | | humaneval | 202.9 (1.00x) | 483.0 (2.38x) | 543.6 (2.68x) | 557.9 (2.75x) | 752.5 (3.71x) | 480.5 (2.37x) | **874.6 (4.31x)** | | mbpp | 204.6 (1.00x) | 487.4 (2.38x) | 550.3 (2.69x) | 543.7 (2.66x) | 751.2 (3.67x) | 445.6 (2.18x) | **807.8 (3.95x)** | | mt-bench | 202.6 (1.00x) | 442.9 (2.19x) | 473.6 (2.34x) | 441.5 (2.18x) | **545.9 (2.69x)** | 338.0 (1.67x) | 515.3 (2.54x) | #### Concurrency 32 | Workload | Baseline | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 | | --- | --- | --- | --- | --- | --- | --- | --- | | gsm8k | 2597.7 (1.00x) | 4462.3 (1.72x) | 5078.0 (1.95x) | 4902.9 (1.89x) | **6158.9 (2.37x)** | 4079.5 (1.57x) | 5648.8 (2.17x) | | math500 | 2615.5 (1.00x) | 4838.6 (1.85x) | 5318.6 (2.03x) | 5601.6 (2.14x) | **6910.5 (2.64x)** | 4855.0 (1.86x) | 6857.8 (2.62x) | | humaneval | 2452.7 (1.00x) | 3838.1 (1.56x) | 5081.4 (2.07x) | 4347.4 (1.77x) | 6666.0 (2.72x) | 3917.2 (1.60x) | **6783.7 (2.77x)** | | mbpp | 2582.3 (1.00x) | 4129.8 (1.60x) | 5265.7 (2.04x) | 4461.0 (1.73x) | **6681.4 (2.59x)** | 3783.7 (1.47x) | 6344.1 (2.46x) | | mt-bench | 2537.3 (1.00x) | 4024.6 (1.59x) | 4425.3 (1.74x) | 3991.7 (1.57x) | **4763.2 (1.88x)** | 3100.3 (1.22x) | 4016.0 (1.58x) | ### Accept Length Mean accept length at concurrency 1. Bold marks the higher value in each matched MTP/DFlash pair. | Workload | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 | | --- | --- | --- | --- | --- | --- | --- | | gsm8k | **3.578** | 3.548 | 5.612 | **5.738** | 7.006 | **7.696** | | math500 | **3.636** | 3.622 | 5.810 | **6.003** | 7.294 | **8.212** | | humaneval | 3.643 | **3.697** | 5.893 | **6.292** | 7.541 | **9.344** | | mbpp | 3.532 | **3.588** | 5.357 | **5.860** | 6.398 | **7.801** | | mt-bench | **3.244** | 3.188 | 4.567 | **4.584** | 5.256 | **5.523** | ## Citation If you find DFlash useful, please cite the original paper: ```bibtex @article{chen2026dflash, title = {{DFlash: Block Diffusion for Flash Speculative Decoding}}, author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian}, journal = {arXiv preprint arXiv:2602.06036}, year = {2026} } ```