--- title: DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken canonical_url: "https://www.modelscope.cn/models/T-HEAD/DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken" md_url: "https://www.modelscope.cn/models/T-HEAD/DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken.md" repository: T-HEAD/DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken last_updated: 2026-08-24 license: mit model_type: - deepseek_v4 architectures: - DeepseekV4ForCausalLM base_model: - deepseek-ai/DeepSeek-V4-Pro base_model_relation: quantized library_name: - safetensors - pytorch frameworks: - Pytorch downloads: 7917 stars: 1 --- # DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken > DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken - T-HEAD 在 ModelScope 开源的模型。📢 Note: This repository was renamed from DeepSeek-V4-Pro-W8A8-INT8 on 2026-08-24. The weights are unchanged; only the model name has been updated. T-HEAD/DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken 是 ModelScope 魔搭社区上的机器学习模型，采用 mit 许可，基于 deepseek-ai/DeepSeek-V4-Pro 构建。 - **Repository**: T-HEAD/DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken - **License**: mit - **Base model**: deepseek-ai/DeepSeek-V4-Pro - **Downloads**: 7917 - **Stars**: 1 - **Last updated**: 2026-08-24 Source: https://www.modelscope.cn/models/T-HEAD/DeepSeek-V4-Pro-Quant-W-INT8-PerChannel-A-INT8-PerToken --- 📢 Note: This repository was renamed from DeepSeek-V4-Pro-W8A8-INT8 on 2026-08-24. The weights are unchanged; only the model name has been updated. # Model Overview - **Model Architecture:** DeepseekV4ForCausalLM - **Input:** Text - **Output:** Text - **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/)/[SGLang](https://docs.sglang.io/) - **Quantization:** - **moe:** `w1`, `w2`, `w3` - **Weight quantization:** INT8, Static, Per-Channel - **Activation quantization:** INT8, Dynamic, Per-Token - **shared_experts:** `w1`, `w2`, `w3` - **Weight quantization:** INT8, Static, Per-Channel - **Activation quantization:** INT8, Dynamic, Per-Token - **self_attn:** `wq_a`, `wq_b`, `wkv`, `wo_a`, `wo_b`, `indexer.wq_b` - **Weight quantization:** INT8, Static, Per-Channel - **Activation quantization:** INT8, Dynamic, Per-Token - **mtp:** `e_proj`, `h_proj` - **Weight quantization:** INT8, Static, Per-Channel - **Activation quantization:** INT8, Dynamic, Per-Token This model was built with DeepSeek-V4-Pro model by applying W8A8-INT8 quantization. # Deployment ### Use with vLLM This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend. ## Evaluation The model was evaluated on IFEval GSM8K C-EVAL MMLU-Pro benchmarks. ### Accuracy
Benchmark DeepSeek-V4-Pro DeepSeek-V4-Pro-W8A8-INT8 (this model) Recovery
IFEval 89.31 89.8 100.55%
GSM8K 96.59 95.9 99.29%
C-Eval 91.01 91 99.99%
MMLU-Pro 82.55 82.3 99.7%
### Reproduction The IFEval GSM8K C-EVAL MMLU-Pro results were obtained using the `EvalScope` and `vLLM` framework. Install T-head vLLM `(Version: 0.20.1)` and EvalScope `(Version: 1.5.1)` in container first. #### Launching server ``` MODEL_PATH= VLLM_USE_MODELSCOPE=true vllm serve $MODEL_PATH \ --tensor-parallel-size 16 \ --trust-remote-code \ --served-model-name DeepSeek-V4-Pro \ --kv-cache-dtype fp8 \ --data-parallel-size 2 --nnodes 2 --node-rank 0 \ --master-addr \ --enable-prefix-caching VLLM_USE_MODELSCOPE=true vllm serve $MODEL_PATH \ --tensor-parallel-size 16 \ --trust-remote-code \ --served-model-name DeepSeek-V4-Pro \ --kv-cache-dtype fp8 \ --data-parallel-size 2 --nnodes 2 --node-rank 1 \ --master-addr --headless \ --enable-prefix-caching ``` #### Evaluating model in a new terminal ``` evalscope eval \ --model DeepSeek-V4-Pro --api-url http://127.0.0.1:8000/v1 --api-key EMPTY \ --eval-type openai_api --datasets gsm8k \ --dataset-args '{"gsm8k": {"few_shot_num": 0, "filters": {"remove_until": ""}}}' \ --eval-batch-size 32 \ --generation-config '{"max_tokens": 32768, "temperature": 1, "top_p": 1, "n": 1}' \ --timeout 60000 --stream --work-dir outputs ``` ### Use with SGLang This model can be deployed efficiently using the [SGLang](https://docs.sglang.io/) backend. ## Evaluation The model was evaluated on IFEval GSM8K C-EVAL MMLU-Pro benchmarks. ### Accuracy
Benchmark DeepSeek-V4-Pro DeepSeek-V4-Pro-W8A8-INT8 (this model) Recovery
IFEval 88.19 89 100.91%
GSM8K 96.36 96 99.63%
C-Eval 89.97 90.1 100.14%
MMLU-Pro 82.94 82.7 99.7%
### Reproduction The IFEval GSM8K C-EVAL MMLU-Pro results were obtained using the `EvalScope` and `SGLang` framework. Install T-head SGLang `(Version: 0.5.12)` and EvalScope `(Version: 1.5.1)` in container first. #### Launching server ``` MODEL_PATH= SGLANG_USE_MODELSCOPE=true python3 -m sglang.launch_server --dist-init-addr --nnodes 2 --node-rank 0 \ --model-path $MODEL_PATH \ --tp-size 32 \ --mem-fraction-static 0.8 \ --speculative-algorithm EAGLE --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ --enable-dp-attention --dp-size 32 \ --quantization w8a8_int8 --trust-remote-code \ --fp8-gemm-backend triton --watchdog-timeout 60000 --soft-watchdog-timeout 60000 \ --dist-timeout 60000 --served-model-name DeepSeek-V4-Pro \ --speculative-attention-mode decode --cuda-graph-max-bs 64 \ --disable-custom-all-reduce --disable-piecewise-cuda-graph --disable-shared-experts-fusion SGLANG_USE_MODELSCOPE=true python3 -m sglang.launch_server --dist-init-addr --nnodes 2 --node-rank 1 \ --model-path $MODEL_PATH \ --tp-size 32 \ --mem-fraction-static 0.8 \ --speculative-algorithm EAGLE --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ --enable-dp-attention --dp-size 32 \ --quantization w8a8_int8 --trust-remote-code \ --fp8-gemm-backend triton --watchdog-timeout 60000 --soft-watchdog-timeout 60000 \ --dist-timeout 60000 --served-model-name DeepSeek-V4-Pro \ --speculative-attention-mode decode --cuda-graph-max-bs 64 \ --disable-custom-all-reduce --disable-piecewise-cuda-graph --disable-shared-experts-fusion ``` #### Evaluating model in a new terminal ``` evalscope eval \ --model DeepSeek-V4-Pro --api-url http://127.0.0.1:8000/v1 --api-key EMPTY \ --eval-type openai_api --datasets gsm8k \ --dataset-args '{"gsm8k": {"few_shot_num": 0, "filters": {"remove_until": ""}}}' \ --eval-batch-size 64 \ --generation-config '{"max_tokens": 32768, "temperature": 1, "top_p": 1, "n": 1}' \ --timeout 60000 --stream --work-dir outputs ``` # License Copyright (c) 2026, T-Head (Shanghai) Semiconductor Co., Ltd.