--- title: DeepSeek-V3.1-AWQ-W4AFP8 canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/DeepSeek-V3.1-AWQ-W4AFP8" md_url: "https://www.modelscope.cn/models/AI-ModelScope/DeepSeek-V3.1-AWQ-W4AFP8.md" repository: AI-ModelScope/DeepSeek-V3.1-AWQ-W4AFP8 last_updated: 2025-08-28 license: mit model_type: - deepseek_v3 architectures: - DeepseekV3ForCausalLM parameters: 96.7B tensor_type: - I32 - F16 - BF16 library_name: - safetensors - pytorch frameworks: - pytorch inference_backends: - "deploy_task text/emb" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 326 stars: 0 --- # DeepSeek-V3.1-AWQ-W4AFP8 > DeepSeek-V3.1-AWQ-W4AFP8 - AI-ModelScope 在 ModelScope 开源的模型。DeepSeek-V3.1-AWQ-W4AFP8 AI-ModelScope/DeepSeek-V3.1-AWQ-W4AFP8 是 ModelScope 魔搭社区上的 96.7B 参数机器学习模型，采用 mit 许可，可用 deploy_task text/emb、sglang 0.5.2、vllm 0.9.2 部署。 - **Repository**: AI-ModelScope/DeepSeek-V3.1-AWQ-W4AFP8 - **License**: mit - **Parameters**: 96.7B - **Inference backends**: deploy_task text/emb, sglang 0.5.2, vllm 0.9.2 - **Downloads**: 326 - **Stars**: 0 - **Last updated**: 2025-08-28 Source: https://www.modelscope.cn/models/AI-ModelScope/DeepSeek-V3.1-AWQ-W4AFP8 --- # DeepSeek-V3.1-AWQ-W4AFP8 This model is a W4AFP8 quantized DeepSeek-V3.1 with AWQ quantizaton. Releated PR：https://github.com/sgl-project/sglang/pull/8573 Releated Project: https://github.com/TMElyralab/sglang/tree/lyra_w4afp8 ## Benchmark Test configuration: input/output len = 1000/1000, qps=64, max_concurrency=64, num_prompt=128 Device: H20 * 8 Compared to the original model： - bs=64，input/output throughput has increased by 56%. - bs=128，input/output throughput has increased by 125%. ``` ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 64 Successful requests: 128 Benchmark duration (s): 105.50 Total input tokens: 128000 Total generated tokens: 128000 Total generated tokens (retokenized): 127551 Request throughput (req/s): 1.21 Input token throughput (tok/s): 1213.24 Output token throughput (tok/s): 1213.24 Total token throughput (tok/s): 2426.49 Concurrency: 63.97 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 52728.31 Median E2E Latency (ms): 52728.33 ---------------Time to First Token---------------- Mean TTFT (ms): 5444.26 Median TTFT (ms): 5425.69 P99 TTFT (ms): 8768.54 ---------------Inter-Token Latency---------------- Mean ITL (ms): 47.33 Median ITL (ms): 44.18 P95 ITL (ms): 46.58 P99 ITL (ms): 46.76 Max ITL (ms): 7819.3 ================================================== ``` ## How To Use Mirror：lmsysorg/sglang:v0.4.6.post5-cu124 or lmsysorg/sglang:v0.5.1.post5-cu126 (cuda12.6 env need to update ptxas to 12.8 on Hopper. [reference](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/README.md)) ``` # 1. Repo clone git clone https://github.com/TMElyralab/sglang && cd sglang git checkout lyra_w4afp8 # 2. SGLang install pip install uv cmake pip install -e "python[all]" # 3. Recompile sgl-kernel cd sgl-kernel & make build # 4. Run SGLang python3 -m sglang.launch_server --model-path /path/to/DeepSeek-R1-AWQ-W4AFP8 --tp 8 --trust-remote-code --host 0.0.0.0 --port 8000 --mem-fraction-static 0.9 --quantization w4a8_machete --cuda-graph-max-bs 128 --max-running-requests 128 ``` ## Citation We are TMElyralab, the Acceleration Team from Tencent Music Entertainment (TME). ``` @Misc{TMElyralab_2025, author = {Sa Xiao, Mian Peng, Haoxiong Su, Kangjian Wu, Bin Wu, Yibo Lu, Qiwen Mao, Wenjiang Zhou}, howpublished = {\url{https://github.com/TMElyralab}}, year = {2025} } ``` ------ # DeepSeek-V3.1
DeepSeek-V3
Homepage Chat Hugging Face
Discord Wechat Twitter Follow
License
## Introduction DeepSeek-V3.1 is a hybrid model that supports both thinking mode and non-thinking mode. Compared to the previous version, this upgrade brings improvements in multiple aspects: - **Hybrid thinking mode**: One model supports both thinking mode and non-thinking mode by changing the chat template. - **Smarter tool calling**: Through post-training optimization, the model's performance in tool usage and agent tasks has significantly improved. - **Higher thinking efficiency**: DeepSeek-V3.1-Think achieves comparable answer quality to DeepSeek-R1-0528, while responding more quickly. DeepSeek-V3.1 is post-trained on the top of DeepSeek-V3.1-Base, which is built upon the original V3 base checkpoint through a two-phase long context extension approach, following the methodology outlined in the original DeepSeek-V3 report. We have expanded our dataset by collecting additional long documents and substantially extending both training phases. The 32K extension phase has been increased 10-fold to 630B tokens, while the 128K extension phase has been extended by 3.3x to 209B tokens. Additionally, DeepSeek-V3.1 is trained using the UE8M0 FP8 scale data format to ensure compatibility with microscaling data formats. ## Model Downloads
| **Model** | **#Total Params** | **#Activated Params** | **Context Length** | **Download** | | :------------: | :------------: | :------------: | :------------: | :------------: | | DeepSeek-V3.1-Base | 671B | 37B | 128K | [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Base) \| [ModelScope](https://modelscope.cn/models/deepseek-ai/DeepSeek-V3.1-Base) | | DeepSeek-V3.1 | 671B | 37B | 128K | [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) \| [ModelScope](https://modelscope.cn/models/deepseek-ai/DeepSeek-V3.1) |
## Chat Template The details of our chat template is described in `tokenizer_config.json` and `assets/chat_template.jinja`. Here is a brief description. ### Non-Thinking #### First-Turn Prefix: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜>` With the given prefix, DeepSeek V3.1 generates responses to queries in non-thinking mode. Unlike DeepSeek V3, it introduces an additional token ``. #### Multi-Turn Context: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜>{response}<｜end▁of▁sentence｜>...<｜User｜>{query}<｜Assistant｜>{response}<｜end▁of▁sentence｜>` Prefix: `<｜User｜>{query}<｜Assistant｜>` By concatenating the context and the prefix, we obtain the correct prompt for the query. ### Thinking #### First-Turn Prefix: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜>` The prefix of thinking mode is similar to DeepSeek-R1. #### Multi-Turn Context: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜>{response}<｜end▁of▁sentence｜>...<｜User｜>{query}<｜Assistant｜>{response}<｜end▁of▁sentence｜>` Prefix: `<｜User｜>{query}<｜Assistant｜>` The multi-turn template is the same with non-thinking multi-turn chat template. It means the thinking token in the last turn will be dropped but the `` is retained in every turn of context. ### ToolCall Toolcall is supported in non-thinking mode. The format is: `<｜begin▁of▁sentence｜>{system prompt}{tool_description}<｜User｜>{query}<｜Assistant｜>` where the tool_description is ``` ## Tools You have access to the following tools: ### {tool_name1} Description: {description} Parameters: {json.dumps(parameters)} IMPORTANT: ALWAYS adhere to this exact format for tool use: <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>tool_call_name<｜tool▁sep｜>tool_call_arguments<｜tool▁call▁end｜>{{additional_tool_calls}}<｜tool▁calls▁end｜> Where: - `tool_call_name` must be an exact match to one of the available tools - `tool_call_arguments` must be valid JSON that strictly follows the tool's Parameters Schema - For multiple tool calls, chain them directly without separators or spaces ``` ### Code-Agent We support various code agent frameworks. Please refer to the above toolcall format to create your own code agents. An example is shown in `assets/code_agent_trajectory.html`. ### Search-Agent We design a specific format for searching toolcall in thinking mode, to support search agent. For complex questions that require accessing external or up-to-date information, DeepSeek-V3.1 can leverage a user-provided search tool through a multi-turn tool-calling process. Please refer to the `assets/search_tool_trajectory.html` and `assets/search_python_tool_trajectory.html` for the detailed template. ## Evaluation | Category | Benchmark (Metric) | DeepSeek V3.1-NonThinking | DeepSeek V3 0324 | DeepSeek V3.1-Thinking | DeepSeek R1 0528 |----------|----------------------------------|-----------------|---|---|---| | General | | | MMLU-Redux (EM) | 91.8 | 90.5 | 93.7 | 93.4 | | MMLU-Pro (EM) | 83.7 | 81.2 | 84.8 | 85.0 | | GPQA-Diamond (Pass@1) | 74.9 | 68.4 | 80.1 | 81.0 | | Humanity's Last Exam (Pass@1) | - | - | 15.9 | 17.7 |Search Agent| | | BrowseComp | - | - | 30.0 | 8.9 | | BrowseComp_zh | - | - | 49.2 | 35.7 | | Humanity's Last Exam (Python + Search) |- | - | 29.8 | 24.8 | | SimpleQA | - | - | 93.4 | 92.3 | Code | | | LiveCodeBench (2408-2505) (Pass@1) | 56.4 | 43.0 | 74.8 | 73.3 | | Codeforces-Div1 (Rating) | - | - | 2091 | 1930 | | Aider-Polyglot (Acc.) | 68.4 | 55.1 | 76.3 | 71.6 | Code Agent| | | SWE Verified (Agent mode) | 66.0 | 45.4 | - | 44.6 | | SWE-bench Multilingual (Agent mode) | 54.5 | 29.3 | - | 30.5 | | Terminal-bench (Terminus 1 framework) | 31.3 | 13.3 | - | 5.7 | Math | | | AIME 2024 (Pass@1) | 66.3 | 59.4 | 93.1 | 91.4 | | AIME 2025 (Pass@1) | 49.8 | 51.3 | 88.4 | 87.5 | | HMMT 2025 (Pass@1) | 33.5 | 29.2 | 84.2 | 79.4 | Note: - Search agents are evaluated with our internal search framework, which uses a commercial search API + webpage filter + 128K context window. Seach agent results of R1-0528 are evaluated with a pre-defined workflow. - SWE-bench is evaluated with our internal code agent framework. - HLE is evaluated with the text-only subset. ### Usage Example ```python import transformers tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3.1") messages = [ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Who are you?"}, {"role": "assistant", "content": "HmmI am DeepSeek"}, {"role": "user", "content": "1+1=?"} ] tokenizer.apply_chat_template(messages, tokenize=False, thinking=True, add_generation_prompt=True) # '<｜begin▁of▁sentence｜>You are a helpful assistant<｜User｜>Who are you?<｜Assistant｜>I am DeepSeek<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜>' tokenizer.apply_chat_template(messages, tokenize=False, thinking=False, add_generation_prompt=True) # '<｜begin▁of▁sentence｜>You are a helpful assistant<｜User｜>Who are you?<｜Assistant｜>I am DeepSeek<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜>' ``` ## How to Run Locally The model structure of DeepSeek-V3.1 is the same as DeepSeek-V3. Please visit [DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3) repo for more information about running this model locally. ## License This repository and the model weights are licensed under the [MIT License](LICENSE). ## Citation ``` @misc{deepseekai2024deepseekv3technicalreport, title={DeepSeek-V3 Technical Report}, author={DeepSeek-AI}, year={2024}, eprint={2412.19437}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2412.19437}, } ``` ## Contact If you have any questions, please raise an issue or contact us at [service@deepseek.com](service@deepseek.com).