--- title: Ling-Flash-2.0-eagle3 canonical_url: "https://www.modelscope.cn/models/AQ-MedAI/Ling-Flash-2.0-eagle3" md_url: "https://www.modelscope.cn/models/AQ-MedAI/Ling-Flash-2.0-eagle3.md" repository: AQ-MedAI/Ling-Flash-2.0-eagle3 last_updated: 2025-12-19 license: "MIT License" pipeline_tag: text-generation tasks: - text-generation model_type: - llama architectures: - LlamaForCausalLMEagle3 parameters: 424.9M tensor_type: - I64 - BF16 - BOOL library_name: - safetensors - pytorch frameworks: - Pytorch inference_backends: - "deploy_task text" - "sglang 0.5.2" downloads: 50 stars: 0 --- # Ling-Flash-2.0-eagle3 > Ling-Flash-2.0-eagle3 - AQ-MedAI 在 ModelScope 开源的模型。Ling-Flash-2.0-eagle3 AQ-MedAI/Ling-Flash-2.0-eagle3 是 ModelScope 魔搭社区上的 424.9M 参数text-generation模型，采用 MIT License 许可，可用 deploy_task text、sglang 0.5.2 部署。 - **Repository**: AQ-MedAI/Ling-Flash-2.0-eagle3 - **License**: MIT License - **Tasks**: text-generation - **Parameters**: 424.9M - **Inference backends**: deploy_task text, sglang 0.5.2 - **Downloads**: 50 - **Stars**: 0 - **Last updated**: 2025-12-19 Source: https://www.modelscope.cn/models/AQ-MedAI/Ling-Flash-2.0-eagle3 --- # Ling-Flash-2.0-eagle3 ## Model Overview **Ling-Flash-2.0-eagle3** is a high-performance draft model specifically designed for inference acceleration, leveraging advanced EAGLE3 speculative sampling technology to achieve a deep balance between inference performance and model stability. The model is trained on **1.4 million high-quality Open-PerfectBlend instruction datasets**, significantly boosting inference throughput while maintaining high accuracy, providing extreme performance for high-load production environments. ## Key Features - **Speculative Sampling Optimization**: Based on EAGLE3 technology, achieving high verification pass rate with speculative length of 4 - **Outstanding Throughput Performance**: FP8 quantization + EAGLE3 solution, throughput improvement up to 94% - **High Accuracy Guarantee**: Maintaining 93%+ accuracy on mainstream benchmarks - **Production-Grade Optimization**: Achieving 3954 tokens/s output throughput on single NVIDIA H200 ## Performance ### Speculative Sampling Efficiency Average Acceptance Length with speculative length of 4: | Benchmark | Average Acceptance Length | |-----------|---------------------------| | HumanEval | 3.100 | | GSM8K | 3.412 | | Math-500 | 3.428 | ### Throughput Improvement Using **FP8 quantization + EAGLE3 optimization**, throughput improvement compared to FP8-only at 32 concurrency: | Benchmark | Throughput Improvement | |-----------|------------------------| | HumanEval | **+71%** | | GSM8K | **+45%** | | Math-500 | **+94%** | ### Ultimate Inference Performance - **Hardware Environment**: NVIDIA H200 single GPU - **Peak Throughput**: Math-500 reaches **3954 tokens/s** at 64 concurrency - **Accuracy**: Maintains 93%-97% high accuracy on mainstream benchmarks ![H200_Accuracy_Refined](https://hackmd.io/_uploads/r1zVyhM7Zg.png) ![H200_Final_Poster_Math-500](https://hackmd.io/_uploads/rkfVJ2zmWg.png) ![H200_Final_Poster_HumanEval](https://hackmd.io/_uploads/H1fN13G7-g.png) ![H200_Final_Poster_GSM8K](https://hackmd.io/_uploads/H1MVyhzmbx.png) *Figure: Throughput performance comparison and accuracy metrics under equal compute on 1xH200* ## Technical Specifications - **Model Architecture**: LlamaForCausalLMEagle3 - **Number of Layers**: 1 layer (Draft Model) - **Hidden Size**: 4096 - **Attention Heads**: 32 (KV heads: 8) - **Intermediate Size**: 14336 - **Vocabulary Size**: 157,184 - **Max Position Embeddings**: 32,768 - **Data Type**: bfloat16 ## Quick Start ### Requirements - NVIDIA GPU - CUDA 12.0+ - PyTorch 2.0+ ### Installation ```bash pip install sglang==0.5.6 ``` and include PR https://github.com/sgl-project/sglang/pull/15119 ### Inference with SGLang ```python python3 -m sglang.launch_server \ --model-path /models/Ling-flash-2.0-FP8 \ --host 0.0.0.0 --port 30012 \ --trust-remote-code \ --attention-backend fa3 \ --mem-fraction-static 0.9 \ --tp-size 1 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path AQ-MedAI/Ling-Flash-2.0-eagle3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 ``` ## Evaluation Results ### Accuracy Comparison | Dataset | FP8 | FP8 + EAGLE3 | |---------|-----|--------------| | HumanEval | 93.29% | 93.29% | | GSM8K | 96.59% | 96.74% | | Math-500 | 95.80% | 96.20% | ### Detailed Throughput Data (tokens/s on 1xH200) **HumanEval:** - Concurrency 1: 196 → 330 (+68%) - Concurrency 4: 513 → 807 (+57%) - Concurrency 8: 725 → 1187 (+64%) - Concurrency 16: 1029 → 1704 (+66%) - Concurrency 32: 1432 → 2451 (+71%) - Concurrency 64: 1931 → 3005 (+56%) **GSM8K:** - Concurrency 1: 186 → 328 (+76%) - Concurrency 4: 469 → 721 (+54%) - Concurrency 8: 673 → 1023 (+52%) - Concurrency 16: 955 → 1412 (+48%) - Concurrency 32: 1364 → 1982 (+45%) - Concurrency 64: 2020 → 2420 (+20%) **Math-500:** - Concurrency 1: 197 → 364 (+85%) - Concurrency 4: 521 → 896 (+72%) - Concurrency 8: 755 → 1354 (+79%) - Concurrency 16: 1103 → 2048 (+86%) - Concurrency 32: 1612 → 3120 (+94%) - Concurrency 64: 2415 → 3954 (+64%) ## Training Data - **Open-PerfectBlend Instruction Set**: 1.4 million high-quality instruction data - **Data Quality**: Rigorously filtered and cleaned to ensure high-quality training data ## Use Cases - High-concurrency inference services - Real-time dialogue systems - Code generation and completion - Mathematical reasoning and computation - Production environments requiring low-latency responses ## Open Source Contribution We actively contribute back to the open-source community. Related optimization achievements have been submitted to the **SGLang community**: - PR #15119: [EAGLE3 Optimization Implementation](https://github.com/sgl-project/sglang/pull/15119) ## Limitations and Notes - This model is a draft model that needs to be used with a target model to achieve speculative sampling - FP8 quantization is recommended for optimal performance - Performance may vary across different hardware platforms - Medical domain applications must comply with relevant regulations; model outputs are for reference only ## Citation If you use this model in your research, please cite: ```bibtex @misc{Ling-flash-2-eagle3, title={Ling-Flash-2.0-eagle3: High-Performance Draft Model for Speculative Decoding}, author={Ant AQ Team}, year={2025}, } ``` ## License The model weights are released under the MIT License. ---