---
title: EAGLE3-LLaMA3.1-Instruct-8B
canonical_url: "https://www.modelscope.cn/models/gpustack/EAGLE3-LLaMA3.1-Instruct-8B"
md_url: "https://www.modelscope.cn/models/gpustack/EAGLE3-LLaMA3.1-Instruct-8B.md"
repository: gpustack/EAGLE3-LLaMA3.1-Instruct-8B
last_updated: 2025-11-26
license: apache-2.0
model_type:
- llama
architectures:
- LlamaForCausalLM
inference_backends:
- "deploy_task text/emb"
- "lmdeploy 0.9.1"
- "lmdeploy_turbomind 0.9.1"
- "sglang 0.5.2"
- "vllm 0.9.2"
downloads: 75
stars: 0
---
# EAGLE3-LLaMA3.1-Instruct-8B
> EAGLE3-LLaMA3.1-Instruct-8B - gpustack 在 ModelScope 开源的模型。| EAGLE | EAGLE-2 | EAGLE-3 | Blog |
gpustack/EAGLE3-LLaMA3.1-Instruct-8B 是 ModelScope 魔搭社区上的机器学习模型，采用 apache-2.0 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。
- **Repository**: gpustack/EAGLE3-LLaMA3.1-Instruct-8B
- **License**: apache-2.0
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Downloads**: 75
- **Stars**: 0
- **Last updated**: 2025-11-26
Source: https://www.modelscope.cn/models/gpustack/EAGLE3-LLaMA3.1-Instruct-8B
---
EAGLE
| EAGLE |
EAGLE-2 |
EAGLE-3 |
Blog |
##
EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a new baseline for fast decoding of Large Language Models (LLMs) with provable performance maintenance. This approach involves extrapolating the second-top-layer contextual feature vectors of LLMs, enabling a significant boost in generation efficiency.
- EAGLE is:
- certified by the third-party evaluation as the **fastest** speculative method so far.
- achieving **2x** speedup on gpt-fast.
- **3x** faster than vanilla decoding (13B).
- **2x** faster than Lookahead (13B).
- **1.6x** faster than Medusa (13B).
- provably maintaining the consistency with vanilla decoding in the distribution of generated texts.
- trainable (within 1-2 days) and testable on 8x RTX 3090 GPUs. So even the GPU poor can afford it.
- combinable with other parallelled techniques such as vLLM, DeepSpeed, Mamba, FlashAttention, quantization, and hardware optimization.
EAGLE-2 uses the confidence scores from the draft model to approximate acceptance rates, dynamically adjusting the draft tree structure, which further enhances performance.
- EAGLE-2 is:
- **4x** faster than vanilla decoding (13B).
- **1.4x** faster than EAGLE-1 (13B).
EAGLE-3 removes the feature prediction constraint in EAGLE and simulates this process during training using training-time testing. Considering that top-layer features are limited to next-token prediction, EAGLE-3 replaces them with a fusion of low-, mid-, and high-level semantic features.
EAGLE-3 further improves generation speed while ensuring lossless performance.
- EAGLE-3 is:
- **5.6** faster than vanilla decoding (13B).
- **1.8x** faster than EAGLE-1 (13B).
_Inference is conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model._
[//]: # ()
[//]: # ()
[//]: # (Using EAGLE-2, the inference speed on 2 RTX 3060 GPUs can be faster than vanilla autoregressive decoding on an A100 GPU.)
## Support
EAGLE has been merged in the following mainstream LLM serving frameworks (listed in alphabetical order).
- AMD ROCm
- AngelSlim
- AWS NeuronX Distributed Core
- CPM.cu
- Intel® Extension for Transformers
- Intel® LLM Library for PyTorch
- MLC-LLM
- NVIDIA NeMo Framework
- NVIDIA TensorRT-LLM
- NVIDIA TensorRT Model Optimizer
- PaddleNLP
- SGLang
- SpecForge
- vLLM
## Reference
For technical details and full experimental results, please check [the paper of EAGLE](https://arxiv.org/pdf/2401.15077.pdf), [the paper of EAGLE-2](https://arxiv.org/pdf/2406.16858), and [the paper of EAGLE-3](https://arxiv.org/pdf/2503.01840).
```
@inproceedings{li2024eagle,
author = {Yuhui Li and Fangyun Wei and Chao Zhang and Hongyang Zhang},
title = {{EAGLE}: Speculative Sampling Requires Rethinking Feature Uncertainty},
booktitle = {International Conference on Machine Learning},
year = {2024}
}
@inproceedings{li2024eagle2,
author = {Yuhui Li and Fangyun Wei and Chao Zhang and Hongyang Zhang},
title = {{EAGLE-2}: Faster Inference of Language Models with Dynamic Draft Trees},
booktitle = {Empirical Methods in Natural Language Processing},
year = {2024}
}
@inproceedings{li2025eagle3,
author = {Yuhui Li and Fangyun Wei and Chao Zhang and Hongyang Zhang},
title = {{EAGLE-3}: Scaling up Inference Acceleration of Large Language Models via Training-Time Test},
booktitle = {Annual Conference on Neural Information Processing Systems},
year = {2025}
}
```