---
title: "SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models"
canonical_url: "https://www.modelscope.cn/papers/125314"
md_url: "https://www.modelscope.cn/papers/125314.md"
arxiv_id: 2503.07605
published: 2025-03-10
last_updated: 2025-03-10
authors:
  - "Xun Liang"
  - "Hanyu Wang"
  - "Huayi Lai"
  - "Simin Niu"
  - "Shichao Song"
  - "Jiawei Yang"
  - "Jihao Zhao"
  - "Feiyu Xiong"
  - "Bo Tang"
  - "Zhiyu Li"
model_name: SEAP
model_developer: "中国人民大学信息学院、上海高级算法研究院"
domain:
  - "自然语言处理"
  - "大语言模型"
  - "模型压缩"
  - "结构化剪枝"
  - "高效推理"
type:
  - "自然语言处理"
  - "大语言模型"
  - "模型压缩"
  - "结构化剪枝"
  - "高效推理"
  - "Computation and Language"
arxiv_url: "https://arxiv.org/abs/2503.07605"
pdf_url: "https://arxiv.org/pdf/2503.07605"
code_link: "https://github.com/IAAR-Shanghai/SEAP"
---

# SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models

> Large Language Models have achieved remarkable success across various natural language processing tasks, yet their high computational cost during inference remains a major bottleneck. This paper introduces Sparse Expert Activation Pruning (SEAP), a…

「SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models」是 ModelScope 魔搭社区收录的论文，arXiv 2503.07605，作者为 Xun Liang, Hanyu Wang, Huayi Lai et al.，发表于 2025-03-10，属于 自然语言处理、大语言模型、模型压缩 领域。

- **ArXiv**: 2503.07605
- **Published**: 2025-03-10
- **Authors**: Xun Liang, Hanyu Wang, Huayi Lai, Simin Niu, Shichao Song, Jiawei Yang, Jihao Zhao, Feiyu Xiong, Bo Tang, Zhiyu Li
- **Model**: SEAP
- **Developer**: 中国人民大学信息学院、上海高级算法研究院
- **Domain**: 自然语言处理, 大语言模型, 模型压缩, 结构化剪枝, 高效推理
- **ArXiv URL**: https://arxiv.org/abs/2503.07605
- **PDF**: https://arxiv.org/pdf/2503.07605
- **Code**: https://github.com/IAAR-Shanghai/SEAP

Source: https://www.modelscope.cn/papers/125314

---

> 解锁大模型脑力：无训练稀疏专家激活剪枝(SEAP)显著提升推理效率

## 摘要

本文探讨了大规模语言模型（LLMs）在各种自然语言处理任务中取得显著成功的同时，其推理过程中的高计算成本仍然是一个主要瓶颈。为了解决这一问题，论文提出了SEAP（Sparse Expert Activation Pruning），一种无需训练的稀疏专家激活剪枝方法。SEAP通过分析隐藏状态和激活模式，识别任务特定的专家激活模式，并动态地选择保留与任务相关的参数，从而减少推理开销并保持任务性能。

具体来说，SEAP通过构建任务特定的知识语料库，提取多层隐藏状态激活，计算神经元重要性得分，并引入基于逻辑函数的稀疏度调整机制，实现动态剪枝。实验结果表明，SEAP在50%剪枝时比现有基线方法WandA和FLAP高出20%以上，在20%剪枝时仅导致2.2%的性能下降。这些发现突显了SEAP的可扩展性和有效性，使其成为优化大规模LLMs的有前途的方法。

## Abstract

Large Language Models have achieved remarkable success across various natural language processing tasks, yet their high computational cost during inference remains a major bottleneck. This paper introduces Sparse Expert Activation Pruning (SEAP), a training-free pruning method that selectively retains task-relevant parameters to reduce inference overhead. Inspired by the clustering patterns of hidden states and activations in LLMs, SEAP identifies task-specific expert activation patterns and prunes the model while preserving task performance and enhancing computational efficiency. Experimental results demonstrate that SEAP significantly reduces computational overhead while maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2% performance drop compared to the dense model. These findings highlight SEAP's scalability and effectiveness, making it a promising approach for optimizing large-scale LLMs.