---
title: "FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation"
canonical_url: "https://www.modelscope.cn/papers/125249"
md_url: "https://www.modelscope.cn/papers/125249.md"
arxiv_id: 2503.06680
published: 2025-03-09
last_updated: 2025-03-09
authors:
  - "Wei Li"
  - "Xin Zhang"
  - "Zhongxin Guo"
  - "Shaoguang Mao"
  - "Wen Luo"
  - "Guangyue Peng"
  - "Yangyu Huang"
  - "Houfeng Wang"
  - "Scarlett Li"
model_name: FEA-Bench
model_developer: "北京大学多媒体信息处理国家重点实验室, 北京大学计算机学院, 微软亚洲研究院"
domain:
  - "自然语言处理"
  - "计算机科学"
  - "软件工程"
  - "深度学习"
type:
  - "自然语言处理"
  - "计算机科学"
  - "软件工程"
  - "深度学习"
  - "Software Engineering (cs.SE)"
arxiv_url: "https://arxiv.org/abs/2503.06680"
pdf_url: "https://arxiv.org/pdf/2503.06680.pdf"
---

# FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation

> Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed…

「FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation」是 ModelScope 魔搭社区收录的论文，arXiv 2503.06680，作者为 Wei Li, Xin Zhang, Zhongxin Guo et al.，发表于 2025-03-09，属于 自然语言处理、计算机科学、软件工程 领域。

- **ArXiv**: 2503.06680
- **Published**: 2025-03-09
- **Authors**: Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, Scarlett Li
- **Model**: FEA-Bench
- **Developer**: 北京大学多媒体信息处理国家重点实验室, 北京大学计算机学院, 微软亚洲研究院
- **Domain**: 自然语言处理, 计算机科学, 软件工程, 深度学习
- **ArXiv URL**: https://arxiv.org/abs/2503.06680
- **PDF**: https://arxiv.org/pdf/2503.06680.pdf

Source: https://www.modelscope.cn/papers/125249

---

> 填补空白：FEA-Bench 为代码库级新功能开发提供全新评估标准

## 摘要

本文提出了一种新的基准测试框架 FEA-Bench，用于评估大型语言模型（LLMs）在代码库级别的新功能实现能力。当前的基准测试主要集中在评估独立编程问题或代码补全任务，而忽略了现实世界中代码库开发的复杂性。FEA-Bench 收集了来自83个 GitHub 仓库的 pull requests，并通过规则和意图过滤构建了专注于新功能开发的任务实例。每个任务实例包含代码更改及其对应的单元测试文件，确保解决方案可以被验证。实验结果显示，现有的 LLMs 在 FEA-Bench 上表现较差，突显了此类代码库级别增量开发的巨大挑战。FEA-Bench 弥补了现有基准测试的不足，为自动化软件工程提供了更全面的评估方法。

## Abstract

Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified. The feature implementation requires LLMs to simultaneously possess code completion capabilities for new components and code editing abilities for other relevant parts in the code repository, providing a more comprehensive evaluation method of LLMs' automated software engineering capabilities. Experimental results show that LLMs perform significantly worse in the FEA-Bench, highlighting considerable challenges in such repository-level incremental code development.