---
title: "LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding"
canonical_url: "https://www.modelscope.cn/papers/123554"
md_url: "https://www.modelscope.cn/papers/123554.md"
arxiv_id: 2503.04359
published: 2025-03-06
last_updated: 2025-03-06
authors:
  - "Jia Li"
  - "Xuyuan Guo"
  - "Lei Li"
  - "Kechi Zhang"
  - "Ge Li"
  - "Jia Li"
  - "Zhengwei Tao"
  - "Fang Liu"
  - "Chongyang Tao"
  - "Yuqi Zhu"
  - "Zhi Jin"
model_name: LONGCODEU
model_developer: "北京大学高可信软件技术教育部重点实验室，北京大学计算机科学系"
domain:
  - "自然语言处理"
  - "软件工程"
  - "机器学习"
type:
  - "自然语言处理"
  - "软件工程"
  - "机器学习"
  - "Software Engineering (cs.SE)"
arxiv_url: "https://arxiv.org/abs/2503.04359"
pdf_url: "https://arxiv.org/pdf/2503.04359.pdf"
---

# LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding

> Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for…

「LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding」是 ModelScope 魔搭社区收录的论文，arXiv 2503.04359，作者为 Jia Li, Xuyuan Guo, Lei Li et al.，发表于 2025-03-06，属于 自然语言处理、软件工程、机器学习 领域。

- **ArXiv**: 2503.04359
- **Published**: 2025-03-06
- **Authors**: Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, Zhi Jin
- **Model**: LONGCODEU
- **Developer**: 北京大学高可信软件技术教育部重点实验室，北京大学计算机科学系
- **Domain**: 自然语言处理, 软件工程, 机器学习
- **ArXiv URL**: https://arxiv.org/abs/2503.04359
- **PDF**: https://arxiv.org/pdf/2503.04359.pdf

Source: https://www.modelscope.cn/papers/123554

---

> 填补空白：LONGCODEU——评估长上下文语言模型长代码理解能力的新标杆

## 摘要

本文提出了一个名为LONGCODEU的基准测试框架，旨在评估长上下文语言模型（LCLMs）在理解长代码方面的能力。当前的LCLMs在软件工程应用中具有巨大潜力，但由于缺乏严谨的评估框架，进展受到限制。LONGCODEU从四个关键方面设计了8项任务来全面评估LCLMs的长代码理解能力，包括代码单元感知、代码单元内部理解、代码单元间关系理解和长代码文档理解。通过这些任务，LONGCODEU不仅涵盖了实际应用场景中的需求，还使用了来自真实世界代码仓库的数据，确保了评估的真实性和可靠性。实验结果显示，现有的LCLMs在处理超过32K长度的长代码时性能显著下降，尤其在跨代码单元关系理解方面表现尤为不足。这为优化LCLMs和推动软件工程领域的进步提供了宝贵见解。

## Abstract

Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for long code understanding. To gap this obstacle, we propose a long code understanding benchmark LONGCODEU from four aspects (8 tasks) to evaluate LCLMs' long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6 general models and 3 code models). Our experimental results reveal key limitations in current LCLMs' capabilities for long code understanding. Particularly, the performance of LCLMs drops dramatically when the long code length is greater than 32K, falling far short of their claimed 128K-1M context windows. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our study provides valuable insights for optimizing LCLMs and driving advancements in software engineering.