---
title: CHAMP
canonical_url: "https://www.modelscope.cn/datasets/gavinharris/CHAMP"
md_url: "https://www.modelscope.cn/datasets/gavinharris/CHAMP.md"
repository: gavinharris/CHAMP
chinese_name: "CHAMP-概念与提示增强型数学数据集"
last_updated: 2025-07-30
license: "Apache License 2.0"
storage_size: "69 MB"
domain:
  - text
tasks:
  - question_answering
languages:
  - en
downloads: 102
stars: 1
---

# CHAMP

> CHAMP - gavinharris 在 ModelScope 开源的数据集。一个用于细粒度分析大型语言模型数学推理能力的竞赛级数据集

gavinharris/CHAMP 是 ModelScope 魔搭社区上的question_answering数据集，涉及 text 领域，存储大小 69 MB，采用 Apache License 2.0 许可。

- **Repository**: gavinharris/CHAMP
- **License**: Apache License 2.0
- **Tasks**: question_answering
- **Domain**: text
- **Storage size**: 69 MB
- **Downloads**: 102
- **Stars**: 1
- **Last updated**: 2025-07-30

Source: https://www.modelscope.cn/datasets/gavinharris/CHAMP

---

## CHAMP：一个用于对大型语言模型的数学推理能力进行精细分析的竞赛级数据集

##### [毛玉军](https://yujunmao1.github.io/) 1，[金允](https://people.csail.mit.edu/yoonkim/) 2，[周一伦](https://yilunzhou.github.io/) 2

1. 独立研究员，2. 麻省理工学院计算机科学与人工智能实验室

#### [论文](https://arxiv.org/pdf/2401.06961.pdf)  [数据集探索器](https://yujunmao1.github.io/CHAMP/explorer.html)  [快速入门指南](https://yujunmao1.github.io/CHAMP/quickstart.html)  [GitHub 代码库](https://github.com/YilunZhou/champ-dataset)

![图片](https://yujunmao1.github.io/CHAMP/imgs/champ.png)

**简而言之：** 我们提供了一个包含高难度高中竞赛级别数学题目的数据集，每个题目都配有与之相关的、对解决该题目有帮助的概念和提示（A），这使得能够对模型理解并应用此类补充信息的能力进行细致分析（B）。此外，我们还为每个题目发布了模型生成的解决方案的平行语料库，并标注了可能出现的错误（如算术错误或理解问题错误），例如在上述图片的右侧面板中所示（C），这有助于进一步对模型进行分析和评估其验证能力（D）。

**亲自动手试试：** 请先访问 [dataset explorer](https://yujunmao1.github.io/CHAMP/explorer.html) 页面，然后按照 [快速入门指南](https://yujunmao1.github.io/CHAMP/quickstart.html) 中的步骤，将该数据集应用于您自己的基准测试中。

**摘要**：近期的大型语言模型（LLM）已展现出一定的数学推理能力。但目前尚不清楚它们在更具挑战性的竞赛级别问题上表现如何。虽然自动生成的中间推理步骤的口头表述（即“思路提示”）已被证明是有帮助的，但是否 LLM 能够利用诸如问题特定提示之类的有益辅助信息尚未得到研究。在本文中，我们提出了一个具有挑战性的基准数据集，以支持此类分析。名为“概念与提示标注数学问题”（CHAMP）的数据集包含了高中数学竞赛问题，并对问题进行了“概念”（即一般数学事实）和“提示”（即问题特定技巧）的标注。这些标注使我们能够探索额外信息（如相关提示、误导性的概念或相关问题）的影响。该基准数据集具有一定的难度，最佳模型在标准设置下的得分仅为 58.1%。有了概念和提示，有时性能会有所提升，这表明某些模型能够利用此类辅助信息。我们还对模型生成的解决方案进行了正确性标注。利用这个语料库，我们发现模型往往通过错误的推理步骤得出正确的最终答案。此外，我们还测试了模型是否能够验证这些解决方案，结果发现大多数模型都难以做到这一点。相关数据集和代码可在项目网站上获取。

该数据集中的问题及解决方案主要来源于亚瑟·恩格尔所著的《问题解决策略》（https://link.springer.com/book/10.1007/b97682）（1998 年出版）。我们对相关材料进行了人工编辑（以符合我们的评估格式，并减少数据集的污染情况），并仅将其发布用于研究目的。不打算用于商业用途。


## CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities

##### [Yujun Mao](https://yujunmao1.github.io/)1, [Yoon Kim](https://people.csail.mit.edu/yoonkim/)2, [Yilun Zhou](https://yilunzhou.github.io/)2

###### 1 Independent Researcher, 2 MIT CSAIL

###### [[Paper](https://arxiv.org/pdf/2401.06961.pdf)]   [[Dataset Explorer](https://yujunmao1.github.io/CHAMP/explorer.html)]   [[Quickstart Guide](https://yujunmao1.github.io/CHAMP/quickstart.html)]   [[GitHub Repo](https://github.com/YilunZhou/champ-dataset)]

![img](https://yujunmao1.github.io/CHAMP/imgs/champ.png)

**TLDR:** We present a dataset of challenging high school competition-level math problems, distinctly annotated with concepts and hints relevant and helpful for each problem (A), which allows for fine-grained analyses of model's ability to understand and apply such supplemental information (B). Additionally, we release a parallel corpus of model-generated solutions for each problem, annotated with the first occurrence of mistake, if present, e.g., arithmetic error or question misunderstanding, right panel of the image above (C), which allows for further model analyses and evaluations of model verification capability (D).

**Try It Yourself:** Start with the [dataset explorer](https://yujunmao1.github.io/CHAMP/explorer.html), and follow the [quickstart guide](https://yujunmao1.github.io/CHAMP/quickstart.html) to use the dataset in your own benchmarking.

**Abstract:** Recent large language models (LLMs) have shown indications of mathematical reasoning ability. However it has not been clear how they would fare on more challenging competition-level problems. And while self-generated verbalizations of intermediate reasoning steps (i.e., chain-of-thought prompting) have been shown to be helpful, whether LLMs can make use of helpful side information such as problem-specific hints has not been investigated before. In this paper, we propose a challenging benchmark dataset for enabling such analyses. The Concept and Hint-Annotated Math Problems (CHAMP) consists of high school math competition problems, annotated with *concepts*, or general math facts, and *hints*, or problem-specific tricks. These annotations allow us to explore the effects of additional information, such as relevant hints, misleading concepts, or related problems. This benchmark is difficult, with the best model only scoring 58.1% in standard settings. With concepts and hints, performance sometimes improves, indicating that some models can make use of such side information. We further annotate model-generated solutions for their correctness. Using this corpus, we find that models often arrive at the correct final answer through wrong reasoning steps. In addition, we test whether models are able to verify these solutions, and find that most models struggle. The dataset and code are available on the project website.

*The problems and solutions in this dataset are mainly sourced from [Problem-Solving Strategies](https://link.springer.com/book/10.1007/b97682) by Arthur Engel (1998). We manually edit the materials (to fit our evaluation format and also to mitigate dataset contamination) and release them solely for research purposes. No commercial use is intended.*

---
license: Apache License 2.0
---
数据集文件元信息以及数据文件，请浏览“数据集文件”页面获取。

当前数据集卡片使用的是默认模版，数据集的贡献者未提供更加详细的数据集介绍，但是您可以通过如下GIT Clone命令，或者ModelScope SDK来下载数据集

#### 下载方法 
:modelscope-code[]{type="sdk"}
:modelscope-code[]{type="git"}
