---
title: DocMath-Eval
canonical_url: "https://www.modelscope.cn/datasets/yale-nlp/DocMath-Eval"
md_url: "https://www.modelscope.cn/datasets/yale-nlp/DocMath-Eval.md"
repository: yale-nlp/DocMath-Eval
last_updated: 2026-07-15
license: mit
storage_size: "124 MB"
downloads: 25166
stars: 3
---

# DocMath-Eval

> DocMath-Eval - yale-nlp 在 ModelScope 开源的数据集。DocMath-Eval 🌐 Homepage | 🤗 Dataset | 📖 arXiv | GitHub

yale-nlp/DocMath-Eval 是 ModelScope 魔搭社区上的数据集，存储大小 124 MB，采用 mit 许可。

- **Repository**: yale-nlp/DocMath-Eval
- **License**: mit
- **Storage size**: 124 MB
- **Downloads**: 25166
- **Stars**: 3
- **Last updated**: 2026-07-15

Source: https://www.modelscope.cn/datasets/yale-nlp/DocMath-Eval

---

## DocMath-Eval
[**🌐 Homepage**](https://docmath-eval.github.io/) | [**🤗 Dataset**](https://huggingface.co/datasets/yale-nlp/DocMath-Eval) | [**📖 arXiv**](https://arxiv.org/abs/2311.09805) | [**GitHub**](https://github.com/yale-nlp/DocMath-Eval)

The data for the paper [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://arxiv.org/abs/2311.09805). 
**DocMath-Eval** is a comprehensive benchmark focused on numerical reasoning within specialized domains. It requires the model to comprehend long and specialized documents and perform numerical reasoning to answer the given question. 

<p align="center">
<img src="figures/overview.png" width="100%">
</p>

## DocMath-Eval Dataset
All the data examples were divided into four subsets:

- **simpshort**, which is reannotated from [TAT-QA](https://aclanthology.org/2021.acl-long.254/) and [FinQA](https://aclanthology.org/2021.emnlp-main.300/), necessitates simple numerical reasoning over short document with one table
- **simplong**, which is reannotated from [MultiHiertt](https://aclanthology.org/2022.acl-long.454/), necessitates simple numerical reasoning over long document with multiple tables;
- **compshort**, which is reannotated from [TAT-HQA](https://aclanthology.org/2022.acl-long.5/), necessitates complex numerical reasoning over short document with one table;
- **complong**, which is annotated from scratch by our team, necessitates complex numerical reasoning over long document with multiple tables.

For each subset, we provide the *testmini* and *test* splits. **Update (July 2026):** the `python_solution` and `ground_truth` annotations for the *test* splits are now publicly released as well.


You can download this dataset by the following command:

```python
from datasets import load_dataset

dataset = load_dataset("yale-nlp/DocMath-Eval")

# print the first example on the complong testmini set
print(dataset["complong_testmini"][0])
```

The dataset is provided in json format and contains the following attributes:

```
{
    "question_id": [string] The question id
    "source": [string] The original source of the example (for simpshort, simplong, and compshort sets)
    "original_question_id": [string] The original question id (for simpshort, simplong, and compshort sets)
    "question": [string] The question text
    "paragraphs": [list] List of paragraphs and tables within the document
    "table_evidence": [list] List of indices in 'paragraphs' that are used as table evidence for the question
    "paragraph_evidence": [list] List of indices in 'paragraphs' that are used as text evidence for the question
    "python_solution": [string] Python-format and executable solution
    "ground_truth": [float] Executed result of 'python_solution'
}
```

## Contact
For any issues or questions, kindly email us at: Yilun Zhao (yilun.zhao@yale.edu).

## Citation

If you use the **DocMath-Eval** benchmark in your work, please kindly cite the paper:

```
@misc{zhao2024docmatheval,
      title={DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents}, 
      author={Yilun Zhao and Yitao Long and Hongjun Liu and Ryo Kamoi and Linyong Nan and Lyuhao Chen and Yixin Liu and Xiangru Tang and Rui Zhang and Arman Cohan},
      year={2024},
      eprint={2311.09805},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.09805}, 
}
```