---
title: LeetCode-O
canonical_url: "https://www.modelscope.cn/datasets/hkust-nlp/LeetCode-O"
md_url: "https://www.modelscope.cn/datasets/hkust-nlp/LeetCode-O.md"
repository: hkust-nlp/LeetCode-O
last_updated: 2025-05-06
license: "Apache License 2.0"
storage_size: "34 MB"
downloads: 837
stars: 0
---

# LeetCode-O

> LeetCode-O - hkust-nlp 在 ModelScope 开源的数据集。This is the LeetCode-O benchmark proposed in CodeI/O paper (Arxiv 2502.07316).

hkust-nlp/LeetCode-O 是 ModelScope 魔搭社区上的数据集，存储大小 34 MB，采用 Apache License 2.0 许可。

- **Repository**: hkust-nlp/LeetCode-O
- **License**: Apache License 2.0
- **Storage size**: 34 MB
- **Downloads**: 837
- **Stars**: 0
- **Last updated**: 2025-05-06

Source: https://www.modelscope.cn/datasets/hkust-nlp/LeetCode-O

---

This is the LeetCode-O benchmark proposed in CodeI/O paper (Arxiv 2502.07316).

The data file is in `leetcode.jsonl` and we provide an example prediction file (gpt-4.1-nano) in `prediction.jsonl`.

To evaluate your model on this benchmark, please prepare your outputs as in the format of `prediction.jsonl`, which is to add an `output` field to each line in `leetcode.jsonl` corresponding to the output of a LLM to the `messages`.

To calculate the scores, please simply follow `evaluate.py`, and you will see the following when evaluating on the example predictions:

```
{'Difficulty_Easy_Example_Acc': 0.8931972789115646,
 'Difficulty_Easy_Question_Acc': 0.7046979865771812,
 'Difficulty_Hard_Example_Acc': 0.6502695417789758,
 'Difficulty_Hard_Question_Acc': 0.2857142857142857,
 'Difficulty_Medium_Example_Acc': 0.7582191780821917,
 'Difficulty_Medium_Question_Acc': 0.46179401993355484,
 'Lang_EN_Example_Acc': 0.7933846850928863,
 'Lang_EN_Question_Acc': 0.6311111111111111,
 'Lang_ZH_Example_Acc': 0.7403715450838242,
 'Lang_ZH_Question_Acc': 0.56,
 'No_Answer': 0.001359311282283643,
 'Overall_Example_Acc': 0.7668781150883552,
 'Overall_Question_Acc': 0.48333333333333334}
```

The main metric of this benchmark is `Overall_Question_Acc`.
