---
title: ShadenA-MathNet-Retrieve
canonical_url: "https://www.modelscope.cn/datasets/hf/ShadenA-MathNet-Retrieve"
md_url: "https://www.modelscope.cn/datasets/hf/ShadenA-MathNet-Retrieve.md"
repository: hf/ShadenA-MathNet-Retrieve
last_updated: 2026-10-10
license: cc-by-4.0
storage_size: "140 MB"
downloads: 151
stars: 0
---

# ShadenA-MathNet-Retrieve

> ShadenA-MathNet-Retrieve - hf 在 ModelScope 开源的数据集。The Math-Aware Retrieval benchmark from MathNet (paper): given an Olympiad problem, retrieve a mathematically equivalent reformulation of it from a large corpus of hard negatives. Each equivalent is provided…

hf/ShadenA-MathNet-Retrieve 是 ModelScope 魔搭社区上的数据集，存储大小 140 MB，采用 cc-by-4.0 许可。

- **Repository**: hf/ShadenA-MathNet-Retrieve
- **License**: cc-by-4.0
- **Storage size**: 140 MB
- **Downloads**: 151
- **Stars**: 0
- **Last updated**: 2026-10-10

Source: https://www.modelscope.cn/datasets/hf/ShadenA-MathNet-Retrieve

---

# MathNet-Retrieve

The **Math-Aware Retrieval** benchmark from [MathNet](https://mathnet.mit.edu)
([paper](https://arxiv.org/abs/2604.18584)): given an Olympiad problem, retrieve a
*mathematically equivalent* reformulation of it from a large corpus of hard
negatives. Each equivalent is provided at three difficulty tiers — `easy`,
`medium`, `hard` — that increasingly disguise the surface form while preserving the
underlying problem, so retrieval gets sharply harder even though the answer is the same.

- **15,000** queries, one gold-equivalent target per tier.
- **3 tiers** (`easy`, `medium`, `hard`), over a shared **117,088-doc** corpus (equivalents at all
  tiers + LLM near-miss hard negatives + distractors).
- Standard **BEIR / MTEB** layout.

## Quick start

```python
from datasets import load_dataset

tier = "hard"  # one of "easy", "medium", "hard"
corpus  = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-corpus", split="test")   # {_id, text}
queries = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-queries", split="test")  # {_id, text}
qrels   = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-qrels", split="test")    # {query-id, corpus-id, score}
```

All three configs share the same corpus and queries; they differ only in which
reformulation is the gold positive. Each query has exactly **one** positive, and the
query's own equivalents at the other tiers act as negatives.

## Tiers

| Tier | Queries | Corpus | What changes |
|---|---|---|---|
| easy | 15,000 | 117,088 | light paraphrase (renamed vars, reordered clauses) |
| medium | 15,000 | 117,088 | moderate restructuring (notation, framing) |
| hard | 15,000 | 117,088 | heavy disguise; minimal lexical overlap |

## Results

Recall@k (%) on the shared corpus; **bold** = best.

| Model | Easy R@1 | Easy R@5 | Medium R@1 | Medium R@5 | Hard R@1 | Hard R@5 |
|---|---|---|---|---|---|---|
| `gemini-embedding-001` | 11.4 | **90.7** | 2.5 | **77.1** | 0.0 | **9.3** |
| `qwen3-embedding-4B` | **14.8** | 83.5 | **2.9** | 71.6 | 0.0 | 8.4 |
| `text-embedding-3-large` | 6.8 | 78.1 | 1.6 | 68.3 | 0.0 | 2.6 |
| `all-mpnet-base-v2` | 6.8 | 82.5 | 1.3 | 51.0 | 0.0 | 1.2 |
| `multi-qa-mpnet-base-dot-v1` | 5.1 | 80.6 | 1.1 | 47.3 | 0.0 | 0.8 |
| `text-embedding-ada-002` | 3.3 | 66.1 | 1.1 | 55.4 | 0.0 | 0.7 |
| `text-embedding-3-small` | 3.6 | 59.8 | 1.4 | 51.5 | 0.0 | 0.5 |

Every model degrades sharply from easy to hard: even the strongest solve `easy` but
collapse on `hard` (Recall@1 ≈ 0; hard Recall@10 tops out near 54%). Retrieving
equivalent problems remains unreliable once the surface form is disguised.

## Caveats

- **Not contamination-clean** — source problems may appear in pretraining data; prefer relative comparisons.
- Equivalents and hard negatives are **LLM-generated and machine-verified**; a small fraction may be imperfect.

## License & citation

Released under **CC BY 4.0**, consistent with [`ShadenA/MathNet`](https://huggingface.co/datasets/ShadenA/MathNet).

```bibtex
@inproceedings{alshammari2026mathnet,
  title     = {MathNet: A Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
  author    = {Alshammari, Shaden and Wen, Kevin and Zainal, Abrar and Hamilton, Mark and
               Safaei, Navid and Albarakati, Sultan and Freeman, William T. and Torralba, Antonio},
  booktitle = {International Conference on Learning Representations},
  year      = {2026},
  url       = {https://mathnet.mit.edu}
}
```

[Project site](https://mathnet.mit.edu) · [Paper](https://arxiv.org/abs/2604.18584) · [Main dataset](https://huggingface.co/datasets/ShadenA/MathNet)