---
title: VisualPuzzles
canonical_url: "https://www.modelscope.cn/datasets/neulab/VisualPuzzles"
md_url: "https://www.modelscope.cn/datasets/neulab/VisualPuzzles.md"
repository: neulab/VisualPuzzles
last_updated: 2026-03-27
license: mit
storage_size: "136 MB"
downloads: 1079
stars: 0
---

# VisualPuzzles

> VisualPuzzles - neulab 在 ModelScope 开源的数据集。VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

neulab/VisualPuzzles 是 ModelScope 魔搭社区上的数据集，存储大小 136 MB，采用 mit 许可。

- **Repository**: neulab/VisualPuzzles
- **License**: mit
- **Storage size**: 136 MB
- **Downloads**: 1079
- **Stars**: 0
- **Last updated**: 2026-03-27

Source: https://www.modelscope.cn/datasets/neulab/VisualPuzzles

---

# VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

[🏠 Homepage](https://neulab.github.io/VisualPuzzles/) | [📊 VisualPuzzles](https://huggingface.co/datasets/neulab/VisualPuzzles) | [💻 Github](https://github.com/neulab/VisualPuzzles) | [📄 Arxiv](https://arxiv.org/abs/2504.10342) | [📕 PDF](https://arxiv.org/pdf/2504.10342) | [🖥️ Zeno Model Output](https://hub.zenoml.com/project/2e727b03-a677-451a-b714-f2c07ad2b49f/VisualPuzzles)

![Puzzle Teaser](https://neulab.github.io/VisualPuzzles/static/img/human_performance.png)

## Overview
**VisualPuzzles** is a multimodal benchmark specifically designed to evaluate **reasoning abilities** in large models while deliberately minimizing reliance on domain-specific knowledge.

Key features:
- 1168 diverse puzzles
- 5 reasoning categories: Algorithmic, Analogical, Deductive, Inductive, Spatial
- Difficulty labels: Easy, Medium, Hard
- Less knowledge-intensive than existing benchmarks (e.g., MMMU)
- More reasoning-complex than existing benchmarks (e.g., MMMU)

## Key Findings
- All models perform worse than humans; most can't surpass even 5th-percentile human performance.
- Strong performance on knowledge-heavy benchmarks does not transfer well.
- Larger models and structured "thinking modes" don't guarantee better results.
- Scaling model size does not ensure stronger reasoning

## Usage

To load this dataset via Hugging Face’s `datasets` library:

```python
from datasets import load_dataset

dataset = load_dataset("neulab/VisualPuzzles")
data = dataset["train"]

sample = data[0]
print("ID:", sample["id"])
print("Category:", sample["category"])
print("Question:", sample["question"])
print("Options:", sample["options"])
print("Answer:", sample["answer"])
```

## Citation

If you use or reference this dataset in your work, please cite:

```bibtex
@article{song2025visualpuzzles,
  title         = {VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge},
  author        = {Song, Yueqi and Ou, Tianyue and Kong, Yibo and Li, Zecheng and Neubig, Graham and Yue, Xiang},
  year          = {2025},
  journal       = {arXiv preprint arXiv:2504.10342},
  url           = {https://arxiv.org/abs/2504.10342}
}
```
