---
title: "VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering"
canonical_url: "https://www.modelscope.cn/papers/125193"
md_url: "https://www.modelscope.cn/papers/125193.md"
arxiv_id: 2503.06492
published: 2025-03-09
last_updated: 2025-03-09
authors:
  - "Yanling Wang"
  - "Yihan Zhao"
  - "Xiaodong Chen"
  - "Shasha Guo"
  - "Lixin Liu"
  - "Haoyang Li"
  - "Yong Xiao"
  - "Jing Zhang"
  - "Qi Li"
  - "Ke Xu"
model_name: VisualSimpleQA
model_developer: "中关村实验室，中国人民大学，腾讯，清华大学"
domain:
  - "自然语言处理"
  - "计算机视觉"
  - "深度学习"
type:
  - "自然语言处理"
  - "计算机视觉"
  - "深度学习"
  - "Computation and Language (cs.CL)"
  - "Computer Vision and Pattern Recognition (cs.CV)"
arxiv_url: "https://arxiv.org/abs/2503.06492"
pdf_url: "https://arxiv.org/pdf/2503.06492.pdf"
---

# VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering

> Large vision-language models (LVLMs) have demonstrated remarkable achievements, yet the generation of non-factual responses remains prevalent in fact-seeking question answering (QA). Current multimodal fact-seeking benchmarks primarily focus on comparing…

「VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering」是 ModelScope 魔搭社区收录的论文，arXiv 2503.06492，作者为 Yanling Wang, Yihan Zhao, Xiaodong Chen et al.，发表于 2025-03-09，属于 自然语言处理、计算机视觉、深度学习 领域。

- **ArXiv**: 2503.06492
- **Published**: 2025-03-09
- **Authors**: Yanling Wang, Yihan Zhao, Xiaodong Chen, Shasha Guo, Lixin Liu, Haoyang Li, Yong Xiao, Jing Zhang, Qi Li, Ke Xu
- **Model**: VisualSimpleQA
- **Developer**: 中关村实验室，中国人民大学，腾讯，清华大学
- **Domain**: 自然语言处理, 计算机视觉, 深度学习
- **ArXiv URL**: https://arxiv.org/abs/2503.06492
- **PDF**: https://arxiv.org/pdf/2503.06492.pdf

Source: https://www.modelscope.cn/papers/125193

---

> VisualSimpleQA：拆分评估大型视觉-语言模型在事实检索问答中的表现

## 摘要

本文提出了一种新的多模态事实检索问答基准VisualSimpleQA，旨在解决当前多模态基准测试中无法有效评估大型视觉-语言模型（LVLM）特定模块性能的问题。VisualSimpleQA通过解耦合的方式分别评估视觉和语言模块的性能，并引入了明确的难度标准来指导人工标注，从而生成更具挑战性的子集VisualSimpleQA-hard。实验结果表明，即使是像GPT-4这样的最先进模型，在多模态事实检索问答任务上的正确率也仅达到60%以上，而在更具挑战性的VisualSimpleQA-hard上则降至30%左右。这表明LVLM在处理复杂视觉识别任务和长尾知识方面仍有很大的改进空间。此外，该基准还具有高质量、多样性和减少评估偏差的特点，确保了数据的真实性和可靠性。

## Abstract

Large vision-language models (LVLMs) have demonstrated remarkable achievements, yet the generation of non-factual responses remains prevalent in fact-seeking question answering (QA). Current multimodal fact-seeking benchmarks primarily focus on comparing model outputs to ground truth answers, providing limited insights into the performance of modality-specific modules. To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking benchmark with two key features. First, it enables streamlined and decoupled evaluation of LVLMs in visual and linguistic modalities. Second, it incorporates well-defined difficulty criteria to guide human annotation and facilitates the extraction of a challenging subset, VisualSimpleQA-hard. Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across these models highlights substantial opportunities for improvement in both visual and linguistic modules. The dataset is available at this https URL.