---
title: "MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning"
canonical_url: "https://www.modelscope.cn/papers/124656"
md_url: "https://www.modelscope.cn/papers/124656.md"
arxiv_id: 2503.07459
published: 2025-03-10
last_updated: 2025-03-10
authors:
  - "Xiangru Tang"
  - "Daniel Shao"
  - "Jiwoong Sohn"
  - "Jiapeng Chen"
  - "Jiayi Zhang"
  - "Jinyu Xiang"
  - "Fang Wu"
  - "Yilun Zhao"
  - "Chenglin Wu"
  - "Wenqi Shi"
  - "Arman Cohan"
  - "Mark Gerstein"
model_name: MEDAGENTS-BENCH
model_developer: "耶鲁大学, 斯坦福大学, 德克萨斯西南医学中心"
domain:
  - "自然语言处理"
  - "医学"
  - "机器学习"
type:
  - "自然语言处理"
  - "医学"
  - "机器学习"
  - "Computation and Language (cs.CL)"
  - "Artificial Intelligence (cs.AI)"
arxiv_url: "https://arxiv.org/abs/2503.07459"
pdf_url: "https://arxiv.org/pdf/2503.07459.pdf"
code_link: "https://github.com/gersteinlab/medagents-benchmark"
---

# MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning

> Large Language Models (LLMs) have shown impressive performance on existing medical question-answering benchmarks. This high performance makes it increasingly difficult to meaningfully evaluate and differentiate advanced methods. We present MedAgentsBench, a…

「MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning」是 ModelScope 魔搭社区收录的论文，arXiv 2503.07459，作者为 Xiangru Tang, Daniel Shao, Jiwoong Sohn et al.，发表于 2025-03-10，属于 自然语言处理、医学、机器学习 领域。

- **ArXiv**: 2503.07459
- **Published**: 2025-03-10
- **Authors**: Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, Arman Cohan, Mark Gerstein
- **Model**: MEDAGENTS-BENCH
- **Developer**: 耶鲁大学, 斯坦福大学, 德克萨斯西南医学中心
- **Domain**: 自然语言处理, 医学, 机器学习
- **ArXiv URL**: https://arxiv.org/abs/2503.07459
- **PDF**: https://arxiv.org/pdf/2503.07459.pdf
- **Code**: https://github.com/gersteinlab/medagents-benchmark

Source: https://www.modelscope.cn/papers/124656

---

> 超越常规：MEDAGENTS-BENCH重新定义复杂医学推理的基准测试

## 摘要

本文提出了一种名为MEDAGENTS-BENCH的基准测试，专注于评估复杂医学推理任务中的多步临床推理、诊断制定和治疗计划。现有医学问答基准测试在简单问题上表现良好，但在复杂场景中仍存在不足。MEDAGENTS-BENCH通过从七个已建立的医学数据集中筛选出真正具有挑战性的问题，解决了现有评价体系的三个关键问题：简单问题过多、采样和评估协议不一致以及缺乏对性能、成本和推理时间之间关系的系统分析。

研究背景方面，大型语言模型（LLMs）在现有医学问答基准测试中表现出色，但难以区分先进方法的真实能力。作者指出，现有的医学推理数据集虽然广泛，但包含大量标准化考试中的简单问题，导致基础模型也能取得高分，无法有效评估推理方法的进步。

提出的方法是通过引入一个严格的数据筛选流程，包括对抗性过滤和污染分析，并结合医学专业人员的人工标注来确保问题的复杂性和合理性。实验结果表明，最新的思考模型如DEEPSEEK R1和OPENAI O3在复杂医学推理任务中表现出色，分别比传统方法提高了15-25%的准确率。此外，基于搜索的代理方法如AFlow提供了最佳的性能-成本比率，能够在较低的计算资源下接近思考模型的表现。

该研究的价值贡献在于提供了一个更严格的基准测试框架，能够更准确地评估和区分不同模型在复杂医学推理任务中的表现，为未来的研究和实际应用提供了重要的参考。

## Abstract

Large Language Models (LLMs) have shown impressive performance on existing medical question-answering benchmarks. This high performance makes it increasingly difficult to meaningfully evaluate and differentiate advanced methods. We present MedAgentsBench, a benchmark that focuses on challenging medical questions requiring multi-step clinical reasoning, diagnosis formulation, and treatment planning-scenarios where current models still struggle despite their strong performance on standard tests. Drawing from seven established medical datasets, our benchmark addresses three key limitations in existing evaluations: (1) the prevalence of straightforward questions where even base models achieve high performance, (2) inconsistent sampling and evaluation protocols across studies, and (3) lack of systematic analysis of the interplay between performance, cost, and inference time. Through experiments with various base models and reasoning methods, we demonstrate that the latest thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in complex medical reasoning tasks. Additionally, advanced search-based agent methods offer promising performance-to-cost ratios compared to traditional approaches. Our analysis reveals substantial performance gaps between model families on complex questions and identifies optimal model selections for different computational constraints. Our benchmark and evaluation framework are publicly available at this https URL.