---
title: "R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization"
canonical_url: "https://www.modelscope.cn/papers/127779"
md_url: "https://www.modelscope.cn/papers/127779.md"
arxiv_id: 2503.12937
published: 2025-08-04
last_updated: 2025-08-04
authors:
  - "Jingyi Zhang"
  - "Jiaxing Huang"
  - "Huanjin Yao"
  - "Shunyu Liu"
  - "Xikun Zhang"
  - "Shijian Lu"
  - "Dacheng Tao"
model_name: R1-VL
model_developer: "Nanyang Technological University、A*STAR、Alibaba Group"
domain:
  - "人工智能"
  - "计算机视觉"
  - "自然语言处理"
  - "多模态大语言模型"
  - "强化学习"
type:
  - "人工智能"
  - "计算机视觉"
  - "自然语言处理"
  - "多模态大语言模型"
  - "强化学习"
  - "Artificial Intelligence"
  - "Computation and Language"
  - "Computer Vision and Pattern Recognition"
  - "Machine Learning"
arxiv_url: "https://arxiv.org/abs/2503.12937"
pdf_url: "https://arxiv.org/pdf/2503.12937"
code_link: "https://github.com/jingyi0000/R1-VL"
---

# R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization

> Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning…

「R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization」是 ModelScope 魔搭社区收录的论文，arXiv 2503.12937，作者为 Jingyi Zhang, Jiaxing Huang, Huanjin Yao et al.，发表于 2025-08-04，属于 人工智能、计算机视觉、自然语言处理 领域。

- **ArXiv**: 2503.12937
- **Published**: 2025-08-04
- **Authors**: Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, Dacheng Tao
- **Model**: R1-VL
- **Developer**: Nanyang Technological University、A*STAR、Alibaba Group
- **Domain**: 人工智能, 计算机视觉, 自然语言处理, 多模态大语言模型, 强化学习
- **ArXiv URL**: https://arxiv.org/abs/2503.12937
- **PDF**: https://arxiv.org/pdf/2503.12937
- **Code**: https://github.com/jingyi0000/R1-VL

Source: https://www.modelscope.cn/papers/127779

---

> R1-VL：通过逐步组相对策略优化学习多模态大语言模型的推理能力

## 摘要

本文提出了StepGRPO（逐步组相对策略优化），一种面向多模态大语言模型（MLLM）的在线强化学习框架。该框架通过引入基于规则的逐步推理准确性奖励（StepRAR）和逐步推理有效性奖励（StepRVR），在无需额外过程奖励模型的情况下，为模型提供密集的逐步奖励信号，从而解决传统结果级奖励带来的稀疏奖励问题。基于该方法，作者开发了R1-VL系列多模态推理模型，在多个基准测试中展现出优于现有开源及闭源模型的推理性能。

## Abstract

Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning paths are. In this work, we aim to enhance the MLLMs' reasoning ability beyond passively imitating positive reasoning paths. To this end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new online reinforcement learning framework that enables MLLMs to self-improve reasoning ability via simple, effective and dense step-wise rewarding. Specifically, StepGRPO introduces two novel rule-based reasoning rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary intermediate reasoning steps via a soft key-step matching technique, while StepRAR rewards reasoning paths that follow a well-structured and logically consistent reasoning process through a reasoning completeness and logic evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive experiments over 8 benchmarks demonstrate the superiority of our methods.