---
title: "SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories"
canonical_url: "https://www.modelscope.cn/papers/125357"
md_url: "https://www.modelscope.cn/papers/125357.md"
arxiv_id: 2503.08625
published: 2025-03-11
last_updated: 2025-03-11
authors:
  - "Muzhi Zhu"
  - "Yuzhuo Tian"
  - "Hao Chen"
  - "Chunluan Zhou"
  - "Qingpei Guo"
  - "Yang Liu"
  - "Ming Yang"
  - "Chunhua Shen"
model_name: SegAgent
model_developer: "浙江大学、蚂蚁集团"
domain:
  - "计算机视觉"
  - "多模态大语言模型"
  - "图像分割"
  - "强化学习"
  - "智能体"
type:
  - "计算机视觉"
  - "多模态大语言模型"
  - "图像分割"
  - "强化学习"
  - "智能体"
  - "Computer Vision and Pattern Recognition"
arxiv_url: "https://arxiv.org/abs/2503.08625"
pdf_url: "https://arxiv.org/pdf/2503.08625"
code_link: "https://github.com/aim-uofa/SegAgent"
---

# SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories

> While MLLMs have demonstrated adequate image understanding capabilities, they still struggle with pixel-level comprehension, limiting their practical applications. Current evaluation tasks like VQA and visual grounding remain too coarse to assess…

「SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories」是 ModelScope 魔搭社区收录的论文，arXiv 2503.08625，作者为 Muzhi Zhu, Yuzhuo Tian, Hao Chen et al.，发表于 2025-03-11，属于 计算机视觉、多模态大语言模型、图像分割 领域。

- **ArXiv**: 2503.08625
- **Published**: 2025-03-11
- **Authors**: Muzhi Zhu, Yuzhuo Tian, Hao Chen, Chunluan Zhou, Qingpei Guo, Yang Liu, Ming Yang, Chunhua Shen
- **Model**: SegAgent
- **Developer**: 浙江大学、蚂蚁集团
- **Domain**: 计算机视觉, 多模态大语言模型, 图像分割, 强化学习, 智能体
- **ArXiv URL**: https://arxiv.org/abs/2503.08625
- **PDF**: https://arxiv.org/pdf/2503.08625
- **Code**: https://github.com/aim-uofa/SegAgent

Source: https://www.modelscope.cn/papers/125357

---

> 模仿人类标注员：SegAgent引领多模态大语言模型迈向精细像素理解的新纪元

## 摘要

本文提出了一种新的分割框架——Human-Like Mask Annotation Task (HLMAT)，该框架使多模态大语言模型（MLLMs）能够通过模仿人类标注员使用交互式分割工具的轨迹来完成像素级理解任务。现有的评估任务如VQA和视觉定位过于粗糙，无法准确评估细粒度的像素理解能力。此外，当前的方法通常需要MLLMs生成隐式标记并通过外部像素解码器进行解码，这会改变MLLM的文本输出空间，影响其语言能力和灵活性。HLMAT将分割任务建模为多步马尔可夫决策过程，使MLLMs能够迭代生成基于文本的点击点，从而在不改变架构或使用隐式标记的情况下实现高质量的掩码生成。通过这种方式，我们开发了SegAgent，一个在人类样式的标注轨迹上微调的模型，其性能与现有最先进的方法相当，并支持额外的任务如掩码细化和标注过滤。实验表明，SegAgent不仅在分割任务中表现出色，还展示了在复杂场景中的鲁棒性。我们的工作为未来探索MLLMs在细粒度视觉感知和多步决策方面的潜力奠定了基础。

## Abstract

While MLLMs have demonstrated adequate image understanding capabilities, they still struggle with pixel-level comprehension, limiting their practical applications. Current evaluation tasks like VQA and visual grounding remain too coarse to assess fine-grained pixel comprehension accurately. Though segmentation is foundational for pixel-level understanding, existing methods often require MLLMs to generate implicit tokens, decoded through external pixel decoders. This approach disrupts the MLLM's text output space, potentially compromising language capabilities and reducing flexibility and extensibility, while failing to reflect the model's intrinsic pixel-level understanding. Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new paradigm where MLLMs mimic human annotators using interactive segmentation tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT enables MLLMs to iteratively generate text-based click points, achieving high-quality masks without architectural changes or implicit tokens. Through this setup, we develop SegAgent, a model fine-tuned on human-like annotation trajectories, which achieves performance comparable to state-of-the-art (SOTA) methods and supports additional tasks like mask refinement and annotation filtering. HLMAT provides a protocol for assessing fine-grained pixel understanding in MLLMs and introduces a vision-centric, multi-step decision-making task that facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations of policy improvement method StaR and PRM-guided tree search further enhance model robustness in complex segmentation tasks, laying a foundation for future advancements in fine-grained visual perception and multi-step decision-making for MLLMs.
