---
title: "Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning"
canonical_url: "https://www.modelscope.cn/papers/125186"
md_url: "https://www.modelscope.cn/papers/125186.md"
arxiv_id: 2503.07002
published: 2025-03-10
last_updated: 2025-03-10
authors:
  - "Jiazheng Liu"
  - "Sipeng Zheng"
  - "Börje F. Karlsson"
  - "Zongqing Lu"
model_name: DiagNote
model_developer: "北京大学、北京智源人工智能研究院"
domain:
  - "计算机视觉"
  - "多模态大语言模型"
  - "多轮对话"
  - "视觉定位"
  - "视觉推理"
type:
  - "计算机视觉"
  - "多模态大语言模型"
  - "多轮对话"
  - "视觉定位"
  - "视觉推理"
  - "Computer Vision and Pattern Recognition"
arxiv_url: "https://arxiv.org/abs/2503.07002"
pdf_url: "https://arxiv.org/pdf/2503.07002"
---

# Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning

> Multimodal large language models (MLLMs), built on large-scale pre-trained vision towers and language models, have shown great capabilities in multimodal understanding. However, most existing MLLMs are trained on single-turn vision question-answering tasks,…

「Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning」是 ModelScope 魔搭社区收录的论文，arXiv 2503.07002，作者为 Jiazheng Liu, Sipeng Zheng, Börje F. Karlsson et al.，发表于 2025-03-10，属于 计算机视觉、多模态大语言模型、多轮对话 领域。

- **ArXiv**: 2503.07002
- **Published**: 2025-03-10
- **Authors**: Jiazheng Liu, Sipeng Zheng, Börje F. Karlsson, Zongqing Lu
- **Model**: DiagNote
- **Developer**: 北京大学、北京智源人工智能研究院
- **Domain**: 计算机视觉, 多模态大语言模型, 多轮对话, 视觉定位, 视觉推理
- **ArXiv URL**: https://arxiv.org/abs/2503.07002
- **PDF**: https://arxiv.org/pdf/2503.07002

Source: https://www.modelscope.cn/papers/125186

---

> DiagNote：通过笔记增强多轮次多模态对话中的聚焦与推理能力

## 摘要

本文针对多轮次多模态对话中现有的大规模多模态语言模型（MLLMs）在复杂对话中的不足，提出了新的方法和数据集。研究背景指出，当前的MLLMs大多训练于单轮次视觉问答任务，难以准确反映真实世界的人类对话情境，尤其是在高分辨率图像和多轮对话中容易出现焦点丢失和幻觉问题。为此，作者引入了MMDiag数据集，该数据集通过精心设计的规则和GPT辅助生成，涵盖了多个场景下的详细多轮对话，包括日常场景、表格数据和Minigrid环境，旨在更真实地模拟实际对话并提供更强的基准测试。

提出的方法是DiagNote，它由两个模块组成：Deliberate模块负责推理，Gaze模块负责聚焦。这两个模块通过交互实现链式思维（Chain-of-Thought）和注释功能，从而在整个多轮对话中保持对关键信息的关注。实验结果表明，DiagNote在MMDiag和其他基准测试上的表现显著优于现有模型，特别是在多模态信息的联合处理和推理方面。此外，MMDiag本身也作为一个更具挑战性的基准，有助于推动多轮次多模态对话领域的进一步研究。

## Abstract

Multimodal large language models (MLLMs), built on large-scale pre-trained vision towers and language models, have shown great capabilities in multimodal understanding. However, most existing MLLMs are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. In this paper, we introduce MMDiag, a multi-turn multimodal dialogue dataset. This dataset is collaboratively generated through deliberately designed rules and GPT assistance, featuring strong correlations between questions, between questions and images, and among different image regions; thus aligning more closely with real-world scenarios. MMDiag serves as a strong benchmark for multi-turn multimodal dialogue learning and brings more challenges to the grounding and reasoning capabilities of MLLMs. Further, inspired by human vision processing, we present DiagNote, an MLLM equipped with multimodal grounding and reasoning capabilities. DiagNote consists of two modules (Deliberate and Gaze) interacting with each other to perform Chain-of-Thought and annotations respectively, throughout multi-turn dialogues. We empirically demonstrate the advantages of DiagNote in both grounding and jointly processing and reasoning with vision and language information over existing MLLMs.