---
title: "Large-scale Pre-training for Grounded Video Caption Generation"
canonical_url: "https://www.modelscope.cn/papers/126916"
md_url: "https://www.modelscope.cn/papers/126916.md"
arxiv_id: 2503.10781
published: 2025-03-13
last_updated: 2025-03-13
authors:
  - "Evangelos Kazakos"
  - "Cordelia Schmid"
  - "Josef Sivic"
model_name: GROVE
model_developer: "捷克技术大学 Prague 的捷克信息学、机器人学与控制论研究所, 法国国家信息与自动化研究所 (Inria)"
domain:
  - "计算机视觉"
  - "自然语言处理"
  - "深度学习"
type:
  - "计算机视觉"
  - "自然语言处理"
  - "深度学习"
  - "Computer Vision and Pattern Recognition (cs.CV)"
arxiv_url: "https://arxiv.org/abs/2503.10781"
pdf_url: "https://arxiv.org/pdf/2503.10781.pdf"
---

# Large-scale Pre-training for Grounded Video Caption Generation

> We propose a novel approach for captioning and object grounding in video, where the objects in the caption are grounded in the video via temporally dense bounding boxes. We introduce the following contributions. First, we present a large-scale automatic…

「Large-scale Pre-training for Grounded Video Caption Generation」是 ModelScope 魔搭社区收录的论文，arXiv 2503.10781，作者为 Evangelos Kazakos, Cordelia Schmid, Josef Sivic，发表于 2025-03-13，属于 计算机视觉、自然语言处理、深度学习 领域。

- **ArXiv**: 2503.10781
- **Published**: 2025-03-13
- **Authors**: Evangelos Kazakos, Cordelia Schmid, Josef Sivic
- **Model**: GROVE
- **Developer**: 捷克技术大学 Prague 的捷克信息学、机器人学与控制论研究所, 法国国家信息与自动化研究所 (Inria)
- **Domain**: 计算机视觉, 自然语言处理, 深度学习
- **ArXiv URL**: https://arxiv.org/abs/2503.10781
- **PDF**: https://arxiv.org/pdf/2503.10781.pdf

Source: https://www.modelscope.cn/papers/126916

---

> GROVE：让视频字幕生成与对象定位更智能、更精准的新突破

## 摘要

本文针对视频中的对象定位和生成描述性字幕的任务，提出了一种新的方法。研究背景方面，现有数据集规模有限且标注方式单一，无法满足复杂场景下多对象的时空定位需求。为解决这一问题，作者提出了GROVE（Grounded Video Caption gEneration）模型，并开发了两种数据集：HowToGround1M和iGround。在方法上，作者设计了一种大规模自动标注方法，通过结合图像级接地字幕生成模型与大型语言模型（LLM），将帧级标注聚合为视频级字幕，并确保时间一致性。具体分为三个阶段：帧级接地字幕生成、视频级字幕聚合以及时间一致的对象框标注。取得的结果方面，GROVE模型在提出的iGround数据集以及VidSTG和ActivityNet-Entities等基准数据集上均达到了最先进的性能。此外，作者还通过广泛的消融实验验证了预训练和微调的重要性，以及模型各组件的关键贡献。本研究的价值在于推动了视频接地字幕生成技术的发展，为人类-机器人交互和具身感知等领域提供了重要支持。

## Abstract

We propose a novel approach for captioning and object grounding in video, where the objects in the caption are grounded in the video via temporally dense bounding boxes. We introduce the following contributions. First, we present a large-scale automatic annotation method that aggregates captions grounded with bounding boxes across individual frames into temporally dense and consistent bounding box annotations. We apply this approach on the HowTo100M dataset to construct a large-scale pre-training dataset, named HowToGround1M. We also introduce a Grounded Video Caption Generation model, dubbed GROVE, and pre-train the model on HowToGround1M. Second, we introduce a new dataset, called iGround, of 3500 videos with manually annotated captions and dense spatio-temporally grounded bounding boxes. This allows us to measure progress on this challenging problem, as well as to fine-tune our model on this small-scale but high-quality data. Third, we demonstrate that our approach achieves state-of-the-art results on the proposed iGround dataset compared to a number of baselines, as well as on the VidSTG and ActivityNet-Entities datasets. We perform extensive ablations that demonstrate the importance of pre-training using our automatically annotated HowToGround1M dataset followed by fine-tuning on the manually annotated iGround dataset and validate the key technical contributions of our model.