---
title: "MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice"
canonical_url: "https://www.modelscope.cn/papers/124803"
md_url: "https://www.modelscope.cn/papers/124803.md"
arxiv_id: 2503.05978
published: 2025-03-07
last_updated: 2025-03-07
authors:
  - "Hongwei Yi"
  - "Tian Ye"
  - "Shitong Shao"
  - "Xuancheng Yang"
  - "Jiantong Zhao"
  - "Hanzhong Guo"
  - "Terrance Wang"
  - "Qingyu Yin"
  - "Zeke Xie"
  - "Lei Zhu"
  - "Wei Li"
  - "Michael Lingelbach"
  - "Daquan Zhou"
model_name: MagicInfinite
model_developer: "Hedra Inc.、HKUST(GZ)、HKU、Peking University"
domain:
  - "计算机视觉"
  - "视频生成"
  - "肖像动画"
  - "数字人"
  - "多模态生成"
type:
  - "计算机视觉"
  - "视频生成"
  - "肖像动画"
  - "数字人"
  - "多模态生成"
  - "Computer Vision and Pattern Recognition"
arxiv_url: "https://arxiv.org/abs/2503.05978"
pdf_url: "https://arxiv.org/pdf/2503.05978"
---

# MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice

> We present MagicInfinite, a novel diffusion Transformer (DiT) framework that overcomes traditional portrait animation limitations, delivering high-fidelity results across diverse character types-realistic humans, full-body figures, and stylized anime…

「MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice」是 ModelScope 魔搭社区收录的论文，arXiv 2503.05978，作者为 Hongwei Yi, Tian Ye, Shitong Shao et al.，发表于 2025-03-07，属于 计算机视觉、视频生成、肖像动画 领域。

- **ArXiv**: 2503.05978
- **Published**: 2025-03-07
- **Authors**: Hongwei Yi, Tian Ye, Shitong Shao, Xuancheng Yang, Jiantong Zhao, Hanzhong Guo, Terrance Wang, Qingyu Yin, Zeke Xie, Lei Zhu, Wei Li, Michael Lingelbach, Daquan Zhou
- **Model**: MagicInfinite
- **Developer**: Hedra Inc.、HKUST(GZ)、HKU、Peking University
- **Domain**: 计算机视觉, 视频生成, 肖像动画, 数字人, 多模态生成
- **ArXiv URL**: https://arxiv.org/abs/2503.05978
- **PDF**: https://arxiv.org/pdf/2503.05978

Source: https://www.modelscope.cn/papers/124803

---

> MagicInfinite：融合文字与声音的无限对话视频生成新突破

## 摘要

本文提出了一种名为MagicInfinite的新颖扩散Transformer (DiT) 框架，旨在克服传统肖像动画的局限性，生成高质量、多样化的角色动画视频。MagicInfinite支持多种面部姿态和多角色场景，并通过输入掩码实现精确的演讲者指定。该方法通过以下三项创新解决了关键挑战：(1) 使用3D全注意力机制和滑动窗口去噪策略，确保不同风格角色的无限视频生成具备时间一致性和视觉质量；(2) 采用两阶段课程学习方案，结合音频进行唇同步、文本表达动态和参考图像保持身份，实现对长序列的灵活多模态控制；(3) 利用区域特定掩码和自适应损失函数平衡全局文本控制和局部音频引导，支持特定演讲者的动画。此外，该方法通过统一步骤和分类器自由引导（CFG）蒸馏技术显著提高了推理效率，在8个H100 GPU上生成10秒540x540p视频仅需10秒，720x720p视频需30秒。实验结果表明，MagicInfinite在唇音同步、身份保持和动作自然性方面表现优异。

## Abstract

We present MagicInfinite, a novel diffusion Transformer (DiT) framework that overcomes traditional portrait animation limitations, delivering high-fidelity results across diverse character types-realistic humans, full-body figures, and stylized anime characters. It supports varied facial poses, including back-facing views, and animates single or multiple characters with input masks for precise speaker designation in multi-character scenes. Our approach tackles key challenges with three innovations: (1) 3D full-attention mechanisms with a sliding window denoising strategy, enabling infinite video generation with temporal coherence and visual quality across diverse character styles; (2) a two-stage curriculum learning scheme, integrating audio for lip sync, text for expressive dynamics, and reference images for identity preservation, enabling flexible multi-modal control over long sequences; and (3) region-specific masks with adaptive loss functions to balance global textual control and local audio guidance, supporting speaker-specific animations. Efficiency is enhanced via our innovative unified step and cfg distillation techniques, achieving a 20x inference speed boost over the basemodel: generating a 10 second 540x540p video in 10 seconds or 720x720p in 30 seconds on 8 H100 GPUs, without quality loss. Evaluations on our new benchmark demonstrate MagicInfinite's superiority in audio-lip synchronization, identity preservation, and motion naturalness across diverse scenarios. It is publicly available at https://www.hedra.com/, with examples at https://magicinfinite.github.io/.
