---
title: Audio-Omni
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/Audio-Omni"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/Audio-Omni.md"
repository: AI-ModelScope/Audio-Omni
last_updated: 2026-06-20
license: cc-by-nc-4.0
pipeline_tag: any-to-any
tasks:
  - any-to-any
model_type:
  - audio_omni
library_name:
  - pytorch
frameworks:
  - pytorch
downloads: 6
stars: 0
---

# Audio-Omni

> Audio-Omni - AI-ModelScope 在 ModelScope 开源的模型。Unified Audio Understanding, Generation, and Editing (SIGGRAPH 2026)

AI-ModelScope/Audio-Omni 是 ModelScope 魔搭社区上的any-to-any模型，采用 cc-by-nc-4.0 许可。

- **Repository**: AI-ModelScope/Audio-Omni
- **License**: cc-by-nc-4.0
- **Tasks**: any-to-any
- **Downloads**: 6
- **Stars**: 0
- **Last updated**: 2026-06-20

Source: https://www.modelscope.cn/models/AI-ModelScope/Audio-Omni

---

# 🎛️ Audio-Omni

**Unified Audio Understanding, Generation, and Editing** (SIGGRAPH 2026)

[![GitHub](https://img.shields.io/badge/GitHub-Repository-black?logo=github)](https://github.com/ZeyueT/Audio-Omni)
[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://zeyuet.github.io/Audio-Omni/)
[![arXiv](https://img.shields.io/badge/arXiv-Paper-red)](https://huggingface.co/papers/2604.10708)

## 📖 Overview

Audio-Omni is the first end-to-end framework that unifies **understanding**, **generation**, and **editing** across general sound, music, and speech domains. It combines a frozen Multimodal Large Language Model (Qwen2.5-Omni) for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis.

## 🎯 Capabilities

- **Understanding**: Audio/video captioning, question answering
- **Generation**: Text-to-Audio, Text-to-Music, Video-to-Audio, Video-to-Music, Text-to-Speech, Voice Conversion
- **Editing**: Add, Remove, Extract, Style Transfer

## 🚀 Quick Start

### Installation

```bash
# Clone the GitHub repository
git clone https://github.com/ZeyueT/Audio-Omni.git
cd Audio-Omni

# Install dependencies
pip install -e .
conda install -c conda-forge ffmpeg libsndfile

# Download model from Hugging Face
huggingface-cli download HKUSTAudio/Audio-Omni --local-dir model/
```

### Sample Usage

```python
from audio_omni import AudioOmni
import torchaudio

# Load model
model = AudioOmni("model/Audio-Omni.json", "model/model.ckpt")

# 1. Understanding
response = model.understand(
    "Describe the sounds in this audio.", 
    audio="example.wav"
)
print(response)

# 2. Generation (Text-to-Audio)
audio = model.generate("T2A", prompt="A clock ticking.")
torchaudio.save("output.wav", audio, model.sample_rate)

# 3. Editing (Add a sound)
audio = model.edit("Add", "input.wav", desc="skateboarding")
torchaudio.save("output_add.wav", audio, model.sample_rate)
```

## 📦 Model Files

- `Audio-Omni.json` — Model configuration
- `model.ckpt` — Model checkpoint (~21 GB)
- `synchformer_state_dict.pth` — Synchformer checkpoint for video conditioning

## 🖥️ Gradio Demo

```bash
# Launch interactive demo
python run_gradio.py \
    --model-config model/Audio-Omni.json \
    --ckpt-path model/model.ckpt \
    --server-port 7777
```

## 📝 Citation

```bibtex
@article{tian2026audio,
  title={Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing},
  author={Tian, Zeyue and Yang, Binxin and Liu, Zhaoyang and Zhang, Jiexuan and Yuan, Ruibin and Yin, Hubery and Chen, Qifeng and Li, Chen and Lyu, Jing and Xue, Wei and others},
  journal={arXiv preprint arXiv:2604.10708},
  year={2026}
}
```

## 📄 License

CC-BY-NC-4.0 (Non-commercial use only). Commercial use of the model weights requires explicit written authorization from the authors. For commercial licensing inquiries, contact: ztianad@connect.ust.hk