---
title: OmniUE-7B
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/OmniUE-7B"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/OmniUE-7B.md"
repository: AI-ModelScope/OmniUE-7B
last_updated: 2026-10-29
license: other
pipeline_tag: feature-extraction
tasks:
  - feature-extraction
model_type:
  - qwen2_5_omni_thinker
architectures:
  - Qwen2OmniNaViTThinkerForConditionalGeneration
base_model:
  - Qwen/Qwen2.5-Omni-7B
base_model_relation: finetune
library_name:
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 5
stars: 2
tags:
  - multimodal-embedding
  - any-to-any-retrieval
  - text-image-retrieval
  - video-retrieval
  - audio-retrieval
  - qwen2.5-omni
---

# OmniUE-7B

> OmniUE-7B - AI-ModelScope 在 ModelScope 开源的模型。OmniUE is an instruction-conditioned universal embedder that maps text, images, video, and audio into one 3,584-dimensional embedding space. It also supports visual interaction through bounding boxes or points,…

AI-ModelScope/OmniUE-7B 是 ModelScope 魔搭社区上的feature-extraction模型，采用 other 许可，基于 Qwen/Qwen2.5-Omni-7B 构建。

- **Repository**: AI-ModelScope/OmniUE-7B
- **License**: other
- **Tasks**: feature-extraction
- **Base model**: Qwen/Qwen2.5-Omni-7B
- **Tags**: multimodal-embedding, any-to-any-retrieval, text-image-retrieval, video-retrieval, audio-retrieval, qwen2.5-omni
- **Downloads**: 5
- **Stars**: 2
- **Last updated**: 2026-10-29

Source: https://www.modelscope.cn/models/AI-ModelScope/OmniUE-7B

---

# OmniUE-7B

OmniUE is an instruction-conditioned universal embedder that maps text, images, video, and audio into one 3,584-dimensional embedding space. It also supports visual interaction through bounding boxes or points, and audio interaction through time spans.

The checkpoint is a LoRA adapter on [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B), with a SAM-Audio + SAM3 interaction module.

## Resources

- [Paper](https://arxiv.org/abs/2608.27044)
- [Webpage](https://wywywang.github.io/omniue-demo/)

## Experimental Results

Best score in each column is highlighted. Rows tagged **Ours** are OmniUE.

### MMEB-v2-video — text-interactive video

![MMEB-v2-video results. OmniUE-7B overall is 51.8.](figures/mmeb-v2.png)

Gains of OmniUE-3B / 7B over the best baseline of the same size: overall +5.1 / +4.3.

### MAEB — text-interactive audio

![MAEB results. OmniUE-7B overall is 53.4.](figures/maeb.png)

OmniUE-7B is best overall (53.4). Against LCO-Emb-7B it leads 7 of 8 meta-tasks. Clustering stays higher for Qwen2-Audio and the CLAP models, and cross-modal retrieval stays higher for LCO-Emb.

### SCaR — visual-interactive text-image-to-text

![SCaR results. OmniUE-7B overall is 55.2.](figures/scar.png)

Overall gain over the best 2B / 7B baseline: +20.7 / +27.4 points. The task is region-level caption retrieval from an image, a text instruction, and a box.

### OmniCHOIR — omni-interactive audio retrieval

![OmniCHOIR results. OmniUE-7B reaches 28.4 Recall@1 with all three prompts.](figures/omnichoir.png)

Recall@1. For baselines that cannot take a span, the pair is “span written as text / audio cropped to the span.” Masks and spans each beat text-only prompting, and using all three prompts is best (28.4).

## Installation

Use a CUDA-enabled Python environment with Python 3.10 or newer:

```bash
git clone https://huggingface.co/Sony/OmniUE-7B
cd OmniUE-7B
pip install -e .
```

`ffmpeg` must be available on `PATH` to decode audio from video files.

The OmniUE adapter, SAM-Audio checkpoint, and SAM3 checkpoint are bundled in this repository. On first use, Hugging Face downloads the Qwen2.5-Omni-7B and T5 base-model dependencies automatically.

## Quick start

```python
from omniue import OmniUE

model = OmniUE.from_pretrained(".")

captions = [
    "A person plays a guitar on a stage.",
    "A dog runs across a field.",
    "A chef chops vegetables in a kitchen.",
]
caption_embeddings = model.encode_text(captions)

query_embedding = model.encode_video(
    "concert.mp4",
    instruction="Find the matching description.",
)
indices, scores = model.cosine_topk(query_embedding, caption_embeddings, k=3)

for index, score in zip(indices[0], scores[0]):
    print(f"{score:.4f}  {captions[index]}")
```

Run the same text/video retrieval example from the command line:

```bash
python examples/retrieve.py --video concert.mp4
```

## API

Every encoder returns a `float32` NumPy array with shape `[batch_size, 3584]`. The output is L2-normalized.

```python
text = model.encode_text(["a dog", "a cat"], instruction="")
image = model.encode_image("image.jpg", instruction="Find the main subject.")
video = model.encode_video("video.mp4", instruction="Represent this video.")
audio = model.encode_audio("audio.wav", instruction="Represent this sound.")
```

Use a visual prompt to embed a selected object:

```python
selected_object = model.encode_image(
    "image.jpg",
    instruction="Represent the selected object.",
    bbox=[0.1, 0.1, 0.9, 0.9],
)
```

Use a temporal audio prompt to embed a selected sound:

```python
selected_sound = model.encode_audio(
    "audio.wav",
    instruction="Represent the selected sound.",
    span=(3.0, 8.0),
)
```

`bbox` and point coordinates are normalized to `[0, 1]`. Bounding boxes use
`[x1, y1, x2, y2]`. Point prompts use `points=[[x, y], ...]` and
`point_labels=[1, 0, ...]`, where `1` denotes foreground and `0` background.

Run the lightweight package check:

```bash
python examples/smoke_test.py
```

Use `--with-model` to exercise text, image, audio, and video inference. Add
`--with-interactions` to test visual and audio interaction paths.

## Limitations and intended use

This release is intended for research on multimodal retrieval and interactive embedding. It has not been evaluated for high-stakes or safety-critical decisions. Performance can vary by domain, modality quality, language, and prompt phrasing. Users are responsible for respecting the licenses and acceptable-use terms of Qwen2.5-Omni, SAM-Audio, SAM3, and any input data.

## Licenses

This repository contains components under different licenses:

- The OmniUE-specific checkpoint weights and original inference wrapper are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), as described in the root `LICENSE`.
- The bundled SAM-Audio code and checkpoint retain Meta's SAM License in `sam-audio/LICENSE`.
- The bundled SAM3 code and `sam3.pt` retain Meta's SAM License in `SAM3_LICENSE`.
- Qwen-derived source files retain their upstream notices and licenses. Qwen2.5-Omni itself is downloaded from its [upstream repository](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) and is not relicensed by this repository.

The CC BY-NC 4.0 license does not apply to the bundled third-party materials.

## Citation

```bibtex
@inproceedings{wang2026omniue,
  title     = {Omni-Interactive Universal Embedder},
  author    = {Wang, Wei-Yao and Tateishi, Kazuya and Cui, Shuyang and
               Simon, Christian and Shibuya, Takashi and Takahashi, Shusuke and
               Mitsufuji, Yuki},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}
```