---
title: Rex-Omni
canonical_url: "https://www.modelscope.cn/models/IDEA-Research/Rex-Omni"
md_url: "https://www.modelscope.cn/models/IDEA-Research/Rex-Omni.md"
repository: IDEA-Research/Rex-Omni
last_updated: 2025-10-20
license: other
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - qwen2_5_vl
architectures:
  - Qwen2_5_VLForConditionalGeneration
base_model:
  - Qwen/Qwen2.5-VL-3B-Instruct
base_model_relation: finetune
parameters: 4.1B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
inference_backends:
  - "deploy_task vlm/text/emb"
  - "lmdeploy 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 2547
stars: 8
tags:
  - vision
  - object-detection
  - multimodal
  - ocr
  - keypoint-detection
  - visual-prompting
  - open-set-detection
  - object-pointing
---

# Rex-Omni

> Rex-Omni - IDEA-Research 在 ModelScope 开源的模型。This model is Rex-Omni, a 3B-parameter Multimodal Large Language Model (MLLM) presented in the paper "Detect Anything via Next Point Prediction". It is compatible with the Hugging Face transformers library and is…

IDEA-Research/Rex-Omni 是 ModelScope 魔搭社区上的 4.1B 参数image-text-to-text模型，采用 other 许可，基于 Qwen/Qwen2.5-VL-3B-Instruct 构建，可用 deploy_task vlm/text/emb、lmdeploy 0.9.1、sglang 0.5.2 部署。

- **Repository**: IDEA-Research/Rex-Omni
- **License**: other
- **Tasks**: image-text-to-text
- **Parameters**: 4.1B
- **Base model**: Qwen/Qwen2.5-VL-3B-Instruct
- **Inference backends**: deploy_task vlm/text/emb, lmdeploy 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: vision, object-detection, multimodal, ocr, keypoint-detection, visual-prompting, open-set-detection, object-pointing
- **Downloads**: 2547
- **Stars**: 8
- **Last updated**: 2025-10-20

Source: https://www.modelscope.cn/models/IDEA-Research/Rex-Omni

---

This model is **Rex-Omni**, a 3B-parameter Multimodal Large Language Model (MLLM) presented in the paper "[Detect Anything via Next Point Prediction](https://huggingface.co/papers/2510.12798)". It is compatible with the Hugging Face `transformers` library and is licensed under the [IDEA License 1.0](https://github.com/IDEA-Research/Rex-Omni/blob/main/LICENSE).

<div align=center>
  <img src="assets/logo.png" width=600 >
</div>

<h1 align="center">Detect Anything via Next Point Prediction</h1>

<div align=center>

<p align="center">
  <a href="https://rex-omni.github.io/">
    <img
      src="https://img.shields.io/badge/RexOmni-Website-BADFDB?style=flat-square&logo=deno&logoColor=violet&color=BADFDB"
      alt="RexThinker Website"
    />
  </a>
  <a href="https://arxiv.org/abs/2510.12798">
    <img
      src="https://img.shields.io/badge/RexOmni-Paper-Red%25red?logo=arxiv&logoColor=red&color=yellow"
      alt="RexThinker Paper on arXiv"
    />
  </a>
  <a href="https://huggingface.co/IDEA-Research/Rex-Omni">
    <img 
        src="https://img.shields.io/badge/RexOmni-Weight-orange?logo=huggingface&logoColor=yellow" 
        alt="RexThinker weight on Hugging Face"
    />
  </a>
  <a href="https://huggingface.co/spaces/Mountchicken/Rex-Omni">
    <img
      src="https://img.shields.io/badge/RexOmni-Demo-orange?logo=huggingface&logoColor=yellow" 
      alt="RexThinker Demo on Hugging Face"
    />
  </a>
  <a href="https://github.com/IDEA-Research/Rex-Omni">
    <img
      src="https://img.shields.io/badge/GitHub-Code-blue?logo=github&logoColor=white"
      alt="GitHub Code"
    />
  </a>
</p>

</div>

> Rex-Omni is a 3B-parameter Multimodal Large Language Model (MLLM) that redefines object detection and a wide range of other visual perception tasks as a simple next-token prediction problem.

<p align="center"><img src="assets/teaser.png" width="95%"></p>



## 🚀 Quick Start

### Installation

```bash
conda create -n rexomni -m python=3.10
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124
git clone https://github.com/IDEA-Research/Rex-Omni.git
cd Rex-Omni
pip install -v -e .
```

## 2. Quick Start: Using Rex-Omni for Detection

```python
from PIL import Image
from rex_omni import RexOmniWrapper, RexOmniVisualize

# Initialize model
model = RexOmniWrapper(
    model_path="IDEA-Research/Rex-Omni",
    backend="transformers"  # or "vllm"
)

# Load image
image = Image.open("your_image.jpg")

# Object Detection
results = model.inference(
    images=image,
    task="detection",
    categories=["person", "car", "dog"]
)

result = results[0]

# 4) Visualize
vis = RexOmniVisualize(
    image=image,
    predictions=result["extracted_predictions"],
    font_size=20,
    draw_width=5,
    show_labels=True,
)
vis.save("visualize.jpg")

```

## 3. Tutorials

We provide a series of tutorials to help you get started with Rex-Omni.
- [Detection Example](https://github.com/IDEA-Research/Rex-Omni/blob/master/tutorials/detection_example/_full_notebook.ipynb)
- [Pointing Example](https://github.com/IDEA-Research/Rex-Omni/blob/master/tutorials/pointing_example/_full_tutorial.ipynb)
- [OCR Example](https://github.com/IDEA-Research/Rex-Omni/blob/master/tutorials/ocr_example/_full_tutorial.ipynb)
- [Keypointing Example](https://github.com/IDEA-Research/Rex-Omni/blob/master/tutorials/keypointing_example/_full_tutorial.ipynb)
- [Visual Prompting Example](https://github.com/IDEA-Research/Rex-Omni/blob/master/tutorials/visual_prompting_example/_full_tutorial.ipynb)
- [Batch Inference Example](https://github.com/IDEA-Research/Rex-Omni/blob/master/tutorials/other_example/batch_inference.py)


## 📄 License

Rex-Omni is licensed under the [IDEA License 1.0](LICENSE), Copyright (c) IDEA. All Rights Reserved. This model is based on Qwen, which is licensed under the [Qwen RESEARCH LICENSE AGREEMENT](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE), Copyright (c) Alibaba Cloud. All Rights Reserved.

## 🔗 Links

- 🏠 [Homepage](https://rex-omni.github.io/)
- 🎮 [Demo](https://huggingface.co/spaces/Mountchicken/Rex-Omni)

## 📧 Contact

For questions and feedback, please contact us at:
- Email: jiangqing@idea.edu.cn
- GitHub Issues: [IDEA-Research/Rex-Omni](https://github.com/IDEA-Research/Rex-Omni/issues)

## 7. Citation
Rex-Omni comes from a series of prior works. If you’re interested, you can take a look.

- [RexThinker](https://arxiv.org/abs/2506.04034)
- [RexSeek](https://arxiv.org/abs/2503.08507)
- [ChatRex](https://arxiv.org/abs/2411.18363)
- [DINO-X](https://arxiv.org/abs/2411.14347)
- [Grounidng DINO 1.5](https://arxiv.org/abs/2405.10300)
- [T-Rex2](https://link.springer.com/chapter/10.1007/978-3-031-73414-4_3)
- [T-Rex](https://arxiv.org/abs/2311.13596)


```bibtex
@misc{jiang2025detectpointprediction,
      title={Detect Anything via Next Point Prediction}, 
      author={Qing Jiang and Junan Huo and Xingyu Chen and Yuda Xiong and Zhaoyang Zeng and Yihao Chen and Tianhe Ren and Junzhi Yu and Lei Zhang},
      year={2025},
      eprint={2510.12798},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.12798}, 
}
```