---
title: ONE-PEACE-4B
canonical_url: "https://www.modelscope.cn/models/iic/ONE-PEACE-4B"
md_url: "https://www.modelscope.cn/models/iic/ONE-PEACE-4B.md"
repository: iic/ONE-PEACE-4B
chinese_name: "ONE-PEACE-通用表征模型-英文-4B"
last_updated: 2024-01-09
license: "Apache License 2.0"
library_name:
  - pytorch
frameworks:
  - pytorch
domain:
  - multi-modal
downloads: 1021
stars: 27
tags:
  - Alibaba
  - ONE-PEACE
  - OFA
  - M6
  - "arxiv:2305.11172"
  - "多模态"
  - "语音"
  - "图像"
  - "文本"
---

# ONE-PEACE-4B

> ONE-PEACE-4B - iic 在 ModelScope 开源的模型。ONE-PEACE是一个4B规模的图文音通用表征模型，可以产出图文音通用表征，实现三个模态的互相检索

iic/ONE-PEACE-4B 是 ModelScope 魔搭社区上的机器学习模型，采用 Apache License 2.0 许可。

- **Repository**: iic/ONE-PEACE-4B
- **License**: Apache License 2.0
- **Tags**: Alibaba, ONE-PEACE, OFA, M6, arxiv:2305.11172, 多模态, 语音, 图像, 文本
- **Downloads**: 1021
- **Stars**: 27
- **Last updated**: 2024-01-09

Source: https://www.modelscope.cn/models/iic/ONE-PEACE-4B

---

# ONE-PEACE

<p align="center">
    <br>
    <img src="assets/logo.png" width="350" />
    <br>
<p>
<p align="center">
        <a href="https://arxiv.org/abs/2305.11172">Paper</a>&nbsp&nbsp ｜ &nbsp&nbsp<a href="https://huggingface.co/spaces/OFA-Sys/ONE-PEACE_Multimodal_Retrieval">Demo</a>&nbsp&nbsp | &nbsp&nbsp<a href="checkpoints.md">Checkpoints</a>&nbsp&nbsp ｜ &nbsp&nbsp<a href="datasets.md">Datasets</a>&nbsp&nbsp | &nbsp&nbsp<a href="https://github.com/OFA-Sys/ONE-PEACE">GitHub</a>
</p>
<br></br>


# ONE-PEACE是什么
ONE-PEACE是一个图文音三模态通用表征模型，在语义分割、音文检索、音频分类和视觉定位几个任务都达到了新SOTA表现，在视频分类、图像分类图文检索、以及多模态经典benchmark也都取得了比较领先的结果。
另外，模型展现出来新的zeroshot能力，即实现了新的模态对齐，比如音频和图像的对齐，或者音频+文字和图像的对齐，而这类数据并没有出现在我们的预训练数据集里。

下面这张图展示了ONE-PEACE的模型架构和预训练任务。借助于扩展友好的架构和模态无关的任务，ONE-PEACE具备扩展到无限模态的潜力

<p align="center">
<img src="assets/one_peace.png" width=100%>
</p>

<br></br>

# 如何玩转ONE-PEACE
## 基础配置
```shell
# modelscope的notebook不需要安装modelscope
# modelscope镜像请使用
# GPU： registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.3.0-py38-torch1.11.0-tf1.15.5-1.8.1
# CPU： registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py38-torch1.11.0-tf1.15.5-1.8.1
# 需要torch<2.0.0
# pip install modelscope
git clone https://github.com/OFA-Sys/ONE-PEACE
cd ONE-PEACE 
pip install -r requirements.txt
```

## 开始玩起来
```python
from modelscope.models import Model
from modelscope.pipelines import pipeline

inference = pipeline('multimodal_embedding', model='damo/ONE-PEACE-4B', model_revision='v1.0.2', use_gpu=False)
text_features = inference(["bird", "dog", "panda"], data_type='text')
image_features = inference(
    ['https://one-peace-shanghai.oss-cn-shanghai.aliyuncs.com/modelscope_case/dog.JPEG', 'https://one-peace-shanghai.oss-cn-shanghai.aliyuncs.com/modelscope_case/panda.JPEG'],
    data_type='image'
)
audio_features = inference(
    ['https://one-peace-shanghai.oss-cn-shanghai.aliyuncs.com/modelscope_case/bird.flac', 'https://one-peace-shanghai.oss-cn-shanghai.aliyuncs.com/modelscope_case/dog.flac'],
    data_type='audio'
)

# compute similarity
i2t_similarity = image_features @ text_features.T
a2t_similarity = audio_features @ text_features.T
print("Image-to-text similarities:", i2t_similarity)
print("Audio-to-text similarities:", a2t_similarity)
```
<br></br>

# 为什么ONE-PEACE是多模态表征模型的最佳选择？
作为一个4B规模的通用表征模型，ONE-PEACE在一系列视觉、语音和多模态任务上取得领先的结果。
此外，ONE-PEACE还具备强大的多模态检索能力，能够完成图文音三模态之间的互相检索。

## 下游任务结果
### 视觉任务
<table border="1" width="100%">
    <tr align="center">
        <th>Task</th><th>Image classification</th><th>Semantic Segmentation</th><th>Object Detection (w/o Object365)</th><th>Video Action Recognition</th>
    </tr>
    <tr align="center">
        <td>Dataset</td><td>Imagenet-1K</td><td>ADE20K</td><td>COCO</td><td>Kinetics 400</td>
    </tr>
    <tr align="center">
        <td>Split</td><td>val</td><td>val</td><td>val</td><td>val</td>
    </tr>
    <tr align="center">
        <td>Metric</td><td>Acc.</td><td>mIoU<sup>ss</sup> / mIoU<sup>ms</sup></td><td>AP<sup>box</sup> / AP<sup>mask</sup></td><td>Top-1 Acc. / Top-5 Acc.</td>
    </tr>
    <tr align="center">
        <td>ONE-PEACE</td><td>89.8</td><td>62.0 / 63.0</td><td>60.4 / 52.9</td><td>88.1 / 97.8</td>
    </tr>
</table>

### 语音（-文本）任务
<table border="1" width="100%">
    <tr align="center">
        <th>Task</th><th colspan="4">Audio-Text Retrieval</th><th colspan="3">Audio Classification</th><th>Audio Question Answering</th>
    </tr>
    <tr align="center">
        <td>Dataset</td><td colspan="2">AudioCaps</td><td colspan="2">Clotho</td><td>ESC-50</td><td>FSD50K</td><td>VGGSound (Audio Only)</td><td>AVQA (Audio + Question)</td>
    </tr>
    <tr align="center">
        <td>Split</td><td colspan="2">test</td><td colspan="2">evaluation</td><td>full</td><td>eval</td><td>test</td><td>val</td>
    </tr>
    <tr align="center">
        <td>Metric</td><td>T2A R@1</td><td>A2T R@1</td><td>T2A R@1</td><td>A2T R@1</td><td>Zero-shot Acc.</td><td>MAP</td><td>Acc.</td><td>Acc.</td>
    </tr>
    <tr align="center">
        <td>ONE-PEACE</td><td>42.5</td><td>51.0</td><td>22.4</td><td>27.1</td><td>91.8</td><td>69.7</td><td>59.6</td><td>86.2</td>
    </tr>
</table>

### 图文多模态任务
<table border="1" width="100%">
    <tr align="center">
        <th>Task</th><th colspan="4">Image-Text Retrieval (w/o ranking)</th><th colspan="3">Visual Grounding</th><th>VQA</th><th>Visual Reasoning</th>
    </tr>
    <tr align="center">
        <td>Dataset</td><td colspan="2">COCO</td><td colspan="2">Flickr30K</td><td>RefCOCO</td><td>RefCOCO+</td><td>RefCOCOg</td><td>VQAv2</td><td>NLVR2</td>
    </tr>
    <tr align="center">
        <td>Split</td><td colspan="2">test</td><td colspan="2">test</td><td>val / testA / testB</td><td>val / testA / testB</td><td>val-u / test-u</td><td>test-dev / test-std</td><td>dev / test-P</td>
    </tr>
    <tr align="center">
        <td>Metric</td><td>I2T R@1</td><td>T2I R@1</td><td>I2T R@1</td><td>T2I R@1</td><td colspan="3">Acc@0.5</td><td>Acc.</td><td>Acc.</td>
    </tr>
    <tr align="center">
        <td>ONE-PEACE</td><td>84.1</td><td>65.4</td><td>97.6</td><td>89.6</td><td>92.58 / 94.18 / 89.26</td><td>88.77 / 92.21 / 83.23</td><td>89.22 / 89.27</td><td>82.6 / 82.5</td><td>87.8 / 88.3</td>
    </tr>
</table>
<br>

## 多模态检索
如下图所示，我们通过case展示了ONE-PEACE的音搜图，音+图搜图，以及音+文搜图的能力。

![a2i](assets/audio2img.png)

![at2i](assets/audiotext2img.png)

![ai2i](assets/audioimg2img.png)
<br></br>

# 模型局限性以及可能的偏差
模型主要使用开源的英文数据进行训练，因此中文的表征能力可能不太理想
<br></br>

# 相关论文以及引用
如果你觉得ONE-PEACE好用，欢迎引用我们的工作：

```BibTeX
@article{wang2023one,
  title={ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities},
  author={Wang, Peng and Wang, Shijie and Lin, Junyang and Bai, Shuai and Zhou, Xiaohuan and Zhou, Jingren and Wang, Xinggang and Zhou, Chang},
  journal={arXiv preprint arXiv:2305.11172},
  year={2023}
}
```