---
title: big_model
canonical_url: "https://www.modelscope.cn/models/mapjack/big_model"
md_url: "https://www.modelscope.cn/models/mapjack/big_model.md"
repository: mapjack/big_model
chinese_name: Emu
last_updated: 2023-08-08
license: "Apache License 2.0"
pipeline_tag: image-captioning
tasks:
  - image-captioning
library_name:
  - pytorch
frameworks:
  - pytorch
downloads: 93
stars: 0
---

# big_model

> big_model - mapjack 在 ModelScope 开源的模型。Emu是一个多模式通才，可以无缝地在多模式上下文中生成图像和文本。Emu使用统一的自回归目标进行训练，即预测下一个元素，包括视觉嵌入和文本标记。在这个目标下训练，Emu可以作为图像到文本和文本到图像任务的通用接口。并且还是开源的，github开源地址为 https://github.com/baaivision/Emu，模型开源地址为…

mapjack/big_model 是 ModelScope 魔搭社区上的image-captioning模型，采用 Apache License 2.0 许可。

- **Repository**: mapjack/big_model
- **License**: Apache License 2.0
- **Tasks**: image-captioning
- **Downloads**: 93
- **Stars**: 0
- **Last updated**: 2023-08-08

Source: https://www.modelscope.cn/models/mapjack/big_model

---

**Emu is a multimodal generalist that can seamlessly generate images and texts in multimodal context**. **Emu** is trained with a unified autoregressive objective, *i.e.*, predict-the-next-element, including both visual embeddings and textual tokens. Trained under this objective, **Emu** can serve as a generalist interface for both image-to-text and text-to-image tasks.

![](assets/Emu.png)

## Generalist Interface

**Emu** serves as a generalist interface capable of diverse multimodal tasks, such as image captioning, image/video question answering, and text-to-image generation, together with new abilities like in-context text and image generation, and image blending:

![](assets/generalist.png)

## Setup

Clone this repository and install required packages:

```shell
git clone https://github.com/baaivision/Emu
cd Emu

pip install -r requirements.txt
```

## Model Weights

We release the pretrained and instruction-tuned weights of **Emu**. Our weights are subject to LLaMA's [license](https://github.com/facebookresearch/llama/blob/main/LICENSE).

| Model name | Weight                                                  |
| ---------- | ------------------------------------------------------- |
| **Emu**    | [🤗 HF link](https://huggingface.co/BAAI/Emu/blob/main/Emu-pretrain.pt) (27GB) |
| **Emu-I**  | [🤗 HF link](https://huggingface.co/BAAI/Emu/blob/main/Emu-instruct.pt) (27GB) |

## Inference

At present, we provide inference code that can process interleaved image-text and **video** as input, and output text.

For instruction-tuned model, we provide examples for image captioning, visual question answering, and interleaved multi-image understanding:

```sh
python inference.py --instruct --ckpt-path $Instruct_CKPT_PATH
```

For pretrained model, we provide an example for in-context learning:

```sh
python inference.py --ckpt-path $Pretrain_CKPT_PATH
```

## Schedule

We are committed to open-sourcing all Emu related materials, including:

- [x] The weights of **Emu** and **Emu-I**
- [x] Inference example for interleaved image-text as input, text as output
- [x] Video inference example
- [ ] Weights of image decoder & image generation/blending example
- [ ] YT-Storyboard-1B pretraining data
- [ ] Pretraining code
- [ ] Instruction tuning code
- [ ] Evaluation code

We hope to foster the growth of our community through open-sourcing and promoting collaboration👬. Let's step towards multimodal intelligence together🍻.

## Acknowledgement

We thank the great work from [LLaMA](https://github.com/facebookresearch/llama), [BLIP-2](https://github.com/salesforce/LAVIS), [Stable Diffusion](https://github.com/CompVis/stable-diffusion), and [FastChat](https://github.com/lm-sys/FastChat).

## Citation

If you find Emu useful for your research and applications, please consider starring this repository and citing:

```
@article{Emu,
  title={Generative Pretraining in Multimodality},
  author={Sun, Quan and Yu, Qiying and Cui, Yufeng and Zhang, Fan and Zhang, Xiaosong and Wang, Yueze and Gao, Hongcheng and Liu, Jingjing and Huang, Tiejun and Wang, Xinlong},
  publisher={arXiv preprint arXiv:2307.05222},
  year={2023},
}
```