---
title: mmE5-mllama-11b-instruct
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/mmE5-mllama-11b-instruct"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/mmE5-mllama-11b-instruct.md"
repository: AI-ModelScope/mmE5-mllama-11b-instruct
last_updated: 2025-02-14
license: mit
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - mllama
architectures:
  - MllamaForConditionalGeneration
parameters: 10.6B
tensor_type:
  - BF16
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - pytorch
language:
  - en
  - ar
  - zh
  - ko
  - ru
  - pl
  - tr
  - fr
inference_backends:
  - "deploy_task vlm/text/emb"
  - "lmdeploy 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 78
stars: 0
tags:
  - mmeb
  - transformers
---

# mmE5-mllama-11b-instruct

> mmE5-mllama-11b-instruct - AI-ModelScope 在 ModelScope 开源的模型。mmE5-mllama-11b-instruct

AI-ModelScope/mmE5-mllama-11b-instruct 是 ModelScope 魔搭社区上的 10.6B 参数image-text-to-text模型，采用 mit 许可，可用 deploy_task vlm/text/emb、lmdeploy 0.9.1、sglang 0.5.2 部署。

- **Repository**: AI-ModelScope/mmE5-mllama-11b-instruct
- **License**: mit
- **Tasks**: image-text-to-text
- **Parameters**: 10.6B
- **Inference backends**: deploy_task vlm/text/emb, lmdeploy 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: mmeb, transformers
- **Downloads**: 78
- **Stars**: 0
- **Last updated**: 2025-02-14

Source: https://www.modelscope.cn/models/AI-ModelScope/mmE5-mllama-11b-instruct

---

## mmE5-mllama-11b-instruct

[mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data](https://arxiv.org/abs/2502.08468.pdf). Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou, arXiv 2024

This model is trained based on [Llama-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision).

[Github](https://github.com/haon-chen/mmE5)

## Train/Eval Data
 - Train data: https://huggingface.co/datasets/Haon-Chen/mmE5-MMEB-hardneg, https://huggingface.co/datasets/Haon-Chen/mmE5-synthetic
 - Eval data: https://huggingface.co/datasets/TIGER-Lab/MMEB-eval, https://huggingface.co/datasets/Haon-Chen/XTD-10

## Experimental Results
Our model achieves SOTA performance on MMEB benchmark.
<img width="900" alt="abs" src="https://raw.githubusercontent.com/haon-chen/mmE5/refs/heads/main/figures//exp_result.jpg">

## Usage

Below is an example we adapted from [VLM2Vec](https://huggingface.co/TIGER-Lab/VLM2Vec-Full).

First clone github
```bash
git clone https://github.com/haon-chen/mmE5.git
pip install -r requirements.txt
```

Then you can enter the directory to run the following command.
```python
from src.model import MMEBModel
from src.arguments import ModelArguments
from src.utils import load_processor
import torch
from transformers import HfArgumentParser, AutoProcessor
from PIL import Image
import numpy as np

model_args = ModelArguments(
    model_name='Haon-Chen/mmE5-mllama-11b-instruct',
    processor_name='meta-llama/Llama-3.2-11B-Vision',
    pooling='last',
    normalize=True,
    model_backbone='mllama')

processor = load_processor(model_args)

model = MMEBModel.load(model_args)
model.eval()
model = model.to('cuda', dtype=torch.bfloat16)


# Image + Text -> Text
inputs = processor(text='<|image|><|begin_of_text|> Represent the given image with the following question: What is in the image', images=[Image.open(
    'figures/example.jpg')], return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
qry_output = model(qry=inputs)["qry_reps"]

string = 'A cat and a dog'
inputs = processor(text=string, return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
tgt_output = model(tgt=inputs)["tgt_reps"]
print(string, '=', model.compute_similarity(qry_output, tgt_output))
## A cat and a dog = tensor([[0.3965]], device='cuda:0', dtype=torch.bfloat16)

string = 'A cat and a tiger'
inputs = processor(text=string, return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
tgt_output = model(tgt=inputs)["tgt_reps"]
print(string, '=', model.compute_similarity(qry_output, tgt_output))
## A cat and a tiger = tensor([[0.3105]], device='cuda:0', dtype=torch.bfloat16)

# Text -> Image
inputs = processor(text='Find me an everyday image that matches the given caption: A cat and a dog.', return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
qry_output = model(qry=inputs)["qry_reps"]

string = '<|image|><|begin_of_text|> Represent the given image.'
inputs = processor(text=string, images=[Image.open('figures/example.jpg')], return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
tgt_output = model(tgt=inputs)["tgt_reps"]
print(string, '=', model.compute_similarity(qry_output, tgt_output))
## <|image|><|begin_of_text|> Represent the given image. = tensor([[0.4219]], device='cuda:0', dtype=torch.bfloat16)

inputs = processor(text='Find me an everyday image that matches the given caption: A cat and a tiger.', return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
qry_output = model(qry=inputs)["qry_reps"]

string = '<|image|><|begin_of_text|> Represent the given image.'
inputs = processor(text=string, images=[Image.open('figures/example.jpg')], return_tensors="pt")
inputs = {key: value.to('cuda') for key, value in inputs.items()}
tgt_output = model(tgt=inputs)["tgt_reps"]
print(string, '=', model.compute_similarity(qry_output, tgt_output))
## <|image|><|begin_of_text|> Represent the given image. = tensor([[0.3887]], device='cuda:0', dtype=torch.bfloat16)
```

## Citation
```
@article{chen2025mmE5,
  title={mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data},
  author={Chen, Haonan and Wang, Liang and Yang, Nan and Zhu, Yutao and Zhao, Ziliang and Wei, Furu and Dou, Zhicheng},
  journal={arXiv preprint arXiv:2502.08468},
  year={2025}
}
```