---
title: Trident-Train
canonical_url: "https://www.modelscope.cn/datasets/OpenBMB/Trident-Train"
md_url: "https://www.modelscope.cn/datasets/OpenBMB/Trident-Train.md"
repository: OpenBMB/Trident-Train
last_updated: 2026-10-16
license: mit
storage_size: "18 GB"
downloads: 71
stars: 0
---

# Trident-Train

> Trident-Train - OpenBMB 在 ModelScope 开源的数据集。Official training data for Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

OpenBMB/Trident-Train 是 ModelScope 魔搭社区上的数据集，存储大小 18 GB，采用 mit 许可。

- **Repository**: OpenBMB/Trident-Train
- **License**: mit
- **Storage size**: 18 GB
- **Downloads**: 71
- **Stars**: 0
- **Last updated**: 2026-10-16

Source: https://www.modelscope.cn/datasets/OpenBMB/Trident-Train

---

<h1 align="center">Trident-Train</h1>

<p align="center">
  Official training data for <a href="https://arxiv.org/abs/2610.11816"><i>Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers</i></a>
</p>
<p align="center">
  <a href="https://chaos-in-texts.github.io/"><img src="https://img.shields.io/badge/%F0%9F%94%B1%20Project%20Page-chaos--in--texts-5b3fd1.svg" alt="Project Page"></a>
  <a href="https://arxiv.org/abs/2610.11816"><img src="https://img.shields.io/badge/arXiv-2610.11816-b31b1b.svg" alt="arXiv"></a>
  <a href="https://github.com/OpenBMB/Trident"><img src="https://img.shields.io/badge/GitHub-OpenBMB%2FTrident-blue.svg" alt="Code"></a>
  <a href="https://huggingface.co/OpenBMB/Trident-Qwen3VL-2B"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Model-Trident--Qwen3VL--2B-yellow.svg" alt="Model"></a>
  <a href="https://huggingface.co/datasets/OpenBMB/Trident-Train"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Data-Trident--Train-orange.svg" alt="Train data"></a>
  <a href="https://huggingface.co/datasets/OpenBMB/Trident-Eval"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Data-Trident--Eval-orange.svg" alt="Eval data"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg" alt="License"></a>
</p>


## Overview

Dense retrievers tend to assign higher similarity scores to *text* representations than to semantically equivalent *image* representations, which makes mixed-modality retrieval fragile. Trident-Train is the **modality-balanced** training set used to mitigate this bias.

Every training instance pairs one query with **three co-equal positive views** of the same document:

| View      | Content                                      | How it is built                                              |
| --------- | -------------------------------------------- | ------------------------------------------------------------ |
| **Text**  | A detailed description of the document image | Generated by [Qwen3-VL-235B-A22B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct) |
| **Image** | The original document image                  | Original image                                               |
| **Fused** | A short summary **+** the original image     | Summary generated by [Qwen3-VL-235B-A22B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct) |

The three views are optimized jointly with the Multi-Positive View InfoNCE loss (see the [paper](https://arxiv.org/abs/2610.11816) and the [code](https://github.com/OpenBMB/Trident)).

Three training files are provided:

| File                 | Domain                            | Source                       | #Queries |
| -------------------- | --------------------------------- | ---------------------------- | -------- |
| `train_vdoc.jsonl`   | Visual documents                  | OpenDocVQA                   | ~41K     |
| `train_nature.jsonl` | Natural images                    | MMEB training sets           | ~170K    |
| `train_all.jsonl`    | Visual documents + natural images | Union of the two files above | ~211K    |

`train_all.jsonl` is the data used to train the released [Trident-Qwen3VL-2B](https://huggingface.co/OpenBMB/Trident-Qwen3VL-2B). The two domain files can be used to train domain-specific retrievers.

## Directory Structure

```
Trident-Train/
├── README.md
├── train_all.jsonl
├── train_vdoc.jsonl
├── train_nature.jsonl
└── images/
    ├── aokvqa.tar.gz
    ├── docvqa.tar.gz
    ├── dude.tar.gz
    ├── fvqa.tar.gz
    ├── infovqa.tar.gz
    ├── mscoco.tar.gz
    ├── openwikitable.tar.gz
    ├── render_images.tar.gz
    ├── visualmrc.tar.gz
    ├── visualnews.tar.gz
    └── wiki.tar.gz
```

After downloading and extracting, the layout matches what the [Trident code](https://github.com/OpenBMB/Trident) expects under `data/`:

```
data/
└── Train/
    ├── train_all.jsonl
    ├── train_vdoc.jsonl
    ├── train_nature.jsonl
    └── images/
        ├── docvqa/ ...
        ├── mscoco/ ...
        └── ...
```

### Image archives

| Archive                | Source dataset                                  |
| ---------------------- | ----------------------------------------------- |
| `aokvqa.tar.gz`        | A-OKVQA                                         |
| `docvqa.tar.gz`        | DocVQA                                          |
| `dude.tar.gz`          | DUDE                                            |
| `fvqa.tar.gz`          | FVQA                                            |
| `infovqa.tar.gz`       | InfographicVQA                                  |
| `mscoco.tar.gz`        | MSCOCO                                          |
| `openwikitable.tar.gz` | OpenWikiTable                                   |
| `render_images.tar.gz` | Rendered images generated for the training data |
| `visualmrc.tar.gz`     | VisualMRC                                       |
| `visualnews.tar.gz`    | VisualNews                                      |
| `wiki.tar.gz`          | Wikipedia images                                |

Each archive extracts into `images/` and keeps its own sub-folder, so image paths in the `.jsonl` files resolve as `data/Train/images/<path>`.

## Download & Extract

### Option 1: one-line script (recommended)

The [Trident repository](https://github.com/OpenBMB/Trident) ships a helper script that downloads and extracts the data in the expected layout:

```bash
git clone https://github.com/OpenBMB/Trident.git && cd Trident

bash data/download_data.sh train                              # everything (train_all.jsonl + all images)
bash data/download_data.sh train --train-subset vdoc          # only train_vdoc.jsonl
bash data/download_data.sh train --train-images docvqa,dude   # only some image archives
bash data/download_data.sh train --mirror                     # use hf-mirror.com
```

### Option 2: manual

```bash
# 1. Download (everything, or pass --include "train_vdoc.jsonl" "images/docvqa.tar.gz" for a subset)
hf download OpenBMB/Trident-Train \
    --repo-type dataset --local-dir data/Train

# 2. Extract the image archives in place
for f in data/Train/images/*.tar.gz; do
    tar -xzf "$f" -C data/Train/images
done

# 3. (optional) remove archives to save disk space
# rm data/Train/images/*.tar.gz
```

Or download from Python:

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="OpenBMB/Trident-Train",
    repo_type="dataset",
    local_dir="data/Train",
    allow_patterns=["train_vdoc.jsonl", "images/docvqa.tar.gz"],   # omit to download everything
)
```

Then train with the code repository, e.g.:

```bash
bash scripts/train.sh configs/trident_qwen3vl.yaml \
     --set data.train_path=data/Train/train_all.jsonl
```

## Data Format

Every `.jsonl` file contains one JSON object per line. Image paths are **relative to `images/`** and use forward slashes.

```json
{"query": [{"text": "How many stores did Saint Laurent operate in Western Europe in 2020?"}],
 "positive": [{"text": "<detailed description>", "id": "d1_text"},
              {"image": "docvqa/d1.png", "id": "d1_image"},
              {"text": "<short summary>", "image": "docvqa/d1.png", "id": "d1_fused"}],
 "doc_id": "d1", "query_id": "q1", "task": "default"}
```

| Field                      | Description                                                  |
| -------------------------- | ------------------------------------------------------------ |
| `query`                    | A list with the query content (`text` and/or `image`).       |
| `positive`                 | Exactly three positive views of the same document, **always in the order text → image → fused**. Each view carries its own `id`. |
| `doc_id`, `query_id`       | Identities of the document and the query. Used to mask false negatives inside a batch. |
| `task`                     | Task tag (`"default"`).                                      |
| `positive_ids`, `group_id` | *Optional.* Extra identity fields for false-negative masking. Missing ids fall back to content hashes. |

| View                    | Fields present        |
| ----------------------- | --------------------- |
| Text                    | `id`, `text`          |
| Image                   | `id`, `image`         |
| Fused (summary + image) | `id`, `text`, `image` |

See [`src/mmemb/data/schema.py`](https://github.com/OpenBMB/Trident/blob/main/src/mmemb/data/schema.py) in the code repository for all supported fields.

> All ids (`id`, `doc_id`, `query_id`) are **strings**.

## Construction

Following Section 4.1 of the paper, we build training instances from OpenDocVQA (visual documents) and the MMEB training sets (natural images). For every image document, [Qwen3-VL-235B-A22B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct) generates (i) a detailed description (Text view) and (ii) a short summary that is paired with the image (Fused view). The original image is kept as the Image view, so each document is represented by three equally weighted positive views.

## License

The annotations released here are under the [MIT License](https://github.com/OpenBMB/Trident/blob/main/LICENSE). The images originate from third-party datasets (see [Image archives](#image-archives)) and remain subject to their **original licenses**; please make sure your use complies with them.

## Citation

If you find this work useful, please cite:

```bibtex
@article{sun2026chaos,
  title={Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers},
  author={Sun, Yubo and Peng, Chunyi and Yan, Yukun and Liu, Zhenghao and Xu, Zhipeng and Mei, Sen and Xin, Linlin and Zeng, Zheni and Sun, Maosong},
  journal={arXiv preprint arXiv:2610.11816},
  year={2026}
}
```
