--- title: ImagenWorld-annotated-set canonical_url: "https://www.modelscope.cn/datasets/TIGER-Lab/ImagenWorld-annotated-set" md_url: "https://www.modelscope.cn/datasets/TIGER-Lab/ImagenWorld-annotated-set.md" repository: TIGER-Lab/ImagenWorld-annotated-set last_updated: 2026-04-19 license: mit storage_size: "37 GB" downloads: 194 stars: 0 --- # ImagenWorld-annotated-set > ImagenWorld-annotated-set - TIGER-Lab 在 ModelScope 开源的数据集。🧮 ImagenWorld – Annotated Set TIGER-Lab/ImagenWorld-annotated-set 是 ModelScope 魔搭社区上的数据集，存储大小 37 GB，采用 mit 许可。 - **Repository**: TIGER-Lab/ImagenWorld-annotated-set - **License**: mit - **Storage size**: 37 GB - **Downloads**: 194 - **Stars**: 0 - **Last updated**: 2026-04-19 Source: https://www.modelscope.cn/datasets/TIGER-Lab/ImagenWorld-annotated-set --- # 🧮 ImagenWorld – Annotated Set [![Paper](https://img.shields.io/badge/%F0%9F%93%84-Paper-b31b1b)](https://arxiv.org/abs/2603.27862) [![Project Website](https://img.shields.io/badge/%F0%9F%8C%90-Project%20Website-deepgray)](https://tiger-ai-lab.github.io/ImagenWorld/) [![Visualization](https://img.shields.io/badge/%F0%9F%93%8A-Visualization-blue)](https://huggingface.co/spaces/TIGER-Lab/ImagenWorld-Visualizer) [![GitHub](https://img.shields.io/badge/Code-GitHub-darkblue)](https://github.com/TIGER-AI-Lab/ImagenWorld) **ImagenWorld** is a large-scale benchmark designed to **evaluate image generation and editing models** across realistic multimodal scenarios. It spans **six tasks** and **six domains**, providing a unified framework for assessing model compositionality, instruction following, and multimodal reasoning. This repository hosts the **Annotated Set** of ImagenWorld, containing **human evaluations** of model-generated outputs. Includes both `train` and `test` splits — only `train` contains human annotations; the `test` split is simply the remaining portion without manual evaluation. --- ## 📦 Dataset Access The dataset is organized as **zipped folders** by task for both `train` and `test` splits. ### 🐍 **Download with Python** ```python from huggingface_hub import snapshot_download import zipfile from pathlib import Path # Download annotated dataset local_path = snapshot_download( repo_id="TIGER-Lab/ImagenWorld-annotated-set", repo_type="dataset", local_dir="ImagenWorld-annotated-set", local_dir_use_symlinks=False, ) # Unzip all tasks for each split for split in ["train", "test"]: split_dir = Path(local_path) / split for zip_file in split_dir.glob("*.zip"): target_dir = split_dir / zip_file.stem target_dir.mkdir(exist_ok=True) with zipfile.ZipFile(zip_file, "r") as zf: zf.extractall(target_dir) print(f"✅ Extracted {zip_file.name} → {target_dir}") ``` --- ### 💻 **Download via Command Line** ```bash hf download TIGER-Lab/ImagenWorld-annotated-set --repo-type dataset --local-dir ImagenWorld-annotated-set cd ImagenWorld-annotated-set && for s in train test; do cd "$s"; for f in *.zip; do d="${f%.zip}"; mkdir -p "$d"; unzip -q "$f" -d "$d"; done; cd ..; done ``` --- ## 📁 Dataset Structure After extraction, your directory will look like this: ``` ImagenWorld-annotated-set/ │ ├── train/ │ ├── TIG.zip │ ├── TIE.zip │ ├── SRIG.zip │ ├── SRIE.zip │ ├── MRIG.zip │ └── MRIE.zip │ ├── test/ │ ├── TIG.zip │ ├── TIE.zip │ ├── SRIG.zip │ ├── SRIE.zip │ ├── MRIG.zip │ └── MRIE.zip ``` After unzipping, each task follows this internal structure: ### 🧩 `train/` split (with human evaluation) ``` TIG/ └── TIG_A_000001/ ├── input/ │ ├── metadata.json │ ├── 1.png │ └── ... └── outputs/ ├── sdxl/ │ ├── annotator1/ │ │ ├── evaluation.json │ │ ├── error_mask.png # optional; only if not 'None' or 'All' │ │ └── ... │ ├── annotator2/ │ ├── annotator3/ │ ├── out.png # model-generated output │ ├── som_segments.png # Set-of-Marks segmentation map (visual) │ └── som_segments.npz # corresponding NumPy map for the above └── gpt-image-1/ ├── ... ``` ### 🧠 `test/` split (without manual evaluation) Same structure as `train/`, except **no `annotatorX/` folders** are included: ``` TIG/ └── TIG_A_000001/ ├── input/ └── outputs/ ├── sdxl/ │ ├── out.png │ ├── som_segments.png │ └── som_segments.npz └── gpt-image-1/ ``` --- ## 🧾 File Descriptions | File | Description | |------|--------------| | `evaluation.json` | JSON file with annotator feedback and per-object or per-segment ratings. | | `error_mask.png` | Binary mask highlighting incorrectly generated regions (if annotator selected specific areas). | | `som_segments.png` | Visual segmentation map generated by the **Set-of-Marks (SoM)** model. | | `som_segments.npz` | NumPy array containing pixel-to-segment mappings corresponding to `som_segments.png`. | | `out.png` | The raw image generated by the model for this condition set. | | `metadata.json` | Input metadata and prompt from the original condition set. | --- ## 📊 Annotation Details - Human annotations were collected from **three independent annotators per model output**. - Each annotator could select: - `None` — no error found - `All` — the entire image contains severe issues - or mark **specific regions** using an error mask (`error_mask.png`). - Evaluations include **object-level**, **segment-level**, and **score-based** ratings. --- ## 🔗 Related Datasets | Component | Description | Repository | |------------|--------------|-------------| | **Condition Set** | Input prompts and reference images. | [`TIGER-Lab/ImagenWorld-condition-set`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld) | | **Model Outputs** | Generated images from all models used in evaluation. | [`TIGER-Lab/ImagenWorld-model-outputs`](https://huggingface.co/datasets/TIGER-Lab/ImagenWorld-model-outputs) | --- ## 🧠 Notes - The **`train/` split** includes **human annotations** from multiple annotators. - The **`test/` split** is simply the remaining portion **without** manual evaluation. - Segmentation files (`som_segments.*`) are included for all models to support error localization and structured comparison. --- ## 📜 Citation If you use **ImagenWorld**, please cite: ```bibtex @inproceedings{ sani2026imagenworld, title={ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks}, author={Samin Mahdizadeh Sani and Max Ku and Nima Jamali and Matina Mahdizadeh Sani and Paria Khoshtab and Wei-Chieh Sun and Parnian Fazel and Zhi Rui Tam and Thomas Chong and Edisy Kin Wai Chan and Donald Wai Tong Tsang and Chiao-Wei Hsu and Lam Ting Wai and Ho Yin Sam Ng and Chiafeng Chu and Chak-Wing Mak and Keming Wu and Hiu Tung Wong and Yik Chun Ho and Chi Ruan and Zhuofeng Li and I-Sheng Fang and Shih-Ying Yeh and Ho Kei Cheng and Ping Nie and Wenhu Chen}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://openreview.net/forum?id=bld9g6jFh9} } @misc{sani2026imagenworldstresstestingimagegeneration, title={ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks}, author={Samin Mahdizadeh Sani and Max Ku and Nima Jamali and Matina Mahdizadeh Sani and Paria Khoshtab and Wei-Chieh Sun and Parnian Fazel and Zhi Rui Tam and Thomas Chong and Edisy Kin Wai Chan and Donald Wai Tong Tsang and Chiao-Wei Hsu and Ting Wai Lam and Ho Yin Sam Ng and Chiafeng Chu and Chak-Wing Mak and Keming Wu and Hiu Tung Wong and Yik Chun Ho and Chi Ruan and Zhuofeng Li and I-Sheng Fang and Shih-Ying Yeh and Ho Kei Cheng and Ping Nie and Wenhu Chen}, year={2026}, eprint={2603.27862}, archivePrefix={arXiv}, primaryClass={cs.GR}, url={https://arxiv.org/abs/2603.27862}, } ```