--- title: Nexus-Gen-Training-Dataset canonical_url: "https://www.modelscope.cn/datasets/DiffSynth-Studio/Nexus-Gen-Training-Dataset" md_url: "https://www.modelscope.cn/datasets/DiffSynth-Studio/Nexus-Gen-Training-Dataset.md" repository: DiffSynth-Studio/Nexus-Gen-Training-Dataset chinese_name: "Nexus-Gen训练数据集" last_updated: 2025-07-28 license: "Apache License 2.0" storage_size: "16 TB" downloads: 240575 stars: 14 --- # Nexus-Gen-Training-Dataset > Nexus-Gen-Training-Dataset - DiffSynth-Studio 在 ModelScope 开源的数据集。Nexus-Gen Training Dataset DiffSynth-Studio/Nexus-Gen-Training-Dataset 是 ModelScope 魔搭社区上的数据集，存储大小 16 TB，采用 Apache License 2.0 许可。 - **Repository**: DiffSynth-Studio/Nexus-Gen-Training-Dataset - **License**: Apache License 2.0 - **Storage size**: 16 TB - **Downloads**: 240575 - **Stars**: 14 - **Last updated**: 2025-07-28 Source: https://www.modelscope.cn/datasets/DiffSynth-Studio/Nexus-Gen-Training-Dataset --- # Nexus-Gen Training Dataset ## Intruduction This dataset contains the complete training data for [Nexus-Gen](https://www.modelscope.cn/models/DiffSynth-Studio/Nexus-GenV2), covering tasks including image understanding, generation, and editing. For details about training Nexus-Gen, please refer to the [tech report](https://arxiv.org/abs/2504.21356) and [github repo](https://github.com/modelscope/Nexus-Gen). All annotations are unified into the [standard messages format](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html), stored in JSONL files. A single annotation example includes: * `images`: List of relative image paths, e.g. `["t2i/Flux/000055/000055_495587.png"]` * `messages`: Messages for training the language model, e.g. ```json [ {"role": "user", "content": "Generate an image according to the following description: A charming cartoon mouse with a light brown fur coat stands upright on its hind legs."}, {"role": "assistant", "content": "Here is an image based on the description: "} ] ``` * id: Unique data identifier, e.g. `"EliGen_495587"` * source: Sample source, e.g. `"EliGen"` The total dataset size is 26.3M samples, with the following distribution: ![Distribution](asserts/dataset_distribution.jpg) To facilitate model training and open-source usage, we further partition the annotations into stage-level annotations and task-level annotations. Since annotation files are large, see ./previews for more sample annotations. ### Stage-Level Annotations 1. `autoregressive_model_pretraining_26.3M.jsonl`: Includes all 26.3 million training samples for pretraining the autoregressive model. 2. `autoregressive_model_aes_finetuning_4.3M.jsonl`: This subset is used for quality fine-tuning of the autoregressive model. Image generation samples are restricted to high-quality subsets, while image editing samples primarily come from the ImagePulse subset. 3. `generation_decoder_1.7M.jsonl`: Subset used to train Nexus-Gen’s generation decoder, containing only high-quality image generation samples. 4. `editing_decoder_0.85M.jsonl`: Subset used to train Nexus-Gen’s editing decoder, exclusively sourced from the ImagePulse image editing dataset. ### Task Level Annotations 1. `image_understanding_5.8M.jsonl`: This task is structured with multimodal inputs (image-text pairs) and text-only outputs, which serves as a direct indicator of model’s chat and understanding ability. While MLLMs inherently possess such cross-modal reasoning capabilities, this task is still critical during training to prevent capacity degradation. We adopt [Cambrian-7M](https://huggingface.co/datasets/nyu-visionx/Cambrian-10M) as the data source, a comprehensive dataset spanning multiple domains including optical character recognition, general visual question answering, language, counting, code, math and science tasks. To improve data quality, we re-annotate the answers for all samples with Qwen2.5-VL-72B. 2. `image_generation_13.3M.jsonl`: The input for this task is the textual description, and the output is an image. Our data sources comprise [Journey DB](https://huggingface.co/datasets/JourneyDB/JourneyDB), [AnyWord](https://modelscope.cn/datasets/iic/AnyWord-3M/summary), [Laion-High-Resolution](https://huggingface.co/datasets/laion/laion-high-resolution), [EliGen TrainSet](https://modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet), [FLUX-Aes](https://huggingface.co/datasets/gogoduan/flux_laion_aes), [FLUX-T2I](https://huggingface.co/datasets/jackyhate/text-to-image-2M) and [Blip3o-60K](https://huggingface.co/datasets/BLIP3o/BLIP3o-60k). To enhance annotation diversity, we employ a dual-captioning paradigm via Qwen2.5-VL-72B, generating both concise captions and elaborate descriptions for each image. During training, we stochastically sample these annotations with stratified ratios (20\% concise vs. 80\% elaborate) to balance brevity and contextual granularity. 3. `image_editing_6.3M.jsonl`: The input for editing task consists of an image and its corresponding editing instruction, and the output denotes the edited image. Our data sources encompass datasets such as [HQ-Edit](https://huggingface.co/datasets/UCSC-VLAA/HQ-Edit), [UltraEdit](https://huggingface.co/datasets/BleachNick/UltraEdit), [OmniEdit](https://huggingface.co/datasets/TIGER-Lab/OmniEdit-Filtered-1.2M), [StyleBooth](https://ali-vilab.github.io/stylebooth-page/) and [ImagePulse](https://modelscope.cn/collections/ImagePulse----tulvmaidong-7c3b8283a43e40). ## How To Use We provide a script to unzip the images from tar files to subfolders in `./images`: ```python import os import subprocess from tqdm import tqdm import multiprocessing from functools import partial source_tar_dir = "tars" target_image_dir = "images" sub_dirs = ['vqa', 'edit', 't2i/Anyword', 't2i/Blip3o', 't2i/Flux', 't2i/JourneyDB', 't2i/Laion'] tar_bases = [os.path.join(source_tar_dir, sub_dir) for sub_dir in sub_dirs] image_bases = [os.path.join(target_image_dir, sub_dir) for sub_dir in sub_dirs] for img_dir in image_bases: os.makedirs(img_dir, exist_ok=True) def unpack_tar(tar_path, dest_dir): cmd = [ "tar", "xf", tar_path, "-C", dest_dir ] subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) return os.path.join(dest_dir, os.path.basename(tar_path).replace('.tar', '')) all_tar_files = [] for tar_base, img_base in zip(tar_bases, image_bases): files = [f for f in os.listdir(tar_base) if f.endswith('.tar')] all_tar_files.extend([(os.path.join(tar_base, f), img_base) for f in files]) def process_tar(args): return unpack_tar(args[0], args[1]) num_processes = min(50, multiprocessing.cpu_count()) with multiprocessing.Pool() as pool: results = [] with tqdm(total=len(all_tar_files), desc="Unpacking TAR files") as pbar: for res in pool.imap_unordered(process_tar, all_tar_files): pbar.update(1) results.append(res) print(f"\nAll TAR files unpacked to: {target_image_dir}") print(f"Total unpacked folders: {len(results)}") ``` Furthermore, you can verify file correctness using the following code and inject your absolute path into the annotations: ```python import os import random import json from tqdm import tqdm def read_jsonl(file_path, num_samples=None): print(f"reading from {file_path}") data_list = [] samples = 0 with open(file_path, 'r', encoding='utf-8') as file: for line in tqdm(file): data = json.loads(line.strip()) data_list.append(data) samples += 1 if num_samples is not None and samples >= num_samples: break print(f"read {len(data_list)} samples") return data_list def save_jsonl(data, file_path): dir_path = os.path.dirname(file_path) os.makedirs(dir_path, exist_ok=True) print(f'saving to {file_path}') with open(file_path, 'w', encoding='utf-8') as file: for item in tqdm(data, desc='Saving', total=len(data)): json.dump(item, file, ensure_ascii=False) file.write('\n') print(f'saved {len(data)} samples') # path to the images directory images_path = "path_to_your_data/images" # path to annotations input_jsonl = 'stage_level_annotations/editing_decoder_0.85M.jsonl' output_jsonl = 'stage_level_annotations/editing_decoder_0.85M_abs_path.jsonl' preview_jsonl = 'previews/editing_decoder_0.85M_abs_path_preview1k.jsonl' datas = read_jsonl(input_jsonl) for data in tqdm(datas): if 'images' in data: images = [os.path.join(images_path, image) for image in data['images']] data['images'] = images save_jsonl(datas, output_jsonl) random.shuffle(datas) preview_set = datas[:1000] for data in preview_set: if 'images' in data: for image in images: assert os.path.exists(image), f"Image path {image} does not exist" save_jsonl(preview_set, preview_jsonl) ``` ### Citation ``` @misc{zhang2025nexusgenunifiedimageunderstanding, title={Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space}, author={Hong Zhang and Zhongjie Duan and Xingjun Wang and Yuze Zhao and Weiyi Lu and Zhipeng Di and Yixuan Xu and Yingda Chen and Yu Zhang}, year={2025}, eprint={2504.21356}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2504.21356}, } ```