--- title: UniDoc-Bench canonical_url: "https://www.modelscope.cn/datasets/Salesforce/UniDoc-Bench" md_url: "https://www.modelscope.cn/datasets/Salesforce/UniDoc-Bench.md" repository: Salesforce/UniDoc-Bench last_updated: 2026-06-27 license: cc-by-nc-4.0 storage_size: "18 GB" downloads: 106692 stars: 1 --- # UniDoc-Bench > UniDoc-Bench - Salesforce 在 ModelScope 开源的数据集。UNIDOC-BENCH Dataset Salesforce/UniDoc-Bench 是 ModelScope 魔搭社区上的数据集，存储大小 18 GB，采用 cc-by-nc-4.0 许可。 - **Repository**: Salesforce/UniDoc-Bench - **License**: cc-by-nc-4.0 - **Storage size**: 18 GB - **Downloads**: 106692 - **Stars**: 1 - **Last updated**: 2026-06-27 Source: https://www.modelscope.cn/datasets/Salesforce/UniDoc-Bench --- # UNIDOC-BENCH Dataset [![arXiv](https://img.shields.io/badge/arXiv-2510.03663-b31b1b.svg)](https://arxiv.org/abs/2510.03663) [![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/) [![GitHub](https://img.shields.io/badge/GitHub-Repository-black?logo=github)](https://github.com/SalesforceAIResearch/UniDoc-Bench) [![Dataset](https://img.shields.io/badge/🤗%20Hugging%20Face-Dataset-blue)](https://huggingface.co/datasets/Salesforce/UniDoc-Bench) A unified benchmark for document-centric multimodal retrieval-augmented generation (MM-RAG). UNIDOC-BENCH Domain Distribution ## Table of Contents - [Dataset Description](#dataset-description) - [Key Features](#key-features) - [Dataset Structure](#dataset-structure) - [Domain Descriptions](#domain-descriptions) - [QA Dataset Format](#qa-dataset-format) - [Field Descriptions](#field-descriptions) - [Query Types](#query-types) - [Data Access](#data-access) - [Download Options](#-download-options) - [Compressed File Sizes](#-compressed-file-sizes) - [Usage](#-usage) - [Loading the Dataset](#-loading-the-dataset) - [Working with PDF Files from Hugging Face](#-working-with-pdf-files-from-hugging-face) - [Building RAG Database from PDFs](#️-building-rag-database-from-pdfs) - [Working with PDF Documents](#-working-with-pdf-documents) - [Working with QA Pairs](#-working-with-qa-pairs) - [Visual Question Answering (VQA)](#-visual-question-answering-vqa) - [Evaluation Paradigms](#evaluation-paradigms) - [Benchmark Results](#benchmark-results) - [Data Quality](#data-quality) - [Citation](#citation) - [License](#license) - [Important Usage Notice](#-important-usage-notice) - [License Terms](#-license-terms) - [Paper](#paper) - [Contact](#contact) - [Acknowledgments](#acknowledgments) ## Dataset Description UNIDOC-BENCH is the first large-scale, realistic benchmark for multimodal retrieval-augmented generation (MM-RAG) and Visual Question Answering (VQA) built from 70,000 real-world PDF pages across eight domains. The dataset extracts and links evidence from text, tables, and figures, then generates 1,700+ multimodal QA pairs spanning factual retrieval, comparison, summarization, and logical reasoning queries. Each QA pair includes image paths (`mage_paths`) for the entire pdf for direct VQA evaluation, making it suitable for both MM-RAG and VQA research. ### Key Features - **70,000 real-world PDF pages** across 8 diverse domains - **1,700+ multimodal QA pairs** with expert validation - **Four evaluation paradigms**: text-only, image-only, multimodal text-image fusion, and multimodal joint retrieval - **Visual Question Answering (VQA) support**: Image paths (`longdoc_image_paths` and `gt_image_paths`) for direct VQA evaluation - **20% expert validation** ensuring high-quality annotations - **Comprehensive query types**: factual retrieval, comparison, summarization, and logical reasoning ### Dataset Structure The dataset is organized as a single split named `test` containing all 1,742 QA pairs. Each QA pair includes a `domain` field that indicates which of the 8 domains it belongs to. To access domain-specific data, filter the test split by the `domain` field. **Dataset splits:** - `test`: 1,742 QA pairs across all 8 domains **Available domains:** - `commerce_manufacturing` - `construction` - `crm` - `education` - `energy` - `finance` - `healthcare` - `legal` ## Domain Descriptions | Domain | PDF Count | Description | Download | |--------|-----------|-------------|----------| | **Healthcare** | 1,100 | Medical documents, research papers, clinical guidelines | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/healthcare_pdfs.tar.gz?download=true) | | **Legal** | 911 | Legal documents, court cases, regulations, contracts | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/legal_pdfs.tar.gz?download=true) | | **Education** | 812 | Educational materials, textbooks, academic papers | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/education_pdfs.tar.gz?download=true) | | **CRM** | 776 | Customer relationship management documents, manuals | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/crm_pdfs.tar.gz?download=true) | | **Energy** | 766 | Energy sector reports, technical specifications | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/energy_pdfs.tar.gz?download=true) | | **Construction** | 736 | Construction plans, safety manuals, technical docs | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/construction_pdfs.tar.gz?download=true) | | **Commerce & Manufacturing** | 719 | Business documents, manufacturing guides, catalogs | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/commerce_manufacturing_pdfs.tar.gz?download=true) | | **Finance** | 621 | Financial reports, banking documents, investment materials | [📥 Download](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/resolve/main/finance_pdfs.tar.gz?download=true) | ## QA Dataset Format The dataset contains 1,742 question-answer pairs in a single test split. Each QA pair has the following structure: ```json { "question": "Which procedures are part of Hunt Design's corrective measures process as defined in QMSP-1011?", "answer": "The procedures part of Hunt Design's corrective measures process as defined in QMSP-1011 include the analysis of data, internal audits, corrective actions, control of non-conforming products, and management review.", "gt_image_paths": [ "/path/to/images/commerce_manufacturing/0355944/0355944_page_0004.png" ], "question_type": "factual_retrieval", "answer_type": "image_only", "domain": "commerce_manufacturing", "longdoc_image_paths": [ "/path/to/images/commerce_manufacturing/0355944/0355944_page_0001.png", "/path/to/images/commerce_manufacturing/0355944/0355944_page_0002.png", "/path/to/images/commerce_manufacturing/0355944/0355944_page_0003.png", "/path/to/images/commerce_manufacturing/0355944/0355944_page_0004.png", "/path/to/images/commerce_manufacturing/0355944/0355944_page_0005.png" ] } ``` ### Field Descriptions - **`question`**: The question text (obscured version for evaluation) - **`answer`**: The complete answer to the question - **`gt_image_paths`**: **Ground-truth image paths** - List of image file paths that contain the answer. These are the exact pages/images needed to answer the question correctly. **Use this for Retrieval evaluation.** - **`longdoc_image_paths`**: **All image paths in this doc** - List of all image file paths from the document(s). These include all pages from the source document(s). **Use this for VQA evaluation.** - **`question_type`**: Type of question (factual_retrieval, comparison, summarization, logical_reasoning) - **`answer_type`**: Answer modality (text_only, image_only, table_required, multimodal) - **`domain`**: Domain category (commerce_manufacturing, construction, crm, education, energy, finance, healthcare, legal) ### Query Types - **Factual Retrieval**: Direct questions about specific information - **Comparison**: Questions requiring comparison between multiple sources - **Summarization**: Questions asking for summaries or overviews - **Logical Reasoning**: Questions requiring inference and reasoning ## Data Access ### 📊 Compressed File Sizes | Domain | File Size | PDF Count | |--------|-----------|-----------| | 🏥 Healthcare | 1.19 GB | 1,098 | | 🎓 Education | 816.9 MB | 812 | | 🏗️ Construction | 844.2 MB | 736 | | ⚡ Energy | 708.4 MB | 766 | | 👥 CRM | 704.7 MB | 776 | | ⚖️ Legal | 541.4 MB | 909 | | 🏭 Commerce & Manufacturing | 504.9 MB | 719 | | 💰 Finance | 360.8 MB | 621 | ## 🚀 Usage ### 📚 Loading the Dataset ```python from datasets import load_dataset # Load the dataset from Hugging Face # The dataset has 8 domain-based splits, each containing 200 QA pairs dataset = load_dataset("Salesforce/UniDoc-Bench") # Available splits: commerce_manufacturing, construction, crm, education, energy, finance, healthcare, legal print(dataset.keys()) # Access QA pairs by domain healthcare_qa = dataset["healthcare"] construction_qa = dataset["construction"] crm_qa = dataset["crm"] education_qa = dataset["education"] energy_qa = dataset["energy"] finance_qa = dataset["finance"] commerce_manufacturing_qa = dataset["commerce_manufacturing"] legal_qa = dataset["legal"] # Or load a specific domain directly healthcare_only = load_dataset("Salesforce/UniDoc-Bench", split="healthcare") ``` ### 📦 Working with PDF Files from Hugging Face The dataset includes compressed PDF files for each domain that can be used for RAG database construction: - [commerce_manufacturing_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/commerce_manufacturing_pdfs.tar.gz) (529 MB) - [construction_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/construction_pdfs.tar.gz) (885 MB) - [crm_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/crm_pdfs.tar.gz) (704 MB) - [education_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/education_pdfs.tar.gz) (816 MB) - [energy_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/energy_pdfs.tar.gz) (708 MB) - [finance_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/finance_pdfs.tar.gz) (360 MB) - [healthcare_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/healthcare_pdfs.tar.gz) (1.19 GB) - [legal_pdfs.tar.gz](https://huggingface.co/datasets/Salesforce/UniDoc-Bench/blob/main/legal_pdfs.tar.gz) (541 MB) ```python from datasets import load_dataset import tarfile import os from huggingface_hub import hf_hub_download # Method 1: Download specific PDF archives def download_pdf_archive(domain_name, save_path="."): """Download PDF archive for a specific domain""" filename = f"{domain_name}_pdfs.tar.gz" file_path = hf_hub_download( repo_id="Salesforce/UniDoc-Bench", filename=filename, repo_type="dataset", local_dir=save_path ) return file_path # Method 2: Extract PDFs for RAG database construction def extract_pdfs_for_rag(domain_name, extract_to="pdf_database"): """Download and extract PDFs for RAG database""" # Download the archive archive_path = download_pdf_archive(domain_name) # Create extraction directory domain_dir = os.path.join(extract_to, domain_name) os.makedirs(domain_dir, exist_ok=True) # Extract PDFs with tarfile.open(archive_path, 'r:gz') as tar: tar.extractall(domain_dir) print(f"Extracted {domain_name} PDFs to {domain_dir}") return domain_dir # Example: Extract healthcare PDFs for RAG healthcare_pdfs = extract_pdfs_for_rag("healthcare") ``` ### 🗄️ Building RAG Database from PDFs ```python import fitz # PyMuPDF from pathlib import Path import json from huggingface_hub import hf_hub_download import tarfile def build_rag_database(domain_name, output_dir="rag_database"): """Build RAG database from domain PDFs""" # Download and extract PDFs archive_path = hf_hub_download( repo_id="Salesforce/UniDoc-Bench", filename=f"{domain_name}_pdfs.tar.gz", repo_type="dataset" ) # Extract to temporary directory temp_dir = f"temp_{domain_name}_pdfs" with tarfile.open(archive_path, 'r:gz') as tar: tar.extractall(temp_dir) # Process PDFs for RAG rag_data = [] pdf_dir = Path(temp_dir) for pdf_file in pdf_dir.glob("**/*.pdf"): doc = fitz.open(pdf_file) for page_num in range(len(doc)): page = doc[page_num] text = page.get_text() images = page.get_images() # Create RAG entry entry = { "source": str(pdf_file), "page": page_num, "text": text, "images": len(images), "domain": domain_name } rag_data.append(entry) doc.close() # Save RAG database os.makedirs(output_dir, exist_ok=True) with open(f"{output_dir}/{domain_name}_rag.json", "w") as f: json.dump(rag_data, f, indent=2) print(f"Built RAG database for {domain_name}: {len(rag_data)} entries") return rag_data # Example: Build RAG database for healthcare healthcare_rag = build_rag_database("healthcare") ``` ### 📄 Working with PDF Documents ```python import fitz # PyMuPDF from pathlib import Path # Load a specific PDF pdf_path = "data/final_database/healthcare/medical_doc.pdf" doc = fitz.open(pdf_path) # Extract text and images for page_num in range(len(doc)): page = doc[page_num] text = page.get_text() images = page.get_images() # Process multimodal content for img_index, img in enumerate(images): xref = img[0] pix = fitz.Pixmap(doc, xref) # Process image... ``` ### ❓ Working with QA Pairs ```python from datasets import load_dataset # Load the dataset from Hugging Face dataset = load_dataset("Salesforce/UniDoc-Bench") # Access the dataset (contains all 1,742 QA pairs) data = dataset["test"] # Filter QA data for a specific domain healthcare_qa = data.filter(lambda x: x["domain"] == "healthcare") construction_qa = data.filter(lambda x: x["domain"] == "construction") # Filter by query type within a domain healthcare_factual = healthcare_qa.filter(lambda x: x["question_type"] == "factual_retrieval") # Filter by answer type within a domain healthcare_image_only = healthcare_qa.filter(lambda x: x["answer_type"] == "image_only") # Filter by multiple criteria healthcare_factual_image = data.filter( lambda x: x["domain"] == "healthcare" and x["question_type"] == "factual_retrieval" and x["answer_type"] == "image_only" ) ``` ### 🖼️ Visual Question Answering (VQA) UNIDOC-BENCH supports **Visual Question Answering (VQA)** evaluation, where models answer questions based on document images. The dataset provides image paths (`image_paths`) that contain the answer, enabling direct VQA evaluation. #### VQA Dataset Structure Each QA item includes: - **`longdoc_image_paths`**: All relevant document pages (for retrieval-based evaluation) - **`question`**: The question to answer - **`answer`**: The ground-truth answer #### VQA Usage Examples ```python from datasets import load_dataset from PIL import Image # Load the dataset dataset = load_dataset("Salesforce/UniDoc-Bench", split="test") # Get an example with all relevant images example = dataset[0] question = example["question"] all_images = example["longdoc_image_paths"] # All relevant document pages # Simulate retrieval: select top-k images top_k = 5 retrieved_images = all_images[:top_k] # Load retrieved images retrieved_image_list = [] for img_path in retrieved_images: image = Image.open(img_path) # Adjust path based on your setup retrieved_image_list.append(image) # Use with your VQA model # model_answer = vqa_model(question, retrieved_image_list) ``` ## Benchmark Results According to the [UNIDOC-BENCH paper](https://arxiv.org/abs/2510.03663): - **Multimodal text-image fusion RAG systems consistently outperform** both unimodal and jointly multimodal embedding-based retrieval - **Neither text nor images alone are sufficient** for optimal document understanding - **Current multimodal embeddings remain inadequate** for complex document-centric tasks ### SOTA Results We evaluate three state-of-the-art vision-language models: **GPT-5**, **Claude-4.5-Sonnet**, and **Gemini-2.5-Pro** across all domains. Results are reported for three retrieval settings: - **topk=10/20**: Retrieval-based evaluation with top-k retrieved pages using the best retrieval method in the paper (T+I) - **GT**: Ground-truth evaluation using the exact pages containing the answer #### Completeness by Domain | Domain | Gemini-2.5-Pro (topk=10) | Gemini-2.5-Pro (topk=20) | Gemini-2.5-Pro (GT) | Claude-4.5-Sonnet (topk=10) | Claude-4.5-Sonnet (topk=20) | Claude-4.5-Sonnet (GT) | GPT-5 (topk=10) | GPT-5 (topk=20) | GPT-5 (GT) | |--------|--------------------------|---------------------------|---------------------|----------------------------|-----------------------------|------------------------|-----------------|-----------------|------------| | Commerce Manufacturing | 0.6871 | 0.7130 | 0.7005 | **0.8040** | **0.7977** | **0.8246** | 0.7229 | 0.7369 | 0.7692 | | Construction | 0.6481 | 0.6380 | 0.6455 | **0.6977** | **0.7016** | **0.7284** | 0.6407 | 0.6367 | 0.6946 | | CRM | 0.6586 | 0.7012 | 0.7511 | **0.7570** | **0.7603** | **0.8157** | 0.6603 | 0.6671 | 0.7707 | | Education | 0.6703 | 0.7131 | 0.7135 | **0.7366** | **0.7817** | **0.8196** | 0.6369 | 0.6947 | 0.7651 | | Energy | 0.6573 | 0.6801 | 0.6332 | **0.8108** | **0.7777** | **0.7978** | 0.7017 | 0.7048 | 0.7098 | | Finance | 0.6738 | 0.6659 | 0.6722 | **0.7753** | **0.7550** | **0.7721** | 0.7079 | 0.7021 | 0.7212 | | Healthcare | 0.6884 | 0.6754 | 0.7494 | **0.7762** | **0.7525** | **0.8459** | 0.6699 | 0.6851 | 0.7582 | | Legal | 0.7299 | 0.7327 | 0.6949 | **0.7673** | **0.7489** | **0.7902** | 0.7049 | 0.7297 | 0.7554 | | **Average** | 0.6767 | 0.6899 | 0.6950 | **0.7656** | **0.7594** | **0.7993** | 0.6806 | 0.6946 | 0.7430 | **Key Findings:** - **Claude-4.5-Sonnet achieves the highest recall (completeness)** across all settings, with an average of **79.93%** (GT), **76.56%** (topk=10), and **75.94%** (topk=20) - **GPT-5** shows strong performance with an average of **74.30%** (GT), **68.06%** (topk=10), and **69.46%** (topk=20) - **Gemini-2.5-Pro** shows competitive performance with an average of **69.50%** (GT), **67.67%** (topk=10), and **68.99%** (topk=20) #### Completeness by Answer Type (GT Setting) - **Text Only**: Claude-4.5-Sonnet (**0.8940**) > GPT-5 (0.8380) > Gemini-2.5-Pro (0.7605) - **Image Only**: GPT-5 (**0.6815**) > Claude-4.5-Sonnet (0.6769) > Gemini-2.5-Pro (0.6332) - **Table Required**: Claude-4.5-Sonnet (**0.8057**) > GPT-5 (0.7530) > Gemini-2.5-Pro (0.7123) - **Image Plus Text**: Claude-4.5-Sonnet (**0.8096**) > GPT-5 (0.7002) > Gemini-2.5-Pro (0.6857) ## Citation If you use this dataset in your research, please cite: ```bibtex @article{peng2025unidoc, title={UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG}, author={Peng, Xiangyu and Qin, Can and Chen, Zeyuan and Xu, Ran and Xiong, Caiming and Wu, Chien-Sheng}, journal={arXiv preprint arXiv:2510.03663}, year={2025} } ``` ## License This dataset is released under the [Creative Commons Attribution-NonCommercial 4.0 International License (CC-BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/). ### ⚠️ Important Usage Notice **🚫 This dataset was generated using GPT-4.1 and should not be used to develop models that compete with OpenAI.** ### License Terms - **Attribution**: You must give appropriate credit, provide a link to the license, and indicate if changes were made. - **Non-Commercial**: You may not use the material for commercial purposes. - **No additional restrictions**: You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. For full license details, see: https://creativecommons.org/licenses/by-nc/4.0/ ## Paper [UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG](https://arxiv.org/abs/2510.03663) ## Contact For questions about this dataset, please open an issue on the [GitHub repository](https://github.com/salesforce/unidoc-bench) or contact the authors. ## Acknowledgments We thank the annotators and domain experts who contributed to the validation of this dataset. This work was supported by Salesforce Research.