|
|
|
Ultra-High-Resolution Remote Sensing Understanding and Reasoning
🤗 Hugging Face · 🤖 ModelScope · 📄 Paper · 💻 Code
--- ## 📚 Introduction **XHRBench** evaluates fine-grained perception and complex reasoning in multimodal large language models using ultra-high-resolution remote-sensing imagery. This repository retains the name **XHRBench** and belongs to the same RSHR benchmark project as [RSHR-Bench](https://huggingface.co/datasets/RL-MIND/RSHR-Bench), with a different file layout. The full benchmark described in the paper contains **5,329 full-scene images**, with a long side of at least **4,000 pixels** and up to approximately **3 × 10⁸ pixels (300 MP)** per image. This repository's `dataset.json` contains **1,932 VQA examples**; the package's example count is distinct from the full benchmark's image count. ## 🎯 Dataset at a Glance | Item | Details | | --- | --- | | Current JSON release | 1,932 VQA examples | | Task categories | 17 original category identifiers | | Images per example | 1 image: 1,862; 2 images: 50; 3 images: 20 | | Data language | English | | Main files | `dataset.json`, `images/`, `data/train-00000-of-00001.parquet` | | Full benchmark scope | 5,329 full-scene images; 9 perception categories and 4 reasoning types | The benchmark draws on **DOTA-v2.0, MiniFrance, FAIR1M, HRSCD, XLRS-Bench**, and the team's **100 MP UAV imagery**. Its two-stage **Human–LLM Adversarial Verification** pipeline combines LLM adversarial filtering with human review to reduce questions answerable from language priors alone. ## 🌟 Task Design | Task family | Evaluation focus | | --- | --- | | Multiple-Choice VQA (MCQ) | Visual decisions within a fixed answer space, including single-turn and multi-turn dialogues | | Open-Ended VQA (OEQ) | Free-form visual understanding and compositional reasoning | | Image Captioning (IC) | Global scene summaries and regional descriptions | | Single-Image Evaluation (SIE) | Multi-scale perception and reasoning within an ultra-high-resolution image | These four families describe the full evaluation design in the paper. The `dataset.json` in this repository contains VQA annotations with answer options. See the [paper](https://arxiv.org/abs/2512.17319) and [code](https://github.com/Yunkaidang/RSHR) for the complete evaluation protocol. ## 📦 Data Format ```text dataset.json # List of VQA annotations images/ # Original images referenced by annotations data/train-00000-of-00001.parquet # Repository-provided Parquet file assets/ # Dataset-card figures and logos ``` | Field | Description | | --- | --- | | `id` | Sample identifier | | `images` | List of relative image paths | | `question` | Question text | | `options` | Mapping from option letters to option text | | `answer` | Gold answer option | | `category` | Original task category identifier | Image paths are relative to the repository root. Category identifiers retain their original spelling. The `train` component in the Parquet filename is a packaging label; it does not by itself define the paper's training/test protocol. ### Category distribution | Category (original identifier) | Examples | | --- | ---: | | `anomaly_detection` | 50 | | `color` | 200 | | `detection` | 100 | | `future_prediction_two_image` | 50 | | `multi_region_join_contrast` | 20 | | `multi_region_join_contrast_singal` | 50 | | `multitrun-future_prediction` | 150 | | `multiturn_object_state_judgement` | 132 | | `multturn_anomaly_detection` | 60 | | `object_clasfication` | 200 | | `object_counting` | 200 | | `object_grounding` | 200 | | `object_state_judgement` | 50 | | `regional_grouding` | 70 | | `regionla_counting` | 100 | | `relation` | 200 | | `shape` | 100 | | **Total** | **1,932** | ## 🚀 Quick Start Inspect the annotations first, then download the images you need. ```python import json from pathlib import Path from modelscope.hub.snapshot_download import dataset_snapshot_download local_dir = Path(dataset_snapshot_download( dataset_id="KAIWANG/XHRBench", revision="master", allow_file_pattern=["dataset.json"], )) with (local_dir / "dataset.json").open(encoding="utf-8") as f: samples = json.load(f) sample = samples[0] print(sample["category"], sample["question"], sample["options"]) # Download only the image needed for this sample. image_dir = Path(dataset_snapshot_download( dataset_id="KAIWANG/XHRBench", revision="master", allow_file_pattern=[sample["images"][0]], )) image_path = image_dir / sample["images"][0] ``` ## 📊 Evaluation and Visualizations The original project reports evaluations of **14 models**, including GPT-4o, Gemini 1.5 Pro, Qwen2.5-VL, and remote-sensing models such as GeoChat and VHM, across multiple-choice VQA, open-ended VQA, and image captioning. The original overview, leaderboard, and task-distribution figures are retained below; consult the paper for model versions and evaluation settings.  ### Evaluation figures 
