--- title: UAV-DualCog canonical_url: "https://www.modelscope.cn/datasets/Lozumi/UAV-DualCog" md_url: "https://www.modelscope.cn/datasets/Lozumi/UAV-DualCog.md" repository: Lozumi/UAV-DualCog last_updated: 2026-08-28 license: CC-BY-4.0 storage_size: "70 GB" domain: - video - image tasks: - video-question-answering - visual-question-answering encoding: - H264 - jpeg resolutions: - "1440 x 1080" - "4096 x 3072" color_space: - rgb downloads: 728247 stars: 2 --- # UAV-DualCog > UAV-DualCog - Lozumi 在 ModelScope 开源的数据集。--> Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs --> Lozumi/UAV-DualCog 是 ModelScope 魔搭社区上的video-question-answering、visual-question-answering数据集，涉及 video、image 领域，存储大小 70 GB，采用 CC-BY-4.0 许可。 - **Repository**: Lozumi/UAV-DualCog - **License**: CC-BY-4.0 - **Tasks**: video-question-answering, visual-question-answering - **Domain**: video, image - **Storage size**: 70 GB - **Downloads**: 728247 - **Stars**: 2 - **Last updated**: 2026-08-28 Source: https://www.modelscope.cn/datasets/Lozumi/UAV-DualCog ---

Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs

_**[Like Liu](https://likeliu.com), Zhengzheng Xu, Haitao He, Hongzhe Li, Shuchang Zhang, [Dian Shao](https://scholar.google.com/citations?user=amxDSLoAAAAJ&hl=en)†**_

†Corresponding Author
Northwestern Polytechnical University
If you like our project, please give us a star ⭐ on GitHub for latest update.
     
arXiv 2026
# UAV-DualCog Dataset Repository Guide Last updated: 2026-08-11 This is the official dataset repository guide for **UAV-DualCog**. ## 1. What UAV-DualCog Is UAV-DualCog is a drone-centric multimodal reasoning benchmark for **dual cognition**: self-aware reasoning and environment-aware reasoning under aerial observation. The release targets two complementary goals: - benchmark evaluation for multimodal foundation models, - reusable structured assets for downstream dataset users. The benchmark is organized around one primary capability axis and one observation axis: - dual cognition: - self-aware reasoning, - environment-aware reasoning; - media: - image tasks, - video tasks. The key point is that **dual cognition** is the capability being evaluated, while **image and video** are the media used to expose that capability. This design yields a benchmark that does not only test answer selection, but also tests whether a model can align its reasoning with spatial evidence or temporal evidence. ## 1.1 Quick Start Recommended entry points: 1. Read this dataset card to understand the release scope and file contracts. 2. Use the benchmark website to inspect task definitions, examples, and leaderboard views: - https://uav-dualcog.lozumi.com/ 3. Use the official code repository for loading, preprocessing, and evaluation: - https://github.com/SmartDianLab/UAV-DualCog 4. Use the AerialVLN simulator package when reproducing simulator-backed collection or rendering: - https://www.kaggle.com/datasets/shuboliu/aerialvln-simulators For detailed benchmark definitions, construction details, and usage instructions, the benchmark website should be treated as the primary external reference. ## 2. Benchmark Scope Current core release: - 12 released benchmark scenes, - 512 validated landmarks, - 4096 image QA samples, - 2048 video QA samples, - 4 image task families, - 2 video task families. All currently released benchmark task files are test-only. The repository does not currently expose public `train` or `validation` splits for task evaluation. The released 12-scene benchmark subset is drawn from a larger reviewed scene pool. In the public repository, the benchmark task layer and the scene asset layer do not have identical scope: - `task_data` currently corresponds to the 12-scene benchmark release; - `scene_data` covers the full set of 18 reviewed scenes that have public geometry and landmark-review assets. This means the repository exposes a broader scene asset pool than the current benchmark task split. Scene-level geometric assets and reviewed landmark assets are provided so that users can inspect the benchmark context rather than treating the task files as opaque black boxes. For clarity: - `scene_data` is a supporting public asset release rather than a training split; - `task_data` is a benchmark evaluation release and should be treated as test data. ## 3. Capability Definition ### 3.1 Self-aware reasoning Self-aware reasoning evaluates whether a UAV agent can reason about itself: - where it is relative to a landmark, - what it will observe after a described motion, - what behavior it is executing, - when that behavior occurs. ### 3.2 Environment-aware reasoning Environment-aware reasoning evaluates whether a UAV agent can reason about the external world from its current motion context: - where the target landmark is relative to the UAV, - which action is appropriate given the landmark-relative situation, - how many times a landmark becomes visible in a mission, - during which time intervals the landmark is visible. ### 3.3 Evidence-aware evaluation UAV-DualCog explicitly separates: - semantic correctness, - evidence grounding. For image tasks, a model is evaluated on both: - selecting the correct answer option, - localizing the landmark with a normalized bounding box. For video tasks, a model is evaluated on both: - predicting the correct semantic answer, - localizing the relevant time interval(s). This is one of the core benchmark design principles: answer-only success is not sufficient if the supporting spatial or temporal evidence is incorrect. ## 4. Task Families ### 4.1 Image tasks (Stage 4) The image branch contains four task families. Each released landmark contributes both `4way` and `8way` difficulty variants. 1. `self_where` - Canonical display name: `Landmark-Relative Position Reasoning` - Cognition: self-aware - Input: one landmark-centric reference image plus one egocentric query observation - Output: one answer option and one landmark bounding box on the query image - Core question: where is the UAV relative to the landmark 2. `self_what` - Canonical display name: `Future Observation Prediction` - Cognition: self-aware - Input: one reference image plus a future-view multiple-choice set - Output: one answer option - Core question: which future observation matches the described motion outcome 3. `env_where` - Canonical display name: `Self-Relative Position Reasoning` - Cognition: environment-aware - Input: one current egocentric observation - Output: one answer option and one landmark bounding box on the query image - Core question: where is the landmark relative to the UAV 4. `env_how` - Canonical display name: `Landmark-Driven Action Decision` - Cognition: environment-aware - Input: one current egocentric observation - Output: one answer option and one landmark bounding box on the query image - Core question: what action decision is appropriate under the current landmark-relative situation ### 4.2 Video tasks (Stage 3) The video branch contains two task families. 1. `self_instance_recognition_joint` - Canonical display name: `Flight Behavior Recognition and Temporal Localization` - Cognition: self-aware - Input: task video plus mission-conditioned context - Output: behavior option(s) and temporal interval(s) - Public reporting also derives: - composite-level semantic accuracy, - atomic-level semantic accuracy, - temporal localization quality. 2. `env_visibility_reasoning` - Canonical display name: `Landmark Visibility Counting and Interval Reasoning` - Cognition: environment-aware - Input: task video plus target landmark reference - Output: visibility count and visible time interval(s) ### 4.3 Task summary table | Task ID | Display name | Modality | Cognition | Main input | Main output | | --- | --- | --- | --- | --- | --- | | `self_where` | Landmark-Relative Position Reasoning | image | self-aware | reference image + query observation | option + bbox | | `self_what` | Future Observation Prediction | image | self-aware | reference image + future-view options | option | | `env_where` | Self-Relative Position Reasoning | image | environment-aware | query observation | option + bbox | | `env_how` | Landmark-Driven Action Decision | image | environment-aware | query observation | option + bbox | | `self_instance_recognition_joint` | Flight Behavior Recognition and Temporal Localization | video | self-aware | task video + mission context | option(s) + interval(s) | | `env_visibility_reasoning` | Landmark Visibility Counting and Interval Reasoning | video | environment-aware | task video + landmark context | count + interval(s) | ## 5. Evaluation Objects and Metrics ### 5.1 Image tasks Image-task prediction objects contain: - `answer_option_id` - optionally `bbox_xyxy_norm` Main metrics include: - option accuracy, - `BBox Acc@50IoU`, - mean IoU. ### 5.2 Video tasks Video-task prediction objects contain: - answer option(s) or behavior label(s), - interval(s) in seconds, - for visibility tasks, visible count. Main metrics include: - semantic correctness, - temporal IoU or interval agreement, - count accuracy for visibility reasoning. The public leaderboard may present aggregated summary views for readability, but the underlying task manifests and experiment outputs retain the task-level prediction structure. ## 6. Repository Scope and Boundary The public repository is the release-facing layer of the dataset. It includes: - scene-level geometry and reviewed landmarks, - released benchmark task assets, - released manifests and render requests, - benchmark-ready media references. The scope is asymmetric by design: - `scene_data` contains the complete 18-scene reviewed scene release; - `task_data` currently contains the 12-scene benchmark task release. The released task layer is also split-asymmetric in another sense: - the repository currently provides public benchmark test data only; - it does not provide public train or validation task splits. It intentionally excludes many internal generation-time artifacts, including: - internal logs, - temporary caches, - internal experiment workspaces, - internal review-only intermediate files not needed for public reproduction. ## 7. Top-Level Layout The public repository is conceptually split into two release layers. ```text scene_data/ airsim_env_*/ pcd_map/ landmarks_raw/ landmarks_review/ task_data/ airsim_env_*/ image_tasks/ assets/ manifests/ render_requests/ selections/ video_tasks/ missions/ datasets/ selections/ ``` ### 7.1 `scene_data` This layer stores scene-level assets and landmark review outputs. Important release note: - `scene_data` is not restricted to the 12 benchmark test scenes. - The current public release contains all 18 reviewed scenes with available scene geometry and landmark-review outputs. - `pcd_map/` - fused point-cloud assets and geometry support files. - `landmarks_raw/` - pre-review landmark candidate outputs. - `landmarks_review/` - reviewed landmark instances and downstream-consumable landmark metadata. ### 7.2 `task_data` This layer stores benchmark task artifacts. - `image_tasks/` - Stage 4 image QA assets, manifests, and render requests. - `video_tasks/` - Stage 3 mission-level task videos, final-task metadata, and released manifests. ## 8. Data Contracts The following files are the main public contracts that downstream users should treat as stable interfaces. ### 8.1 Scene review contract `scene_data//landmarks_review/.valid_instances.json` This is the reviewed landmark handoff file used by later stages. It provides: - stable landmark instance ids, - reviewed category/subcategory/description fields, - reference RGB view assets, - geometry and instance context needed for task generation. ### 8.2 Image-task manifest contract `task_data//image_tasks/manifests/.latest_manifest.json` Top-level fields include: - generation metadata, - scene id and engine, - released task types and difficulty sets, - `samples`. Each sample contains fields such as: - `sample_id` - `landmark_id` - `task_family` - `task_group` - `difficulty` - `reference_image` - `reference_image_with_bbox` - `reference_bbox_xyxy_norm` - `target_image` - `answer_bbox_xyxy_norm` - `task_type` - `label_options` - `answer_option_id` - `prompt_text` - `user_prompt` - `system_prompt` This contract is sufficient for benchmark inference on image tasks. Representative sample shape: ```json { "sample_id": "env_7_20_120_self_shared_4way_000001_where", "task_type": "self_where", "task_group": "self-aware", "difficulty": "4way", "landmark_id": "20_120", "reference_image_with_bbox": "task_data/airsim_env_7/image_tasks/assets/reference_bbox/20_120/....jpg", "target_image": "scene_data/airsim_env_7/landmarks_raw/rgb_views/20_120/....jpg", "label_options": [ {"option_id": "A", "label": "..."}, {"option_id": "B", "label": "..."} ], "answer_option_id": "D", "answer_bbox_xyxy_norm": [0.31, 0.27, 0.58, 0.76] } ``` ### 8.3 Video-task manifest contract `task_data//video_tasks/datasets/.latest_manifest.json` Top-level fields include: - generation metadata, - scene id and engine, - released forms, - task-group flags, - `samples`, - manifest-level `summary`. Each sample contains fields such as: - `sample_id` - `form` - `task_group` - `task_name` - `task_display_name` - `mission_id` - `mission_family` - `landmark_id` - `reference_image_with_bbox` - `overview_image` - `keyframe_board_image` - `video_path` - `video_web_path` - `fps` - `frame_count` - `flight_description` - `visible_count` - `visible_intervals_sec` - `difficulty_band` - `choice_options` - `answer_option_ids` - `answer_items` This contract is the benchmark-facing video task interface. Representative sample shape: ```json { "sample_id": "env_7_batch_env_7_10_55_atomic_0075_self_instance_recognition_joint_000001", "form": "self_instance_recognition_joint", "task_group": "self-state", "mission_id": "batch_env_7_10_55_atomic_0075", "landmark_id": "10_55", "reference_image_with_bbox": "task_data/airsim_env_7/video_tasks/cache/assets/reference_bbox/10_55/....jpg", "video_path": "task_data/airsim_env_7/video_tasks/missions/.../final_task/task_rgb.mp4", "video_web_path": "task_data/airsim_env_7/video_tasks/missions/.../final_task/task_rgb_web.mp4", "fps": 5, "frame_count": 157, "visible_count": 1, "visible_intervals_sec": [{"start_sec": 0.0, "end_sec": 2.7}], "difficulty_band": "easy", "choice_options": [ {"option_id": "A", "label": "..."}, {"option_id": "B", "label": "..."} ], "answer_option_ids": ["C"], "answer_items": [ {"option_id": "C", "label": "...", "intervals_sec": [{"start_sec": 1.2, "end_sec": 6.8}]} ] } ``` ### 8.4 Mission-level Stage 3 contract `task_data//video_tasks/missions//final_task/task_data.json` This file is the mission-level ground-truth contract behind Stage 3 tasks. It contains: - `video` - media paths, - frame manifests, - fps, - frame counts, - video dimensions, - capture dimensions; - `target_presence` - frame-level or interval-level target presence information; - `task_tracks` - task-specific supervision for: - `environmental_awareness`, - `self_state_awareness`. This file is the correct entry point when a user needs mission-level temporal supervision rather than only released sample-level manifests. In practice: - use `video_tasks/datasets/.latest_manifest.json` for benchmark inference and leaderboard-style evaluation; - use `missions//final_task/task_data.json` when mission-level temporal supervision or frame-level inspection is needed. ## 9. Media and Path Semantics Image and video paths stored in manifests are release-facing references, not arbitrary internal cache paths. For Stage 4: - `reference_image_with_bbox` points to the released reference image with GT bbox overlay, - `target_image` points to the released query observation. Depending on task subtype and release path, Stage 4 media may point either to: - released task assets under `task_data/.../image_tasks/assets/...`, or - scene-level source views under `scene_data/.../landmarks_raw/rgb_views/...`. For Stage 3: - `video_path` points to the released main task video, - `video_web_path` points to a web-playable derivative when available, - `reference_image_with_bbox`, `overview_image`, and `keyframe_board_image` provide auxiliary evidence views, - `task_data.json -> video.frames_manifest` and `frame_index_map` support frame-level inspection. If `video_web_path` is empty for a given sample, downstream users should fall back to `video_path`. ## 10. Usage and Reproduction Pointers This repository guide intentionally focuses on **release scope and data contracts**. To avoid divergence and duplicated maintenance, detailed operational steps (environment setup, stage-by-stage commands, benchmark execution, and evaluation scripts) are not repeated here. Please use the following as the canonical operational references: - Benchmark website (recommended reading order and Usage page): - https://uav-dualcog.lozumi.com/ - https://uav-dualcog.lozumi.com/usage/ - Official code repository (latest runnable commands and config templates): - https://github.com/SmartDianLab/UAV-DualCog - Simulator package used by construction-stage reproduction: - https://www.kaggle.com/datasets/shuboliu/aerialvln-simulators Practical split for external users: - Use **this dataset guide** for file contracts, task semantics, and manifest field definitions. - Use **website + GitHub** for concrete execution instructions and reproducibility workflows. ## 11. Benchmark Provenance The public release is produced by the four-stage UAV-DualCog construction pipeline: - Stage 1: scene point-cloud collection and fusion, - Stage 2: landmark mining, review, and structured annotation, - Stage 3: behavior-driven mission generation and video task construction, - Stage 4: landmark-centered image QA generation. The benchmark website provides: - task explanations, - prompt templates, - examples, - leaderboard views, - analysis pages. Official benchmark site: - https://uav-dualcog.lozumi.com/ Official code repository: - https://github.com/SmartDianLab/UAV-DualCog ## 12. Practical Notes for External Users - Field names should be consumed in their canonical JSON form. - Task ids such as `self_where` or `env_visibility_reasoning` should be treated as stable benchmark identifiers. - Display names on the website are reader-facing aliases; manifests retain machine-facing ids. - Some repository paths may differ slightly across mirrors or release bundles. The canonical structure is the contract described in this guide. - For actual loading and benchmark evaluation, prefer the official GitHub implementation instead of reimplementing parsers from scratch: - https://github.com/SmartDianLab/UAV-DualCog - For detailed benchmark definitions, construction explanations, and usage walkthroughs, prefer the public benchmark website: - https://uav-dualcog.lozumi.com/ - For simulator-backed reproduction, use the released AerialVLN simulator package: - https://www.kaggle.com/datasets/shuboliu/aerialvln-simulators ## 13. Citation and License - License: follow the repository card and platform metadata for the active release. - Citation: cite the UAV-DualCog dataset release and record the repository version/date used in evaluation. - Benchmark-facing supplementary explanations are maintained at: - https://uav-dualcog.lozumi.com/ - Official loading and evaluation code is maintained at: - https://github.com/SmartDianLab/UAV-DualCog - Simulator dependency for reproduction is maintained at: - https://www.kaggle.com/datasets/shuboliu/aerialvln-simulators