---
title: SIS-Motion-54K-Dataset
canonical_url: "https://www.modelscope.cn/datasets/choucisan/SIS-Motion-54K-Dataset"
md_url: "https://www.modelscope.cn/datasets/choucisan/SIS-Motion-54K-Dataset.md"
repository: choucisan/SIS-Motion-54K-Dataset
last_updated: 2026-07-15
license: apache-2.0
storage_size: "78 GB"
downloads: 430074
stars: 2
---

# SIS-Motion-54K-Dataset

> SIS-Motion-54K-Dataset - choucisan 在 ModelScope 开源的数据集。SIS-Motion-54K AirScape video dataset.

choucisan/SIS-Motion-54K-Dataset 是 ModelScope 魔搭社区上的数据集，存储大小 78 GB，采用 apache-2.0 许可。

- **Repository**: choucisan/SIS-Motion-54K-Dataset
- **License**: apache-2.0
- **Storage size**: 78 GB
- **Downloads**: 430074
- **Stars**: 2
- **Last updated**: 2026-07-15

Source: https://www.modelscope.cn/datasets/choucisan/SIS-Motion-54K-Dataset

---

# SIS-Motion-54K

<p align="center">
  <img src="images/poster.jpeg" alt="SIS-Motion-54K" width="900"/>
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2607.12477"><img src="https://img.shields.io/badge/arXiv-2607.12477-b31b1b?style=for-the-badge&logo=arxiv" alt="arXiv"></a>
  <a href="https://choucisan.github.io/publications/self-in-space"><img src="https://img.shields.io/badge/Website-Project_Page-blue?style=for-the-badge" alt="Website"></a>
  <a href="https://github.com/IntelliSensing/Self-in-Space"><img src="https://img.shields.io/badge/GitHub-Self--in--Space-181717?style=for-the-badge&logo=github" alt="GitHub"></a>
  <a href="https://huggingface.co/datasets/choucsan/SIS-Motion-54K"><img src="https://img.shields.io/badge/%F0%9F%A4%97_HuggingFace-Dataset-yellow?style=for-the-badge" alt="Hugging Face"></a>
  <a href="https://www.modelscope.cn/datasets/choucisan/SIS-Motion-54K-Dataset"><img src="https://img.shields.io/badge/ModelScope-Dataset-5E4AF5?style=for-the-badge&logo=modelscope" alt="ModelScope"></a>
  <a href="https://choosealicense.com/licenses/apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-green?style=for-the-badge" alt="License"></a>
</p>

SIS-Motion-54K is a **motion-aware instruction-tuning dataset** built from the [AirScape](https://huggingface.co/datasets/EmbodiedCity/AirScape-Dataset) training split. It is designed to fine-tune the [SIS-Motion](https://huggingface.co/choucsan/SIS-Motion) model for joint understanding of *space* (environment) and *self* (agent motion) in embodied UAV scenarios.

> **Important:** This dataset is **strictly separated** from the [SIS-Bench](https://huggingface.co/datasets/choucisan/SIS-Bench) evaluation benchmark. It contains only perception and memory tasks — no reasoning-level data — so any generalization gains on downstream reasoning tasks can be attributed to improved spatial-temporal representations rather than task-specific memorization.

---

## Data Processing Pipeline

SIS-Motion-54K is constructed from the AirScape dataset through the same pipeline used for SIS-Bench (see the paper for details). The key steps are:

<p align="center">
  <img src="images/pipeline.webp" alt="Data Processing Pipeline" width="800"/>
</p>

### 1. Source Data

AirScape provides large-scale, diverse UAV video footage with rich meta-annotations including GPS coordinates, camera parameters, flight trajectories, and scene labels. We use the **training split only**, keeping the validation/test splits untouched for SIS-Bench construction.

### 2. Video Construction

We organize the raw footage into four video types to match different cognitive demands:
- **Single Video**: A self-contained clip for instantaneous perception evaluation
- **Concatenated Video**: 2–4 clips composed together to add cross-segment temporal dependency for memory tasks
- **Long Video**: Preserves long-horizon motion and scene evolution
- **Shuffled Video**: Permuted segments to require recovery of original temporal order

### 3. Task Annotation

Each video is annotated with one or more QA pairs based on the task design taxonomy. The annotation pipeline combines rule-based extraction from AirScape meta-annotations with LLM-assisted polishing to produce natural, fluent question-answer pairs. All MCQ distractors are automatically generated through strategies including direction reversal, attribute substitution, and spatial rearrangement, then verified for unambiguous correctness.

---

## Dataset Composition

SIS-Motion-54K contains **54,298** entries from **11,763** unique AirScape videos, organized into three complementary formats:

<p align="center">
  <img src="images/data.jpeg" alt="Task Distribution" width="700"/>
</p>

### Format 1: Multiple-Choice QA (23,406 entries)

Covering 9 perception and memory tasks, this format directly aligns with the SIS-Bench evaluation protocol:

| Task | Category | Entries | Description |
|---|---|---|---|
| `object_existence` | Spatial Cognition — Perception | 1,238 | Identify whether a given object appears in the video |
| `object_attribute` | Spatial Cognition — Perception | 1,238 | Recognize object properties (color, shape, material) |
| `relative_direction` | Spatial Cognition — Perception | 1,238 | Determine spatial relations between objects |
| `landmark_appearance_order` | Spatial Cognition — Memory | 1,200 | Recall the temporal order of landmark appearances |
| `landmark_recall` | Spatial Cognition — Memory | 1,200 | Remember which landmark appeared before/after a given event |
| `positional_relationship` | Spatial Cognition — Memory | 1,139 | Identify relative positions across multiple clips |
| `mc_qa` | Self-Awareness | 16,153 | Recognize the drone's own flight actions (action recognition) |
| `open_qa` | Self-Awareness | 16,153 | Free-form action descriptions for generative training |

### Format 2: Open-Ended QA (16,153 entries)

Action-related free-form supervision that encourages the model to explicitly generate motion-aware content rather than merely selecting from options:

- *"What flight action is the drone performing in the second clip?"*
- *"Describe the drone's motion intention shown in the video."*

These prompts strengthen embodied self-awareness by forcing the model to articulate agent dynamics in natural language.

### Format 3: Description (14,739 entries)

Continuous supervision for jointly grounding agent dynamics and scene semantics:

| Sub-type | Entries | Example Prompt |
|---|---|---|
| `motion_intention` | 11,763 | *"Describe the drone's motion intention shown in the video."* |
| `static_description` | 1,488 | *"Describe the road layout, surrounding vegetation, and terrain features visible in this aerial footage."* |
| `dynamic_description` | 1,488 | *"What dynamic activities involving vehicles can be observed in the third clip?"* |

Motion intention descriptions provide dense, frame-aligned supervision for how the drone moves, while scene descriptions ground the visual context.

### Task Distribution

| Category | Entries | QA Format |
|---|---|---|
| Self-Awareness (action recognition, motion description) | 44,079 | MCQ + Open |
| Spatial Cognition — Perception | 3,714 | MCQ |
| Spatial Cognition — Memory (landmark, position, order) | 3,539 | MCQ |
| Scene Description (static + dynamic) | 2,976 | Open |
| **Total** | **54,298** | |

---

## Data Format

Each line in `SIS-Motion-54K.jsonl` is a JSON object. Fields vary slightly by task type and QA format.

### Common Fields

| Field | Type | Description |
|---|---:|---|
| `id` | string | Unique entry identifier |
| `video_name` | string | Source AirScape video name |
| `video_path` | string | Relative path to the video file |
| `task_type` | string | One of 11 task categories (see table above) |
| `qa_type` | string | `mc` for multiple-choice or `open` for open-ended |
| `question` | string | Natural-language question |
| `answer` | string | Ground-truth answer (option label for MCQ, free text for open-ended) |
| `options` | object | (MCQ only) Four options `A`–`D` |

### MCQ Example

```json
{
  "id": "object_existence_AirScape_Train_4392",
  "video_name": "AirScape_Train_4392",
  "video_path": "AirScape_dataset/AirScape_Train_4392.mp4",
  "task_type": "object_existence",
  "qa_type": "mc",
  "question": "Which of the following objects does not appear in the video?",
  "options": {
    "A": "A small rectangular structure with a dark roof on the left side",
    "B": "A white sedan traveling on the asphalt road",
    "C": "A red vehicle visible in the distance",
    "D": "A yellow car driving toward the camera"
  },
  "answer": "D"
}
```

### Open-Ended QA Example

```json
{
  "id": "10190_motion",
  "video_name": "AirScape_Train_5915",
  "video_path": "AirScape_dataset/AirScape_Train_5915.mp4",
  "task_type": "motion_intention",
  "qa_type": "open",
  "question": "Describe the drone's motion intention shown in the video.",
  "answer": "While the drone was flying forward, it gradually rotated to the left, tracking the white car as it moved along the road. At the same time, the camera angle remained stable, keeping the car at the center of the field of view."
}
```

---

## Relationship to SIS-Motion and SIS-Bench

```
AirScape (training split)
    │
    ├──> SIS-Motion-54K (this dataset) ──> SFT ──> SIS-Motion model
    │        • 54,298 entries
    │        • Perception + Memory only
    │        • MCQ + Open QA + Description
    │
    └──> SIS-Bench (held-out split)
             • 13 tasks, 4,856 questions
             • Perception + Memory + Reasoning
             • Evaluation only — never used in training
```

**Key design principle:** SIS-Motion-54K is constructed exclusively from the AirScape training split. The SIS-Bench evaluation benchmark uses AirScape validation/test data plus other sources (UrbanVideo-Bench, VisDrone, OpenUAV) with no overlap. This strict separation ensures that evaluation results reflect genuine generalization rather than data leakage.

---

## Quick Start

### Download

The dataset is distributed across two platforms. The video files are large — we host the annotation file on HuggingFace and the complete dataset (including videos) on ModelScope.

**Step 1 — Download the annotation file (`SIS-Motion-54K.jsonl`) from HuggingFace:**

```bash
pip install datasets

from datasets import load_dataset
dataset = load_dataset("choucsan/SIS-Motion-54K")
print(dataset["train"][0])
```

Or via CLI:

```bash
huggingface-cli download choucsan/SIS-Motion-54K \
  --repo-type dataset \
  --local-dir SIS-Motion-54K
```

**Step 2 — Download the video files (`AirScape_dataset/`) from ModelScope:**

```bash
pip install modelscope

modelscope download --dataset choucisan/SIS-Motion-54K-Dataset \
  --local-dir SIS-Motion-54K \
  --include "AirScape_dataset/**"
```

After both steps, your local directory contains the JSONL annotation file alongside the `AirScape_dataset/` video folder. The `video_path` field in each JSONL record points to the correct relative path.

> **Why two platforms?** The video files total hundreds of GB. ModelScope supports larger file storage for datasets, so the full package is hosted there at [choucisan/SIS-Motion-54K-Dataset](https://modelscope.cn/datasets/choucisan/SIS-Motion-54K-Dataset). If you only need the annotations, HuggingFace is sufficient.

### Load and Inspect

```python
import json

with open("SIS-Motion-54K.jsonl", "r") as f:
    entries = [json.loads(line) for line in f]

print(f"Total entries: {len(entries)}")

# Filter by task type
mcq = [e for e in entries if e["qa_type"] == "mc"]
open_qa = [e for e in entries if e["qa_type"] == "open"]
print(f"MCQ: {len(mcq)}, Open-ended: {len(open_qa)}")

# Inspect a sample
sample = entries[0]
print(f"Task: {sample['task_type']}")
print(f"Q: {sample['question']}")
if sample["qa_type"] == "mc":
    for k, v in sample["options"].items():
        marker = " ✓" if k == sample["answer"] else ""
        print(f"  {k}: {v[:60]}...{marker}")
else:
    print(f"A: {sample['answer'][:100]}...")
```

---

## Usage with SIS-Motion

This dataset is the **official training set** for the [SIS-Motion](https://huggingface.co/choucsan/SIS-Motion) model. To reproduce training:

```bash
git clone https://github.com/IntelliSensing/Self-in-Space.git
cd Self-in-Space

# Set up the training environment
bash scripts/setup_conda.sh motion
conda activate sis-motion-motion

# Launch SFT with SIS-Motion-54K
# (refer to the repository for full training scripts)
```

The training uses Qwen2.5-VL as the base model with LoRA fine-tuning (`r=32`, `alpha=64`) on the LLM backbone, plus training the motion-visual connector from scratch. Only the perception and memory entries are used — reasoning tasks are intentionally excluded to test for generalization.

---

## Citation

If you use SIS-Motion-54K in your research, please cite:

```bibtex
@misc{zou2026sis,
      title={Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence},
      author={Zhishan Zou and Guoyan Sun and Zhiwei Wei and Jiancheng Pan and Yujie Li and Mugen Peng and Wenjia Xu},
      year={2026},
      eprint={2607.12477},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.12477},
}
```

---

## Contact

For questions, corrections, or collaboration requests:

[choucisan@gmail.com](mailto:choucisan@gmail.com)