--- title: SIS-Motion canonical_url: "https://www.modelscope.cn/models/choucisan/SIS-Motion" md_url: "https://www.modelscope.cn/models/choucisan/SIS-Motion.md" repository: choucisan/SIS-Motion last_updated: 2026-07-15 license: apache-2.0 base_model: - Qwen/Qwen2.5-VL-7B-Instruct base_model_relation: adapter parameters: 20.2M tensor_type: - BF16 library_name: - peft - lora - safetensors downloads: 7 stars: 2 tags: - peft - lora - vision-language - video - optical-flow - uav - drone - motion - embodied-ai - spatial-intelligence - vlm - vla - world-model --- # SIS-Motion > SIS-Motion - choucisan 在 ModelScope 开源的模型。SIS-Motion is a motion-aware extension of a standard video MLLM, designed to investigate whether explicitly modeling self-related dynamics can improve the joint understanding of space and self in embodied UAV… choucisan/SIS-Motion 是 ModelScope 魔搭社区上的 20.2M 参数机器学习模型，采用 apache-2.0 许可，基于 Qwen/Qwen2.5-VL-7B-Instruct 构建。 - **Repository**: choucisan/SIS-Motion - **License**: apache-2.0 - **Parameters**: 20.2M - **Base model**: Qwen/Qwen2.5-VL-7B-Instruct - **Tags**: peft, lora, vision-language, video, optical-flow, uav, drone, motion, embodied-ai, spatial-intelligence, vlm, vla, world-model - **Downloads**: 7 - **Stars**: 2 - **Last updated**: 2026-07-15 Source: https://www.modelscope.cn/models/choucisan/SIS-Motion --- # SIS-Motion

SIS-Motion

arXiv Website GitHub Hugging Face ModelScope License

SIS-Motion is a motion-aware extension of a standard video MLLM, designed to investigate whether explicitly modeling self-related dynamics can improve the joint understanding of *space* and *self* in embodied UAV scenarios. > **Important:** SIS-Motion is **NOT a state-of-the-art model.** It is a **controlled experiment** where the only difference from the visual-only backbone is the addition of a frozen optical-flow estimator, a motion-feature branch, and a lightweight fusion connector. All other settings — base model, training data, LoRA rank, learning rate, and optimization — are kept identical. This ensures that any observed gain can be cleanly attributed to motion-aware integration rather than differences in scale or recipe. --- ## Architecture SIS-Motion extends Qwen2.5-VL with a motion-aware branch. Video frames are processed in parallel through a shared vision encoder — once for appearance and once for optical-flow-derived motion features — then fused via a lightweight additive connector before the language model.

SIS-Motion Framework

The forward path is: ```text video frames ├── Qwen2.5-VL vision encoder (frozen) ───────────────> visual tokens └── VideoFlow MOFNet (frozen, FP32) └── forward/backward flow [T, 4, H/8, W/8] └── forward flow (dx, dy) └── pseudo-images [magnitude, dx, dy] └── Qwen2.5-VL vision encoder (shared, frozen) -> motion tokens motion tokens -> LayerNorm -> Linear(3584, 2048) -> GELU -> Linear(2048, 3584) visual tokens + projected motion tokens -> Qwen2.5-VL language model with LoRA ``` ### 1. Appearance branch The standard Qwen2.5-VL video processor and frozen vision encoder produce the appearance tokens used by the base model. ### 2. Motion branch VideoFlow MOFNet receives at least three RGB frames, resizes them to `320x480`, and predicts forward and backward optical flow at `1/8` resolution. SIS-Motion uses the forward `(dx, dy)` channels, clamps invalid/extreme values, and builds a three-channel pseudo-image `[magnitude, dx, dy]`. These pseudo-images are patchified with the Qwen2.5-VL video grid and passed through the same frozen Qwen vision encoder to produce token-aligned motion features. ### 3. Additive connector The trained `visual_flow` connector applies LayerNorm and a two-layer MLP to each motion token, then adds the result to the corresponding appearance token: ```text motion' = Linear(GELU(Linear(LayerNorm(motion)))) fused = visual + motion' ``` The fused tokens replace the normal video-token embeddings before the language model forward pass. During inference, the LoRA adapter is merged into the base language model. | Component | Parameters | Training state | Included here | | --- | ---: | --- | --- | | Qwen2.5-VL-7B base model and vision encoder | See base model | Frozen base weights; LoRA on LLM | No, downloaded separately | | VideoFlow MOFNet | 13,453,240 | Frozen | Yes | | `visual_flow` connector | 14,692,864 | Trained | Yes | | LoRA adapter (`r=32`, `alpha=64`) | 20,185,088 | Trained | Yes | | **Trainable parameters** | **34,877,952** | LoRA + connector | Yes | | **Bundled model parameters** | **48,331,192** | Includes frozen MOFNet | Yes | --- ## Why a Controlled Experiment? Current MLLMs show a consistent gap: they are substantially better at modeling *space* (objects, scenes, layouts) than *self* (motion, actions, agent state). To test whether explicit motion cues can alleviate this, we constructed SIS-Motion under strict controls: - ✅ **Same base model** — Qwen2.5-VL-7B-Instruct - ✅ **Same training data** — SIS-Motion-54K (perception + memory tasks only) - ✅ **Same LoRA configuration** — `r=32`, `α=64`, identical target modules - ✅ **Same optimization** — identical learning rate, schedule, and batch size - ⬜ **Only difference** — the optical-flow-driven motion branch + additive connector The visual-only SFT baseline serves as the direct comparison point. Any difference in results can be interpreted as the marginal contribution of motion-aware modeling. --- ## Training Configuration | Setting | Value | | ---------------- | ---------------------------------------------- | | Base model | `Qwen/Qwen2.5-VL-7B-Instruct` | | Training data | SIS-Motion-54K (MCQ + Open QA + Description) | | Data scope | Perception & memory tasks (no reasoning tasks) | | LoRA rank `r` | 32 | | LoRA alpha | 64 | | LoRA dropout | 0.05 | | LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj` | | Optimizer | AdamW | | Precision | BF16 | | Trainable params | 34,877,952 (20,185,088 LoRA + 14,692,864 connector) | | Frozen params | Visual encoder, VideoFlow optical-flow estimator, LLM base (~7B) | --- ## Results ### SIS-Bench | Model | Spatial Avg | Self Avg | Overall | | ------------------------ | ----------- | -------- | -------- | | ZeroShot (Qwen2.5-VL 3B) | 65.8 | 40.5 | 53.6 | | SFT visual-only | 72.0 | 60.3 | 66.4 | | **SIS-Motion** | **74.2** | **63.7** | **69.1** | Motion-aware modeling improves both spatial cognition (+2.2) and self-awareness (+3.4) over the visual-only SFT baseline. The gains are most pronounced on memory-level tasks, where temporal integration matters most. ### Downstream Transfer — Zero-Shot UAV Navigation Evaluated on the [OpenUAV-QA](https://huggingface.co/datasets/choucsan/OpenUAV-QA) navigation benchmark without any task-specific fine-tuning: | Model | Accuracy | | ------------------------ | --------- | | Qwen2.5-VL 3B (backbone) | 71.2% | | **SIS-Motion** | **92.2%** | The +21.0 point gain demonstrates that motion-aware representations transfer beyond the original benchmark to practical UAV decision-making scenarios. > **Note on reported results:** The results above are from the paper's **3B** backbone (Qwen2.5-VL-3B-Instruct). This repository hosts a **7B** variant (Qwen2.5-VL-7B-Instruct) with the same architecture and training protocol. The 3B model can be reproduced using the open-source [training code](https://github.com/IntelliSensing/Self-in-Space). We release the 7B version here as a stronger off-the-shelf checkpoint; the controlled-experiment conclusion (motion-aware gains over visual-only SFT) holds across both scales. --- ## Repository Files ```text sis_motion_config.json Architecture manifest (connector type, paths) adapter_config.json LoRA configuration (r=32, alpha=64) adapter_model.safetensors LoRA weights (20,185,088 params) connector_weights.pt Motion fusion MLP weights (14,692,864 params) MOF_kitti.pth Frozen VideoFlow MOFNet checkpoint (13,453,240 params) images/poster.jpeg Model poster ``` **Note:** The Qwen2.5-VL-7B-Instruct base weights are **NOT included** in this repository. They are downloaded automatically from HuggingFace at inference time. --- ## Usage This model package is not a standalone Transformers architecture. It must be loaded through the SIS-Motion evaluator, which constructs the motion branch before applying the PEFT adapter. ### 1. Clone the codebase ```bash git clone https://github.com/IntelliSensing/Self-in-Space.git cd Self-in-Space ``` ### 2. Create the motion environment With Conda: ```bash bash scripts/setup_conda.sh motion conda activate sis-motion-motion ``` Or with uv: ```bash bash scripts/setup_uv.sh motion source .venv-motion/bin/activate ``` The motion environment uses Python 3.10, PyTorch 2.6 with CUDA 12.4, Transformers 4.57.1, PEFT 0.18.1, and FlashAttention 2.7.4. ### 3. Prepare SIS-Bench Place SIS-Bench under the registered data root or override the paths directly: ```text data/SIS-Bench/SIS-Bench.jsonl data/SIS-Bench/frames/ ``` ### 4. Run inference The evaluator automatically downloads this model from HuggingFace, reads `sis_motion_config.json`, loads the VideoFlow optical-flow estimator (`MOF_kitti.pth`), applies the LoRA adapter, loads `connector_weights.pt`, and runs the full appearance-plus-motion pipeline: ```bash MODEL_PATH=choucsan/SIS-Motion \ bash scripts/eval_motion.sh ``` An already downloaded model directory is also accepted: ```bash MODEL_PATH=/path/to/SIS-Motion \ DATA_FILE=/path/to/SIS-Bench.jsonl \ FRAMES_DIR=/path/to/SIS-Bench/frames \ bash scripts/eval_motion.sh ``` The Qwen2.5-VL-7B base model is fetched from the `base_model_name_or_path` stored in `adapter_config.json`; it is not duplicated in this repository. > **Note on the optical flow estimator:** The default checkpoint (`MOF_kitti.pth`) is a VideoFlow MOFNet pretrained on KITTI. Replacing it with another optical-flow estimator (e.g., RAFT or GMFlow) requires adapting the optical-flow loading and output-conversion logic. The connector weights may not transfer to a different flow backbone and would likely require retraining. --- ## Limitations - **Not a SOTA model.** SIS-Motion is a controlled research prototype built on a 7B backbone. Stronger base models would likely yield higher absolute scores under the same setup. - **Optical flow estimation.** The VideoFlow optical-flow estimator runs on each input video, adding compute not required by the visual-only baseline. - **Training scope.** SIS-Motion-54K covers perception and memory tasks. Reasoning-level improvements come from generalization, not direct supervision. - **Inference cost.** The runtime adds 28,146,104 module parameters over the base path (13,453,240 frozen MOFNet + 14,692,864 connector). The 20,185,088 LoRA parameters are merged into the base model before generation. --- ## Citation If you use SIS-Motion in your research, please cite: ```bibtex @misc{zou2026sis, title={Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence}, author={Zhishan Zou and Guoyan Sun and Zhiwei Wei and Jiancheng Pan and Yujie Li and Mugen Peng and Wenjia Xu}, year={2026}, eprint={2607.12477}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2607.12477}, } ``` --- ## Contact For questions or collaboration requests: [choucisan@gmail.com](mailto:choucisan@gmail.com)