---
title: RynnUnify-Base
canonical_url: "https://www.modelscope.cn/models/DAMO_Academy/RynnUnify-Base"
md_url: "https://www.modelscope.cn/models/DAMO_Academy/RynnUnify-Base.md"
repository: DAMO_Academy/RynnUnify-Base
last_updated: 2026-10-26
license: apache-2.0
pipeline_tag: any-to-any
tasks:
- any-to-any
model_type:
- cosmos3_omni
architectures:
- Cosmos3OmniModel
base_model:
- nvidia/Cosmos3-Nano
base_model_relation: finetune
parameters: 15.7B
tensor_type:
- BF16
library_name:
- safetensors
language:
- en
downloads: 13
stars: 0
tags:
- rynnunify
- embodied-ai
- world-model
- visual-planning
- video-generation
---
# RynnUnify-Base
> RynnUnify-Base - DAMO_Academy 在 ModelScope 开源的模型。RynnUnify-Base: SFT foundation model of the RynnUnify unified embodied brain.
DAMO_Academy/RynnUnify-Base 是 ModelScope 魔搭社区上的 15.7B 参数any-to-any模型，采用 apache-2.0 许可，基于 nvidia/Cosmos3-Nano 构建。
- **Repository**: DAMO_Academy/RynnUnify-Base
- **License**: apache-2.0
- **Tasks**: any-to-any
- **Parameters**: 15.7B
- **Base model**: nvidia/Cosmos3-Nano
- **Tags**: rynnunify, embodied-ai, world-model, visual-planning, video-generation
- **Downloads**: 13
- **Stars**: 0
- **Last updated**: 2026-10-26
Source: https://www.modelscope.cn/models/DAMO_Academy/RynnUnify-Base
---
### RynnUnify
Agentic Embodied Intelligence via Reciprocal VLM Reasoning and World Modeling
Project Page |
Hugging Face |
ModelScope |
arXiv |
Paper
## Introduction
We present **RynnUnify**, a **Mixture-of-Transformers** architecture with an autoregressive **Reasoner** and a rectified-flow video **Generator**. Built on Cosmos3, RynnUnify connects embodied understanding, spatial grounding, and long-horizon visual planning through interleaved video-language reasoning.
- **Unified Embodied Brain.** RynnUnify brings embodied cognition, spatial grounding, and video generation into one architecture, connecting scene understanding with predictions of how the physical world evolves.
- **Interleaved Visual Planning.** RynnUnify alternates subtask reasoning with video prediction, using each generated outcome to plan the next step. Cross-chunk scene memory, self-forcing training, and adaptive durations support coherent visual planning over long horizons.
- **Agentic Interleaved Thinking.** An RL-trained agent learns when to answer directly and when to generate visual evidence. Through BCPO, it learns to formulate generation requests, interpret predicted outcomes, and select the reasoning path that benefits the task.
- **Connect Prediction to Action.** The World-Action Model (**WAM**) extension translates predictive world representations into robot control through joint video-action training, supporting composite manipulation tasks.
## Model Architecture
RynnUnify adopts a **16B unified embodied model** comprising an 8B autoregressive reasoner and an 8B video generator. The Reasoner predicts the next subtask, visual description, and duration. Through **joint attention**, the Generator attends to the Reasoner’s representations and predicts a video segment conditioned on scene memory and recent motion.
## Agentic Interleaved Thinking
Agentic interleaved thinking enhances embodied reasoning by using world modeling to imagine future scene dynamics and bring the resulting visual evidence into the reasoning process. We introduce **Branch-Conditional Policy Optimization (BCPO)** to learn both reasoning behavior and generator invocation.
BCPO samples four direct and four generation-assisted trajectories for each routable question. Within-branch reward comparisons improve reasoning behavior; differences between branch rewards supervise the routing policy. At inference time, the agent selects a branch from the original input and optionally requests generated visual evidence before answering. Only the agent is updated during training; the generator remains frozen.
See the [algorithm](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/docs/algorithm.md), [reward definitions](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/docs/rewards.md), and [service interfaces](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/docs/services.md).
## Performance
- Embodied Understanding and Reasoning
General Embodied Understanding
Interleaved Thinking on Public Benchmarks and Rynn-GEU
- Visual Planning
Long-horizon Visual Planning on Rynn-VP
Visual Planning Visualizations
- World-Action Modeling
Robot Manipulation on RoboCasa365
Real-Robot Evaluation on Astribot S1
Real-Robot Execution Examples
---
## Model Zoo
| Model | Base Model | HuggingFace | ModelScope |
| :--- | :--- | :---: | :---: |
| RynnUnify-Base | Cosmos3 Nano | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnUnify-Base) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnUnify-Base) |
| RynnUnify-Agentic-Reasoner | RynnUnify-Base | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnUnify-Agentic-Reasoner) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnUnify-Agentic-Reasoner) |
| RynnUnify-WAM-RoboCasa | RynnUnify-Base | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnUnify-WAM-RoboCasa) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnUnify-WAM-RoboCasa) |
## Quick Start
Run each workflow from the [repository](https://github.com/alibaba-damo-academy/RynnUnify) root. Use separate environments for Agentic RL and Visual Planning, with Linux, CUDA-compatible PyTorch and compatible checkpoints.
### SFT Training
Please refer to [RynnScale](https://github.com/alibaba-damo-academy/RynnScale) for Reasoner SFT and [cosmos-framework](https://github.com/NVIDIA/cosmos-framework) for Generator training.
### Agentic RL (BCPO)
Install the BCPO package in a separate environment:
```bash
cd BCPO
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[train,eval,serve]'
```
Prepare a compatible Reasoner checkpoint, [training/evaluation JSONL](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/docs/data.md), and a frozen Generator service. Start a Judge service for `text` / `mess` tasks; see [service setup](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/README.md#deployment). Then train and evaluate:
```bash
CUDA_VISIBLE_DEVICES=0 \
REASONER_MODEL=/path/to/reasoner \
TRAIN_DATA=/path/to/train.jsonl EVAL_DATA=/path/to/eval.jsonl \
GENERATOR_URL=http://127.0.0.1:18900/generate_video \
JUDGE_URL=http://127.0.0.1:8001/v1 JUDGE_MODEL=judge \
RUN_DIR=runs/bcpo BATCH_SIZE=1 bash scripts/train_and_evaluate.sh
```
Use a new `RUN_DIR` for each run. The script validates data, trains the Reasoner, then evaluates `train/final/` and writes results to `eval/`. Omit the Judge variables for tasks that do not need it. Start with `BATCH_SIZE=1` and adjust for available memory; each question produces eight trajectories. See [training options](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/docs/training.md).
### Visual Planning Inference
Prepare the compatible Cosmos framework, a checkpoint with a visual Reasoner, and an episode JSONL with your input images. See [setup and data format](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/VisualPlan/README.md).
```bash
cd VisualPlan
CHECKPOINT_PATH=/path/to/compatible/hf_export \
TEST_DATA=/path/to/episodes.jsonl \
OUTPUT_DIR="$PWD/outputs/demo" \
NPROC_PER_NODE=1 bash run_interleave.sh
```
Add `DRY_RUN=1` to inspect the command without loading models. Results include generated clips, a stitched rollout and JSON summaries.
### Embodied Understanding Evaluation
Please refer to [RynnScale](https://github.com/alibaba-damo-academy/RynnScale) for embodied understanding and reasoning evaluation using the Reasoner from RynnUnify-Base, and to the [BCPO evaluation guide](https://github.com/alibaba-damo-academy/RynnUnify/blob/main/BCPO/docs/evaluation.md) for Agentic Interleaved Thinking evaluation.
---
## Citation
```bibtex
@article{damo2026rynnunify,
title={RynnUnify: Towards Unified Embodied Brain},
author={Minghao Zhu and Bohan Hou and Kun Xiang and
Zixuan Wang and Yinzhou Tang and Yifei Yang and
Zhikai Wang and Tianyi Zhang and Kehan Li and
Haoyu Zhao and Siteng Huang and Leigang Qu and
Sicong Leng and Dongchi Huang and Mingxiu Chen and
Shijian Lu and Yue Wang and Xiaodan Liang and
Zhenyu Zeng and Deli Zhao and Chengju Liu and
Jianfei Yang and Xin Li},
journal={arXiv preprint arXiv:2610.32965},
year={2026},
url={https://arxiv.org/abs/2610.32965}
}
```