💫 Project Page | 🤗 Hugging Face | 🤖 ModelScope | 🚀 Demo | 📄 arXiv
--- ## 🌟 Abstract We introduce **RynnWorld-Teleop**, a robot-centric generative world model that instantiates the paradigm of **digital teleoperation**—decoupling robot data collection from physical hardware constraints. By transforming an operator's real-time hand-pose stream into high-fidelity egocentric robotic videos from a single reference image, RynnWorld-Teleop enables the scaling of expert trajectories in a purely virtual environment. Our framework integrates depth-aware skeletal conditioning with a progressive human-to-robot training curriculum, allowing it to inherit rich manipulation priors from large-scale human datasets. To support interactive use, we distill the model into a **causal, autoregressive student capable of real-time streaming**. Policies trained exclusively on RynnWorld-Teleop synthetic data achieve effective zero-shot Sim2Real transfer, demonstrating its power as a high-fidelity data engine for scaling dexterous robotic learning. --- ## 📦 This Repository This repository hosts the **distilled causal (streaming) checkpoint** of RynnWorld-Teleop. It is designed for **frame-by-frame, real-time** generation with a sliding KV cache. ### Files | File | Description | | :--- | :--- | | `ema_weights.bin` | EMA weights of the causal transformer (distilled from the SFT teacher) | | `control_patch_embedding.bin` | Conv3d patch embedding that injects hand-pose control video | | `control_scale.bin` | Learnable scalar for control-signal fusion | | `control_running_stats.bin` | Running stats for control normalization | ### Model Zoo | Model | HuggingFace | ModelScope | | :--------------- | :---------: | :--------: | | SFT (teacher) | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnWorld-Teleop) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnWorld-Teleop) | | Causal (streaming, this repo) | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnWorld-Teleop-Causal) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnWorld-Teleop-Causal) | --- ## 🚀 Quick Start Please refer to the [code repository](https://github.com/alibaba-damo-academy/RynnWorld-Teleop) for the full setup, training, and inference pipeline. ### 🔧 Environment Setup ```bash conda create -n "rynnworld-teleop" python=3.10 -y conda activate rynnworld-teleop pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu121 pip install -r requirements.txt ``` ### 📖 Download the Checkpoint Our model is developed on top of [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers). Download the base model and place it under `pretrained/`: ``` RynnWorld-Teleop/ └── pretrained/ ├── Wan2.2-TI2V-5B-Diffusers/ │ ├── model_index.json │ ├── scheduler/ │ ├── transformer/ │ ├── vae/ │ └── ... └── RynnWorld-Teleop-Causal/ ├── ema_weights.bin ├── control_patch_embedding.bin ├── control_scale.bin └── control_running_stats.bin ``` Download the causal weights: ```bash mkdir -p pretrained/RynnWorld-Teleop-Causal huggingface-cli download Alibaba-DAMO-Academy/RynnWorld-Teleop-Causal --local-dir pretrained/RynnWorld-Teleop-Causal ``` --- ## 🌊 Streaming Inference (Real-time) For real-time streaming inference with the distilled causal model, use `inference_streaming.py`, which supports frame-by-frame generation with a KV cache. ### Basic Usage ```bash python inference_streaming.py \ --image first_frame.png \ --control_video control.mp4 \ --checkpoint pretrained/RynnWorld-Teleop-Causal \ --output results/streaming_demo ``` ### Advanced Options **FP8 Quantization** (Hopper GPUs: H100/H800 only): ```bash python inference_streaming.py \ --image first_frame.png \ --control_video control.mp4 \ --checkpoint pretrained/RynnWorld-Teleop-Causal \ --output results/streaming_fp8 \ --fp8 ``` - Automatically detects GPU compute capability - Skips with warning on non-Hopper GPUs - Requires `torchao` package **torch.compile** (faster inference): ```bash python inference_streaming.py \ --image first_frame.png \ --control_video control.mp4 \ --checkpoint pretrained/RynnWorld-Teleop-Causal \ --output results/streaming_compiled \ --compile ``` - First sample includes compile overhead (~30-60s) - Subsequent samples run 1.3–1.6× faster **Batch Inference from Dataset**: ```bash python inference_streaming.py \ --data_json data/sample_data.json \ --checkpoint pretrained/RynnWorld-Teleop-Causal \ --output_dir results/batch_demo \ --num_samples_per_dataset 3 \ --fp8 --compile ``` ### Streaming Model Architecture The streaming model (`core/streaming/` in the code repo) implements: - **Causal Attention** — frame-by-frame generation with a sliding KV cache - **Control Patch Embedding** — skeleton / hand-pose conditioning - **Dynamic Cache** — efficient memory management for long sequences Key components: - `WanCausalTransformer3DModel` — causal transformer with KV-cache support - `DynamicCache` — sliding-window KV cache with sink-frame preservation - `WanStreamingPipeline` — frame-by-frame inference pipeline --- ## 📑 Citation If you find this project useful, please cite: ```bibtex @article{rynnworld_teleop, title = {RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation}, author = {DAMO Academy, Alibaba Group}, year = {2026}, } ``` ## License Apache License 2.0 — see [LICENSE](https://github.com/alibaba-damo-academy/RynnWorld-Teleop/blob/main/LICENSE) for details.