---
title: RynnLAM
canonical_url: "https://www.modelscope.cn/models/DAMO_Academy/RynnLAM"
md_url: "https://www.modelscope.cn/models/DAMO_Academy/RynnLAM.md"
repository: DAMO_Academy/RynnLAM
last_updated: 2026-10-08
license: apache-2.0
pipeline_tag: robotics
tasks:
  - robotics
language:
  - en
downloads: 0
stars: 0
tags:
  - latent-action-model
  - latent-action
  - vision-language-action
  - vla
  - robot-manipulation
  - self-supervised
  - embodied-ai
  - pytorch
---

# RynnLAM

> RynnLAM - DAMO_Academy 在 ModelScope 开源的模型。RynnLAM — motion-grounded latent action model

DAMO_Academy/RynnLAM 是 ModelScope 魔搭社区上的robotics模型，采用 apache-2.0 许可。

- **Repository**: DAMO_Academy/RynnLAM
- **License**: apache-2.0
- **Tasks**: robotics
- **Tags**: latent-action-model, latent-action, vision-language-action, vla, robot-manipulation, self-supervised, embodied-ai, pytorch
- **Downloads**: 0
- **Stars**: 0
- **Last updated**: 2026-10-08

Source: https://www.modelscope.cn/models/DAMO_Academy/RynnLAM

---

# RynnLAM — motion-grounded latent action model

A Latent Action Model that compresses the dynamics between two RGB frames into a compact,
embodiment-agnostic latent action. Trained self-supervised to predict future visual features,
regress dynamic 3D flow, and decouple camera motion from ego motion, so the latent describes
physical motion rather than 2D appearance change.

This is the checkpoint used to densely label the Stage-1 pretraining corpus behind
[RynnVLA-Latent](https://github.com/alibaba-damo-academy/RynnVLA-Latent). It is published so
that the latent actions in a downstream VLA run are reproducible from a fixed, verifiable
labeler rather than from a description of one.

## What it produces

The default representation is `ktoken_zcam`, **608 floats per frame pair**:

| slice | dims | content |
| --- | --- | --- |
| `k_tokens.flatten(1)` | 512 | 8 K-tokens x 64, from the K-token compressor |
| `z` | 64 | latent action |
| `camera_pose_latent` | 32 | camera ego-motion, decoupled from scene motion |

Six other representations are available from the same forward pass: `ktoken`, `z`, `zcam`,
`hints_pool`, `full`, `encoder`.

## Architecture

397.3 M parameters, 684 tensors, float32.

| module | tensors | params | role |
| --- | --- | --- | --- |
| `encoder` | 407 | 304.4 M | DA3-Large / DINOv2-family backbone over frame pairs |
| `flow_decoder` | 104 | 40.7 M | dynamic 3D flow regression |
| `latent_encoder` | 99 | 20.3 M | self-attention latent encoder |
| `hint_compressor` | 9 | 16.9 M | compresses motion hints into the 8 K-tokens |
| `recon_decoder_ktoken` | 44 | 14.7 M | K-token reconstruction decoder |
| `cam_dec` / `cam_dec_adversarial` | 20 | 0.2 M | camera-pose decoder |

Key config: `model_version=v5`, `flow_decoder_version=v5`, `latent_encoder_type=selfattn`,
`use_k_token_recon=True`, `k_token_source=features`, `num_k_tokens=8`,
`k_token_bottleneck_dim=64`, `latent_dim=64`, `camera_pose_latent_dim=32`,
`motion_hint_dim=128`, `embed_dim=1024`, `patch_size=14`, `target_hw=[238, 322]`.

**The artifact is self-contained.** `encoder_checkpoint_path` is `None` and the 407 encoder
tensors are inside the file, so loading it downloads nothing and needs no separate encoder
weights. The loader pins `encoder_finetune_mode="freeze"` because there is nothing left to
initialize.

## Usage

Install [RynnVLA-Latent](https://github.com/alibaba-damo-academy/RynnVLA-Latent) first; this
checkpoint is loaded by its `rynnlam` package.

Label a corpus from a metadata manifest:

```bash
python scripts/label_latent.py \
    --checkpoint RynnLAM.pt \
    --metadata <manifest.json> \
    --output-dir <out> \
    --device cuda --precision bf16 --representation ktoken_zcam \
    --gap 4 --pair-stride 4 --normalization imagenet --bucket auto
```

Each episode yields a `latent.npz` with `latent_action` `float16 [N, 608]`, `pair_indices`
`int32 [N, 2]`, and a `meta` JSON string recording `schema_version`, `code_shape`, and the
`checkpoint_sha256` of the labeler that produced it. Those `gap`/`pair_stride`/`normalization`/
`bucket` values are the ones the Stage-1 corpus was labeled with; changing them produces a
different, incompatible corpus.

Or encode frame pairs directly:

```python
from rynnlam.inference import RynnLAMEncoder

enc = RynnLAMEncoder("RynnLAM.pt", device="cuda", precision="bf16")
latents = enc(images, representation="ktoken_zcam")   # [batch, 608]
```

`images` is `[batch, 2, height, width, 3]`, float32 in `[0, 1]`, RGB. Along the pair axis,
index 0 is the earlier frame `t` and index 1 is the later frame `t + gap`; the latent describes
the motion from 0 to 1. Out-of-range or non-finite input raises rather than being clamped.
Frames are resized to `target_hw = [238, 322]` internally.

Loading never requires `--trust-checkpoint`: the file reads under `torch.load(...,
weights_only=True)` and then strict-loads with 0 trainable parameters.

## Provenance and integrity

```
sha256 (this file)  0ab843b0b308357db48b50d3b27ce5c39eac134fa9d5f7fe5172c3320aded424
sha256 (internal)   d066f871fe6e05525ab7ad48c706cd3f564cc8546f9a67af2d597426d560ed56
size                1,589,652,281 bytes
```

Verify a download:

```bash
sha256sum RynnLAM.pt
```

**Both digests are published deliberately.** The second is the hash of the file as it was used
internally, and it is the value written into the `meta` of every latent `.npz` in the Stage-1
corpus. This file has the *same weights* — all 684 tensors are bit-identical — and differs only
in the config block, which was sanitized for release. So a corpus labeled internally records the
internal digest, and a corpus you label with this file records the public one. If you are
checking whether your latents came from these weights, compare against the internal digest; if
you are checking whether your download is intact, compare against the public one.

The equivalence is not asserted, it was measured: labeling the same videos with each file
produced `float16 [24, 608]` arrays that are bit-identical, `maxabsdiff = 0.0`.

## What was removed from the config, and why

The internal checkpoint carried 143 config entries; this one carries 134. The weights were not
touched. Removed:

| key | why |
| --- | --- |
| `encoder_checkpoint_path` | absolute path on an internal mount; also dead at load time, the encoder is in the file |
| `output_root`, `resume` | internal training-run locations |
| `safetensors_root` | internal location of the LAM training corpus |
| `dataset_sampling_weights`, `dataset_sampling_temperature`, `dataset_role_sampling_weights` | the training corpus composition and its sampling recipe are not part of this release |
| `val_scene_id` | an identifier into that corpus |
| `wandb_project`, `oss_mode` | internal experiment-tracking and storage-backend flags |

None of these reach the model: `rynnlam.model.build_model` filters the config by the `RynnLAM`
constructor signature, so training-only keys are dropped at load time regardless. They were
removed because they disclosed internal infrastructure, not because they affected behaviour —
which is why the bit-identical-latents check above is the meaningful guarantee.

The remaining 134 entries are architecture and loss hyperparameters, and are what
`build_model` consumes.

## Requirements

Verified against `torch 2.7.1+cu126` on an NVIDIA H20, and under `weights_only=True` on CPU.
`precision="bf16"` needs a GPU with bfloat16 support; use `precision="fp32"` otherwise. The
encoder is DA3-Large-class, so budget roughly 1.6 GB of disk and a few GB of host memory to
load, plus GPU memory for the pair forward pass.

## Limitations

- The latent is a *relative* action between two frames. It has no metric scale and no absolute
  pose; converting it to a robot action space is what RynnVLA-Latent's Stage-2 post-training is
  for.
- `target_hw = [238, 322]` is fixed by training. Input at other aspect ratios is bucketed and
  resized by the labeler, which is a distribution shift the model was not trained to handle.
- No optimizer, scheduler, or EMA state is included. This is an inference artifact and cannot
  resume the training run that produced it.

## License

Apache-2.0. Note that the RynnVLA-Latent source license covers code, not weights or datasets;
this file is released separately under the same license. Training-data provenance is not
redistributed here — obtaining this checkpoint grants no rights to any corpus it was trained on.