--- title: Dexterity-BEV-Dataset canonical_url: "https://www.modelscope.cn/datasets/dexforce/Dexterity-BEV-Dataset" md_url: "https://www.modelscope.cn/datasets/dexforce/Dexterity-BEV-Dataset.md" repository: dexforce/Dexterity-BEV-Dataset chinese_name: "Dexterity-BEV：时空对齐机器人操作数据集" last_updated: 2026-07-18 license: "Apache License 2.0" storage_size: "46 GB" downloads: 11129 stars: 0 --- # Dexterity-BEV-Dataset > Dexterity-BEV-Dataset - dexforce 在 ModelScope 开源的数据集。The Dexterity-BEV Dataset is a spatiotemporally aligned robot manipulation dataset designed for generalizable and cross-embodiment robot policy learning. It aligns multi-view visual observations, robot… dexforce/Dexterity-BEV-Dataset 是 ModelScope 魔搭社区上的数据集，存储大小 46 GB，采用 Apache License 2.0 许可。 - **Repository**: dexforce/Dexterity-BEV-Dataset - **License**: Apache License 2.0 - **Storage size**: 46 GB - **Downloads**: 11129 - **Stars**: 0 - **Last updated**: 2026-07-18 Source: https://www.modelscope.cn/datasets/dexforce/Dexterity-BEV-Dataset --- # DexForce Manipulation Dataset ## Introduction The **DexForce Manipulation Dataset** is a large-scale, 3D aligned manipulation dataset designed to train generalizable vision-language-action models (VLA) and world-action models (WAM). The dataset unifies multiple public and internal robot manipulation datasets. Then, we perform filtering, data-compression (for RGB and depth frames into video), 3D alignment, clip segmentation and language annotation. It provides all necessary inputs to train policies with **3D-aware multi-view observations** (depth observation, camera intrinsic/extrinsic parameters) and **aligned SE(3) actions**, towards robust cross-embodiment, cross-camera, and cross-scene generalization. ![roadmap.jpg](./roadmap.jpg) ## Dataset Highlights - **3D Input Alignment** – We perform convention alignment, verification and/or direct estimation of camera intrinsic/extrinsic parameters. This is a process combines manual operations (assisted by a customized GUI), classical algorithms and vision foundation models. - **Aligned Action Space** – Robot end-effector poses are expressed as SE(3) poses in the same world frame as the camera extrinsic. This is achieved and verified by rendering the URDF on camera view and aligned with camera observation. - **Sub-Task Clip Segmentation and Language Annotation** (to be released) – We segment the trajectory as sub-task clips, and perform language annotations. This is a pipeline combines classical agorithms and VLM annotation. This data would be later released separately. - **Data Compression** – The dataset are compressed for the efficiency of storage, transmission and loading. In particular, we encode the RGB image frames as mp4 video bytes (lossy compression), and encode depth image frames as avi vidoe bytes (lossless compression). - **Multi-Dataset Integration** – Includes aligned data from RoboTwin, LIBERO, RoboCasa, Robomind, Droid, Agibot, egocentric human demonstrations, and DexForce internal datasets. ## Source Datasets | Dataset | Embodiments | Task Types | |---------|-------------|------------| | **RoboTwin 2.0** | Dual-arm Agilex | Bimanual manipulation, domain randomized | | **LIBERO** | Single-arm Franka | Object manipulation, goal-conditioned, long-horizon | | **RoboCasa** | Single-arm Franka | Household tasks, pick-and-place | | **Robomind 2.0** | Multi-embodiment | Diverse manipulation skills | | **Droid** | Franka | In-the-wild real-world manipulation | | **Agibot** | Dual-arm Agibot | About 200 diverse manipulation tasks | | **Egocentric Demonstrations** | Human hand | Deictic actions, hand-object interaction | | **DexForce Internal** | W1 humanoid, A1 semi-humanoid, Agilex | Diverse single- and bi-arm manipulation tasks | ## Data Content and Format All trajectories are stored in the **HDF5 format**, where each trajectory is saved as a single, (mostly) self-contained `.hdf5` file ready for visualization. All observations, actions, and metadata for a trajectory are packed within one file, making it portable and easy to distribute. We do not enforce key naming convention across source datasets. Due to the heterogeneity of data sources (varying state representations, camera configurations), we preserve original key names whenever possible for compatibility and ease of verification. Key mapping across datasets is handled at the DataLoader level via configuration files, allowing each model to define its own key mapping and compute derived quantities on the fly. Example configuration: ```json "keys_config": { "libero": { "main_rgb": "/observation/agentview_rgb", "left_rgb": "/observation/eye_in_hand_rgb", "joint_angle": "/jps" }, "cobotmagic": { "main_rgb": "/observations/images/cam_high", "left_rgb": "/observations/images/cam_left_wrist", "right_rgb": "/observations/images/cam_right_wrist", "joint_angle": "/observations/qpos" } } ``` ### Key Data Types and Conventions #### Data Synchronization and Temporal Indexing - All data within a trajectory is temporally synchronized and indexed by `knot_index` (integer), which serves as a logical timestamp. Thus, each trajectory supports a `get_data_dict_at_knot(knot_idx: int) -> dict[str, Tensor]` interface that returns all data for a given knot. - However, data from different sensors (e.g., images from main and left-wrist cameras) are not captured at idential time (despite synchronization) in the source dataset. Thus, you might observe inconsistency among different observations (e.g., gripper close in main camera but open in wrist camera). Actually, this leads to a lot of effort in 3D alignment processing. #### Parallel-Jaw Gripper TCP Frame Convention The Tool Center Point (TCP) frame for a parallel-jaw gripper follows this convention: - **Origin**: Midpoint between the two fingertips. - **Z-axis**: Points in the direction the fingers are facing (forward). - **X-axis**: Points along the finger opening/closing direction. - **Y-axis**: Defined as the cross product of Z and X. - When a wrist camera is present, the camera mounting direction is the **+Y** direction to resolve the remaining sign ambiguity. ![TCP](./TCP.jpg) #### Camera Frame Convention Camera reference frames follow the standard pinhole convention (consistent with OpenCV): - **Z-axis**: Points along the optical axis, from the camera outward into the scene (near to far). - **X-axis**: Points to the right in the image plane. - **Y-axis**: Points downward in the image plane. ![Camera Frame Convention](./Camera.jpg) ### Image Data: Resolution, Compression, and Storage - **Resolution**: All stored images (RGB and depth) have a maximum resolution of 640×480. If the original resolution exceeds this limit, images are resized while preserving the aspect ratio (fixed aspect ratio resize). Depth maps use nearest-neighbor interpolation (`cv2.INTER_NEAREST`) to avoid introducing invalid depth values, and camera intrinsics are adjusted accordingly. - **RGB Frames**: Encoded as **MP4 video** (lossy compression). Multi-frame RGB images are encoded as MP4 video bytes and stored within the HDF5 file. Images are 3-channel uint8 RGB; decoded frames can be directly converted to PIL Image. Lossy compression is acceptable and has been validated in the VLA and WAM training. - **Depth Frames**: Encoded as **AVI video** using lossless compression (MP4 is not used due to its lossy nature). Depth images are stored as single-channel uint16 in **millimeters (mm)**. ## License Aligned DexWorld provides standardized representations, alignment annotations, and processing tools built upon multiple source datasets. Each source dataset remains subject to its original license and terms of use. Certain source data are not redistributed and must be obtained directly from the original providers. ## Citation If you use this dataset in your research, please consider citing: ``` @inproceedings{zhou2025dexbev, title={Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning}, author={Huayi Zhou and Wei Gao and Dekun Lu and Ruiji Liu and Zhanqi Zhang and Ziyang Zhang and Jian Chen and Wenlve Zhou and Sheng Xu and Shumin Li and Kangyi Guo and Shichen Xu and Zixin Huang and Yongyi Su and Kui Jia}, year={2025} } ``` ``` @misc{dexforce_dataset, title={DexForce Manipulation Dataset}, author={DexForce Technology}, year={2025}, howpublished={\url{https://modelscope.cn/datasets/dexforce/Dexterity-BEV-Dataset}}, } ``` ## Contact For questions about the dataset, data processing pipeline, or usage, please contact the authors via the project page.