--- title: CoVLA-Dataset canonical_url: "https://www.modelscope.cn/datasets/AI-ModelScope/CoVLA-Dataset" md_url: "https://www.modelscope.cn/datasets/AI-ModelScope/CoVLA-Dataset.md" repository: AI-ModelScope/CoVLA-Dataset chinese_name: CoVLA-Dataset last_updated: 2026-06-27 license: "Apache License 2.0" storage_size: "421 GB" downloads: 43385 stars: 2 --- # CoVLA-Dataset > CoVLA-Dataset - AI-ModelScope 在 ModelScope 开源的数据集。CoVLA-Dataset is a dataset comprising real-world driving videos spanning more than 80 hours. This dataset leverages a novel, scalable approach based on automated data processing and a caption generation… AI-ModelScope/CoVLA-Dataset 是 ModelScope 魔搭社区上的数据集，存储大小 421 GB，采用 Apache License 2.0 许可。 - **Repository**: AI-ModelScope/CoVLA-Dataset - **License**: Apache License 2.0 - **Storage size**: 421 GB - **Downloads**: 43385 - **Stars**: 2 - **Last updated**: 2026-06-27 Source: https://www.modelscope.cn/datasets/AI-ModelScope/CoVLA-Dataset --- # CoVLA-Dataset

WACV 2025 Oral

**CoVLA-Dataset** is a dataset comprising real-world driving videos spanning more than 80 hours. This dataset leverages a novel, scalable approach based on automated data processing and a caption generation pipeline to generate accurate driving trajectories paired with detailed natural language descriptions of driving environments and maneuvers. It includes 10,000 30-second video clips, paired with trajectory targets and language annotations generated from CAN data and front camera footage. For more details, please visit our project page [https://turingmotors.github.io/covla-ad/](https://turingmotors.github.io/covla-ad/). Overview of the Dataset ## Data fields | **Key** | **Value** | |-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **image** | sample | **frame_id** | 329 | | **vEgo** | 10.03304386138916 | | **vEgoRaw** | 10.020833015441895 | | **aEgo** | 0.46339523792266846 | | **steeringAngleDeg** | 0.6606917381286621 | | **steeringTorque** | -83.0 | | **brake** | 0.0 | | **brakePressed** | false | | **gas** | 0.0949999988079071 | | **gasPressed** | true | | **doorOpen** | false | | **seatbeltUnlatched** | false | | **gearShifter** | drive | | **leftBlinker** | false | | **rightBlinker** | false | | **orientations_calib** | [2.3436582957260557, 0.5339828947300967, 1.3629659149020594] | | **orientations_ecef** | [2.3389552760497168, 0.5209895497170147, 1.353589728168173] | | **orientations_ned** | [0.0025234392011709832, 0.03227332984737223, -2.2615545172406692] | | **positions_ecef** | [-3980150.365520416, 3315762.367044255, 3708484.8043875922] | | **velocities_calib** | [9.879017074377433, -0.011840230995096795, 0.024564830387060477] | | **velocities_ecef** | [1.7610653813101715, 8.306048478869922, -5.0501415195236214] | | **accelerations_calib** | [0.27428175425116946, 0.12695569343062033, -0.10788516598110376] | | **accelerations_device** | [0.27649870813464505, 0.12283225142665075, -0.10699598243696486] | | **angular_velocities_calib** | [0.0026360116259363207, 0.004025109052377312, -0.00268604793365312] | | **angular_velocities_device** | [0.0027046335044321763, 0.003985098643058938, -0.0026774727056080635] | | **timestamp** | 1666768003100 | | **extrinsic_matrix** | [[-0.014968783967196942, -0.9998879633843899, -4.85357778264491e-05, 0.0], [0.003242381996824406, 1.2705494208814505e-22, -0.9999947418769201, 1.2200000286102295], [0.9998827102283637, -0.014968862590224792, 0.0032420187150516235, 0.0], [0.0, 0.0, 0.0, 1.0]] | | **intrinsic_matrix** | [[2648.0, 0.0, 964.0], [0.0, 2648.0, 604.0], [0.0, 0.0, 1.0]] | | **trajectory_count** | 60 | **trajectory** | [[0.0, -0.0, 0.0], [0.4950813837155965, 0.0002547887961875119, 0.0021622613513301494], [0.9982726849068438, 0.0056820013761280435, 0.008019814119642137], [1.5000274952496726, 0.0059424162043407655, 0.010366395805683198], [1.9714437957699504, 0.012072826164266363, 0.017691995618773503], [2.4978684260880795, 0.011601311998705278, 0.02386450425538476], [3.010815767380653, 0.01801527128027971, 0.03445721142353303], [3.507063998218958, 0.01701233281058208, 0.038337927578102234], [4.012620624170714, 0.024100599226699392, 0.045395340010689886], [4.514833598833565, 0.02495601111254716, 0.049133835162865874], [5.017161220493318, 0.03149524423866552, 0.05523633716707353], [5.51940086207554, 0.030085354586579783, 0.0629749739561262], [6.03533332268388, 0.033231232243281575, 0.07405741372199495], [6.537391640025451, 0.03051862039002601, 0.08446890058718093], [7.048671047316283, 0.038067441674022755, 0.09575308668400331], [7.55109590134654, 0.03431035592675249, 0.10149061037170799], [8.059086339126619, 0.042729229684233254, 0.10987290009657202], [8.52910950102711, 0.0361088815813233, 0.11378430761802129], [9.057420775293076, 0.04137374154529525, 0.11942084703760691], [9.56262721211865, 0.03109799109499287, 0.12977970617751178], [10.063355428131272, 0.031333084537993515, 0.14035971143267495], [10.564434359898204, 0.017195610432229166, 0.1523663378360089], [11.067897560093263, 0.015964352684473423, 0.16406888445093548], [11.538305780022156, -0.005298283878670548, 0.1742140300896913], [12.075701271632234, -0.013420317597075168, 0.18348975369247966], [12.57096145582652, -0.03554497226615074, 0.19262208554922391], [13.056727974695047, -0.049523833398930905, 0.20499172623121895], [13.57320019988525, -0.07404168277320623, 0.2147168274517664], [14.071046794195906, -0.07897020052519861, 0.2226606968611588], [14.538997968829394, -0.10256663521468153, 0.23212175013944475], [15.065563638878904, -0.1167763891342656, 0.2408806359762134], [15.55517235904856, -0.14451920391994882, 0.25216240490966046], [16.060878282608606, -0.16233348628138394, 0.2632859326068148], [16.56483257931331, -0.19450740003278352, 0.2759509621400748], [17.07504346354488, -0.20961007460225636, 0.2872020973948639], [17.570511139964328, -0.24407735023413227, 0.2988333759290186], [18.080854607007407, -0.26610472918341704, 0.31079787209427323], [18.59049646081564, -0.30203120166074393, 0.32420489211775605], [19.084604904329886, -0.32645382033607545, 0.34009129336809085], [19.63586747214387, -0.3657491267911684, 0.36160403574838995], [20.150774198207014, -0.3911628434767098, 0.3794843533824954], [20.666409031086534, -0.4277895374294906, 0.3966498889678393], [21.180019483939088, -0.458788621000815, 0.40918684690405555], [21.70154365867492, -0.490295037823036, 0.4215196756925751], [22.215836213869263, -0.5137219901398054, 0.43110400157805046], [22.728543128259094, -0.5469874293170292, 0.43765565801467715], [23.241384685764114, -0.571926428598341, 0.44717513184957286], [23.767547695050713, -0.6043245975304662, 0.461574553355278], [24.28656672722109, -0.6346200237378623, 0.4724674608211735], [24.809254106414958, -0.6666944983890827, 0.4837900904850035], [25.32670419066385, -0.6950365293779399, 0.49242971573187705], [25.821659469023185, -0.7245096058611247, 0.501958900633962], [26.370683105552995, -0.7502297676767269, 0.5119963762627946], [26.89741612568614, -0.7821684517861108, 0.5221244715437666], [27.398996546439875, -0.8079249885664631, 0.5315545062900925], [27.948541952161346, -0.8422125069605737, 0.5412863716179301], [28.473985439321677, -0.8667390346179196, 0.5504376709603114], [28.973416127630998, -0.8956524219968989, 0.5591793882574851], [29.520360419055418, -0.9238145692677926, 0.5670525689197388], [30.058969038298454, -0.9543859547220845, 0.5786333952064608]] | | **caption** | The ego vehicle is moving straight at a moderate speed following leading car with acceleration. There is a traffic light close to the ego vehicle displaying a green signal. It is sunny. The car is driving on a wide road. No pedestrians appear to be present. What the driver of ego vehicle should be careful is to keep an eye on the traffic light and be prepared to stop if the light changes. ## Usage We provide a simple tutorial. Please refer to [`tutorial.ipynb`](tutorial.ipynb) for instructions on how to load the data. ## License This repository includes data under our CoVLA-Dataset Licensing Terms and Conditions and the VideoLLaMA 2 licenses. Please make sure to review both licenses carefully. The video clips and CAN data are under the CoVLA-Dataset Licensing Terms and Conditions, while the natural language descriptions are under the [VideoLLaMA 2 license](https://github.com/DAMO-NLP-SG/VideoLLaMA2?tab=readme-ov-file#-license). VideoLLaMA2 License: > This project is released under the Apache 2.0 license as found in the LICENSE file. The service is a research preview intended for non-commercial use ONLY, subject to the model Licenses of LLaMA and Mistral, Terms of Use of the data generated by OpenAI, and Privacy Practices of ShareGPT. Please get in touch with us if you find any potential violations. ## Acknowledgements This dataset is based on results obtained from a project, JPNP20017, subsidized by the New Energy and Industrial Technology Development Organization (NEDO). ## Citation ``` @InProceedings{covla_wacv2025, author = {Arai, Hidehisa and Miwa, Keita and Sasaki, Kento and Watanabe, Kohei and Yamaguchi, Yu and Aoki, Shunsuke and Yamamoto, Issei}, title = {CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving}, booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)}, month = {February}, year = {2025}, pages = {1933-1943} } @article{damonlpsg2024videollama2, title={VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs}, author={Cheng, Zesen and Leng, Sicong and Zhang, Hang and Xin, Yifei and Li, Xin and Chen, Guanzheng and Zhu, Yongxin and Zhang, Wenqi and Luo, Ziyang and Zhao, Deli and Bing, Lidong}, journal={arXiv preprint arXiv:2406.07476}, year={2024}, url = {https://arxiv.org/abs/2406.07476} }