---
title: Video_Description_Editing
canonical_url: "https://www.modelscope.cn/datasets/modelscope/Video_Description_Editing"
md_url: "https://www.modelscope.cn/datasets/modelscope/Video_Description_Editing.md"
repository: modelscope/Video_Description_Editing
last_updated: 2024-09-02
license: "Apache License 2.0"
storage_size: "5.5 MB"
domain:
  - configs
  - text
tasks:
  - "{config_name=vatex, data_files=[{path=vatex.csv, split=train}]}"
  - multi-modal
language:
  - en
  - zh
downloads: 822
stars: 1
---

# Video_Description_Editing

> Video_Description_Editing - modelscope 在 ModelScope 开源的数据集。VATEX-EDIT and EMMAD-EDIT are two datasets that support Video Description Editing task. Each data sample is a quadruple (video, command, reference caption, edited caption) .

modelscope/Video_Description_Editing 是 ModelScope 魔搭社区上的{config_name=vatex, data_files=[{path=vatex.csv, split=train}]}、multi-modal数据集，涉及 configs、text 领域，存储大小 5.5 MB，采用 Apache License 2.0 许可。

- **Repository**: modelscope/Video_Description_Editing
- **License**: Apache License 2.0
- **Tasks**: {config_name=vatex, data_files=[{path=vatex.csv, split=train}]}, multi-modal
- **Domain**: configs, text
- **Storage size**: 5.5 MB
- **Downloads**: 822
- **Stars**: 1
- **Last updated**: 2024-09-02

Source: https://www.modelscope.cn/datasets/modelscope/Video_Description_Editing

---

# Dataset

VATEX-EDIT and EMMAD-EDIT are two datasets that support  **Video Description Editing** task.  Each data sample is a quadruple *(video, command, reference caption, edited caption)* . 

You can download the whole dataset annotation files and video features here. All files are structured as the followings:

VATEX-EDIT和EMMAD-EDIT是两个支持 **视频描述编辑** 任务的数据集。每个数据样本是一个四元组 *(video, command, reference caption, edited caption)* 。

你可以在这里下载整个数据集注释文件和视频特征。所有文件的结构如下:

```
dataset/
├── EMMAD-EDIT
│   ├── clip_cn_feats/
│   ├── example_videos/
│   ├── middle_files/
│   ├── abstract_attr/
│   ├── training.json
│   ├── validation.json
│   └── test.json
└── VATEX-EDIT
    ├── blip_en_feats/
    ├── clip_en_feats/
    ├── example_videos/
    ├── middle_files/
    ├── training.json
    ├── validation.json
    └── test.json

```

## VATEX-EDIT

It is automatically built upon the English video captioning dataset [VATEX (EN)](https://arxiv.org/pdf/1904.03493.pdf) describing over 600 human activities. The VATEX-EDIT dataset contains 34,269 videos and 1,057,956 *(video, command, reference caption, edited caption)* quadruples. We follow the original VATEX dataset to get training, validation and public test split according to video ids. 

它是在英语视频字幕数据集[VATEX (EN)](https://arxiv.org/pdf/1904.03493.pdf)上自动构建的，描述了600多种人类活动。VATEX-EDIT数据集包含34,269个视频和1,057,956个 *(video, command, reference caption, edited caption)* 四元组。遵循原始的VATEX数据集，根据视频id进行training、validation和public test划分。

| #Videos                | #Editing instances         |
| ---
---
---
---
---
---
---
- | ---
---
---
---
---
---
---
---
-- |
| Train / Val / Test     | Train / Val / Test         |
| 25,467 / 2,935 / 5,867 | 784,805 / 91,513 / 181,638 |

**Annotation format:**

```
{
    "vid": video id, 
    "dtype": specific type of editing command,
    "command": operation (add/delete), 
    "attr": edited attribute, 
    "atype": type of attribute (verb/noun/modifier), 
    "oldcap": reference caption, 
    "reference": positioned reference using <mask> if assigning positions, 
    "newcap": edited caption,
}
```
| dtype       | global_len_add  | global_len_dele | global_attr_add    | global_attr_dele   | local_len_add     | local_len_dele    | local_attr_add   |
| ---
---
---
-- | ---
---
---
---
---
 | ---
---
---
---
---
 | ---
---
---
---
---
---
 | ---
---
---
---
---
---
 | ---
---
---
---
---
-- | ---
---
---
---
---
-- | ---
---
---
---
---
- |
| **command** | *<add, - , - >* | *<del, - , - >* | *<add, - , attr >* | *<del, - , attr >* | *<add, pos , - >* | *<del, pos , - >* | *<add,pos,attr>* |

**Download**
```python
# Use modelscope sdk to load the dataset: (streaming mode)
ds = MsDataset.load('modelscope/Video_Description_Editing', subset_name='vatex', split='train', use_streaming=True)
    for item in ds:
        print(item)

# name <str>: the file base name
# path:FILE <str>: local path
```


**[Note]** Due to the legal and privacy concerns, we cannot directly share the original videos from VATEX dataset. You can get the related original videos following the instructions of [VATEX dataset website](https://eric-xw.github.io/vatex-website/download.html).

**[注意]** 出于法律和隐私方面的考虑，我们不能直接分享VATEX数据集的原始视频。您可以按照[VATEX数据集网站](https://eric-xw.github.io/vatex-website/download.html)的说明获取相关的原始视频。

**Data Example**
<video src="QAgcSr8Khus_000012_000022.mp4"></video>
```
{
    "vid": "QAgcSr8Khus_000012_000022", 
    "dtype": "local_attr_add", 
    "command": "<add>", 
    "attr": "drop", 
    "atype": "verb", 
    "oldcap": "a man pets a cat that 's there .", 
    "reference": "a man <mask> pets a cat that 's there ."
    "newcap": ["a man drops some garbage off outdoors and pets a cat that 's there ."], 
}
```


## EMMAD-EDIT

It is manually collected based on the Chinese E-commerce video captioning dataset [E-MMAD](https://e-mmad.github.io/e-mmad.net/index.html). Given the product video, the original advertising video description and external product information, we further manually annotate video description editing samples. 

The E-MMAD dataset has overall 23,960 editing instances for 12,295 product videos with two remarkable characteristics, i.e. long videos/descriptions and diverse attributes. The average video length is 27.1 seconds and the average description length is around 100 words.  

它是基于中国电商视频字幕数据集[E-MMAD](https://e-mmad.github.io/e-mmad.net/index.html)人工采集的。给定产品视频、原始广告视频描述和外部产品信息，进一步手动标注视频描述编辑样本。

E-MMAD数据集共有12,295个产品视频的23,960个编辑实例，具有两个显著的特点，即长视频/描述和多样化属性。视频平均长度为27.1s，平均描述长度在100字左右。

|                   | #Videos               | #Editing instances     |
| ---
---
---
---
---
-- | ---
---
---
---
---
---
---
 | ---
---
---
---
---
---
---
- |
|   EMMAD-EDIT                 | Train / Val / Test    | Train / Val / Test     |
| *specific subset*        | 16,176 / 5,418 / 5,502 | 31,610 / 10,586 / 10,737 |
| *abstract subset* | 15,955 / 5,328 / 5,432 | 15,959 / 5,328 / 5,432  |

**Annotation format:**

```
{
    "vid": video id, 
    "dtype": specific type of editing command,
    "command": operation (add/delete), 
    "attr": edited attribute, 
    "atype": type of attribute (specific/abstract), 
    "oldcap": reference caption, 
    "reference": positioned reference using <mask> if assigning positions, 
    "newcap": edited caption,
    "allattr": product structure information,
    "video_url": url to download the original video,
    "video_title": video title,
}
```
| dtype       | global_len_add  | global_len_dele | global_attr_add    | global_attr_dele   | local_len_add     | local_len_dele    | local_attr_add   |
| ---
---
---
-- | ---
---
---
---
---
 | ---
---
---
---
---
 | ---
---
---
---
---
---
 | ---
---
---
---
---
---
 | ---
---
---
---
---
-- | ---
---
---
---
---
-- | ---
---
---
---
---
- |
| **command** | *<add, - , - >* | *<del, - , - >* | *<add, - , attr >* | *<del, - , attr >* | *<add, pos , - >* | *<del, pos , - >* | *<add,pos,attr>* |

**Download**
```python
# Use modelscope sdk to load the dataset: (streaming mode)
ds = MsDataset.load('modelscope/Video_Description_Editing', subset_name='emmad', split='train', use_streaming=True)
for item in ds:
    print(item)
    
# name <str>: the file base name
# path:FILE <str>: local path
```

We provide the frame clip features  and you can download the original video using the video url. The more challenging subset of *abstract attribute*  is put under `EMMAD-EDIT/abstract_attr/`.

我们提供frame clip features，您可以使用视频url下载原始视频。更具有挑战性的 *抽象属性* 的子集放在`EMMAD-EDIT/abstract_attr/`下。

**Data Example**
<video src="200563409291.mp4"></video>
```
{
    "vid": 200563409291, 
    "dtype": "local_attr_add", 
    "command": "<add>", 
    "attr": "实用,俏皮", 
    "atype": "specific", 
    "oldcap": "经典水桶包，子母包设计，再现繁盛时代的小鹿包。外形酷似水桶，包身圆润，别致的子母包设计，追求简约、时尚，凸现自我的创意个性与个人色彩。", 
    "reference": "经典水桶包，子母包设计，再现繁盛时代的小鹿包。外形酷似水桶，包身圆润<mask>，别致的子母包设计，追求简约<mask>、时尚，凸现自我的创意个性与个人色彩。", 
    "newcap": "经典水桶包，子母包设计，再现繁盛时代的小鹿包。外形酷似水桶外 型，包身圆润又不失俏皮的造型，别致的子母包设计，追求简约实用的时尚，凸现自我的创意个性与个人色彩。"
    "allattr": "品类:斜挎包,单肩包,水桶包,女包;时间季节:2020;新品:新款;风格:时尚,休闲,简约,潮流,欧美时尚;修饰:手提,小鹿;人群:女士;上市时间:2018年春夏;大小:中;箱包硬度:软;款式:单肩包;里料材质:织物;背包方式:单肩斜挎手提;内部结构:手机袋,证件袋,拉链暗袋;品牌:VANESSA HOGAN;颜色分类:香草白1,黑色,香草白,婴儿粉;皮革材质:牛皮;是否可折叠:否;适用场景:休闲;图案:纯色;质地:牛皮;流行元素:车缝线;货号:VH1804158020405;肩带样式:单根;形状:水桶形;销售渠道类型:商场同款(线上线下都销售);流行款式名称:水桶包;成色:全新;提拎部件类型:软把;闭合方式:磁扣;适用对象:青年;有无夹层:无;", 
}
```

## Citation

```
@article{Yao2023VDE,
  title={Edit As You Wish: Video Description Editing with Multi-grained Commands},
  author={Linli Yao and Yuanmeng Zhang and Ziheng Wang and Xinglin Hou and Tiezheng Ge and Yuning Jiang and Qin Jin},
  journal={arXiv preprint arXiv:2305.08389},
  year={2023}
}
```
