---
title: wiki-document-segmentation
canonical_url: "https://www.modelscope.cn/datasets/modelscope/wiki-document-segmentation"
md_url: "https://www.modelscope.cn/datasets/modelscope/wiki-document-segmentation.md"
repository: modelscope/wiki-document-segmentation
chinese_name: "文本分割-中文-通用领域"
last_updated: 2022-12-05
license: "Apache License 2.0"
storage_size: "1.8 KB"
domain:
  - text
tasks:
  - document-segmentation
language:
  - zh
downloads: 9267
stars: 1
---

# wiki-document-segmentation

> wiki-document-segmentation - modelscope 在 ModelScope 开源的数据集。基于wiki的文本分割数据集

modelscope/wiki-document-segmentation 是 ModelScope 魔搭社区上的document-segmentation数据集，涉及 text 领域，存储大小 1.8 KB，采用 Apache License 2.0 许可。

- **Repository**: modelscope/wiki-document-segmentation
- **License**: Apache License 2.0
- **Tasks**: document-segmentation
- **Domain**: text
- **Storage size**: 1.8 KB
- **Downloads**: 9267
- **Stars**: 1
- **Last updated**: 2022-12-05

Source: https://www.modelscope.cn/datasets/modelscope/wiki-document-segmentation

---

## 数据集描述
文本分割fine tune 样例数据

### 数据集简介
该数据集为文本分割模型中文fintune的样例数据，数据采样自中文wiki数据集。

### Clone with HTTP
* https://www.modelscope.cn/datasets/modelscope/wiki-document-segmentation.git

### 数据集加载方式
```python
from modelscope.msdatasets import MsDataset
from modelscope.utils.constant import DownloadMode
input_config_kwargs = {'delimiter': '\t'}
data = MsDataset.load(
    'wiki-document-segmentation',
    namespace='modelscope',
    download_mode=DownloadMode.FORCE_REDOWNLOAD,
    subset_name="default",
    **input_config_kwargs)
    
print(data["validation"][0])
```


### Citation
```
@inproceedings{DBLP:conf/asru/ZhangCLLW21,
  author    = {Qinglin Zhang and
               Qian Chen and
               Yali Li and
               Jiaqing Liu and
               Wen Wang},
  title     = {Sequence Model with Self-Adaptive Sliding Window for Efficient Spoken
               Document Segmentation},
  booktitle = {{IEEE} Automatic Speech Recognition and Understanding Workshop, {ASRU}
               2021, Cartagena, Colombia, December 13-17, 2021},
  pages     = {411--418},
  publisher = {{IEEE}},
  year      = {2021},
  url       = {https://doi.org/10.1109/ASRU51503.2021.9688078},
  doi       = {10.1109/ASRU51503.2021.9688078},
  timestamp = {Wed, 09 Feb 2022 09:03:04 +0100},
  biburl    = {https://dblp.org/rec/conf/asru/ZhangCLLW21.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```