---
title: wikipedia-cn-20230720-filtered
canonical_url: "https://www.modelscope.cn/datasets/AI-ModelScope/wikipedia-cn-20230720-filtered"
md_url: "https://www.modelscope.cn/datasets/AI-ModelScope/wikipedia-cn-20230720-filtered.md"
repository: AI-ModelScope/wikipedia-cn-20230720-filtered
chinese_name: wikipedia-cn-20230720-filtered
last_updated: 2023-12-08
license: cc-by-sa-3.0
storage_size: "497 MB"
domain:
  - configs
  - size_categories
  - task_categories
  - language
tasks:
  - "{config_name=default, data_files=[{path=wikipedia-cn-20230720-filtered.jsonl, split=train}]}"
  - "100K<n<1M"
  - text-generation
  - zh
downloads: 46727
stars: 7
---

# wikipedia-cn-20230720-filtered

> wikipedia-cn-20230720-filtered - AI-ModelScope 在 ModelScope 开源的数据集。本数据集基于中文维基2023年7月20日的dump存档。作为一项以数据为中心的工作，本数据集仅保留了 254,547条 质量较高的词条内容。具体而言： 过滤了Template, Category, Wikipedia, File, Topic, Portal, MediaWiki, Draft, Help等特殊类型的词条…

AI-ModelScope/wikipedia-cn-20230720-filtered 是 ModelScope 魔搭社区上的{config_name=default, data_files=[{path=wikipedia-cn-20230720-filtered.jsonl, split=train}]}、100K<n<1M、text-generation数据集，涉及 configs、size_categories、task_categories 领域，存储大小 497 MB，采用 cc-by-sa-3.0 许可。

- **Repository**: AI-ModelScope/wikipedia-cn-20230720-filtered
- **License**: cc-by-sa-3.0
- **Tasks**: {config_name=default, data_files=[{path=wikipedia-cn-20230720-filtered.jsonl, split=train}]}, 100K<n<1M, text-generation, zh
- **Domain**: configs, size_categories, task_categories, language
- **Storage size**: 497 MB
- **Downloads**: 46727
- **Stars**: 7
- **Last updated**: 2023-12-08

Source: https://www.modelscope.cn/datasets/AI-ModelScope/wikipedia-cn-20230720-filtered

---

本数据集基于中文维基2023年7月20日的dump存档。作为一项以数据为中心的工作，本数据集仅保留了 `254,547条` 质量较高的词条内容。具体而言：
* 过滤了Template, Category, Wikipedia, File, Topic, Portal, MediaWiki, Draft, Help等特殊类型的词条
* 使用启发式的方法和自有的NLU模型过滤了一部分质量较低的词条
* 过滤了一部分内容较为敏感或存在争议性的词条。
* 进行了简繁转换和习惯用词转换，确保符合中国大陆地区的习惯用词。


This dataset is based on the Chinese Wikipedia dump archive from July 20th, 2023. As a data-centric effort, the dataset retains `254,574` high-quality entries. Specifically:
* Entries of special types such as Template, Category, Wikipedia, File, Topic, Portal, MediaWiki, Draft, and Help have been filtered out.
* A heuristic approach and proprietary NLU models have been used to filter out some low-quality entries.
* Entries with sensitive or controversial content have also been filtered out.
* To ensure compliance with language usage in mainland China, the dataset underwent conversions from simplified to traditional Chinese, as well as colloquial language conversions.
