---
title: WuDaoCorpus-200G-ChatGLM-tokenized
canonical_url: "https://www.modelscope.cn/datasets/whynlp/WuDaoCorpus-200G-ChatGLM-tokenized"
md_url: "https://www.modelscope.cn/datasets/whynlp/WuDaoCorpus-200G-ChatGLM-tokenized.md"
repository: whynlp/WuDaoCorpus-200G-ChatGLM-tokenized
last_updated: 2024-03-19
license: CC-BY-NC-SA-4.0
storage_size: "63 GB"
downloads: 1621
stars: 1
---

# WuDaoCorpus-200G-ChatGLM-tokenized

> WuDaoCorpus-200G-ChatGLM-tokenized - whynlp 在 ModelScope 开源的数据集。Shuffled WuDao corpus tokenized with ChatGLM3 tokenizer.

whynlp/WuDaoCorpus-200G-ChatGLM-tokenized 是 ModelScope 魔搭社区上的数据集，存储大小 63 GB，采用 CC-BY-NC-SA-4.0 许可。

- **Repository**: whynlp/WuDaoCorpus-200G-ChatGLM-tokenized
- **License**: CC-BY-NC-SA-4.0
- **Storage size**: 63 GB
- **Downloads**: 1621
- **Stars**: 1
- **Last updated**: 2024-03-19

Source: https://www.modelscope.cn/datasets/whynlp/WuDaoCorpus-200G-ChatGLM-tokenized

---

数据集文件元信息以及数据文件，请浏览“数据集文件”页面获取。

当前数据集卡片使用的是默认模版，数据集的贡献者未提供更加详细的数据集介绍，但是您可以通过如下GIT Clone命令，或者ModelScope SDK来下载数据集

#### 下载方法 
:modelscope-code[]{type="sdk"}
:modelscope-code[]{type="git"}



About 63G disk space is required to store the dataset.

### Who will be interested?
This dataset is for pre-training large language models. We tokenized the shuffled WuDao corpus with ChatGLM3 tokenizer (`THUDM/chatglm3-6b`) and concatenated the tokenized sentences into a single tensor. Users can determine the block size and split size to fit their own needs.

### Load dataset
In `load.py`, change the split size in `get_length`, set proper `block_size`. Then it will produce an Iterable Dataset that has the expected size for pre-training.

```python
from load import get_datasets
path_to_dataset = "." # Change to the path where the dataset is stored
dataset = get_datasets(path_to_dataset, block_size=2048)
```

### Data Structure
Each data point contains the following:
- `input_ids`: The input token ids. Size: (block_size, )
- `attention_mask`: The attention mask, guaranteed to be all ones. Size: (block_size, )
- `labels`: The same as `input_ids`. Size: (block_size, )

This data structure is compatible with `transformers` library. Users can directly pass it to the huggingface `Trainer` for pre-training.


### Number of Tokens
Total number of tokens: 48625290841