---
title: SlimPajama-627B-Llama-tokenized
canonical_url: "https://www.modelscope.cn/datasets/whynlp/SlimPajama-627B-Llama-tokenized"
md_url: "https://www.modelscope.cn/datasets/whynlp/SlimPajama-627B-Llama-tokenized.md"
repository: whynlp/SlimPajama-627B-Llama-tokenized
last_updated: 2024-03-19
license: other
storage_size: "794 GB"
downloads: 319432
stars: 1
---

# SlimPajama-627B-Llama-tokenized

> SlimPajama-627B-Llama-tokenized - whynlp 在 ModelScope 开源的数据集。SlimPajama tokenized with Llama tokenizer.

whynlp/SlimPajama-627B-Llama-tokenized 是 ModelScope 魔搭社区上的数据集，存储大小 794 GB，采用 other 许可。

- **Repository**: whynlp/SlimPajama-627B-Llama-tokenized
- **License**: other
- **Storage size**: 794 GB
- **Downloads**: 319432
- **Stars**: 1
- **Last updated**: 2024-03-19

Source: https://www.modelscope.cn/datasets/whynlp/SlimPajama-627B-Llama-tokenized

---

数据集文件元信息以及数据文件，请浏览“数据集文件”页面获取。

当前数据集卡片使用的是默认模版，数据集的贡献者未提供更加详细的数据集介绍，但是您可以通过如下GIT Clone命令，或者ModelScope SDK来下载数据集

#### 下载方法 
:modelscope-code[]{type="sdk"}
:modelscope-code[]{type="git"}



About 800G disk space is required to store the dataset.

### Who will be interested?
This dataset is for pre-training large language models. We tokenized the original dataset and concatenated the tokenized sentences into a single tensor. Users can determine the block size and split size to fit their own needs.

### Load dataset
In `load.py`, change the split size in `get_length`, set proper `block_size`. Then it will produce an Iterable Dataset that has the expected size for pre-training.

```python
from load import get_datasets
path_to_dataset = "." # Change to the path where the dataset is stored
dataset = get_datasets(path_to_dataset, block_size=2048)
```

### Data Structure
Each data point contains the following:
- `input_ids`: The input token ids. Size: (block_size, )
- `attention_mask`: The attention mask, guaranteed to be all ones. Size: (block_size, )
- `labels`: The same as `input_ids`. Size: (block_size, )

This data structure is compatible with `transformers` library. Users can directly pass it to the huggingface `Trainer` for pre-training.


### Number of Tokens
| Split      | Chunk   | Number of Tokens |
| ---------- | ------- | ---------------- |
| train      | chunk1  | 68911783918      |
| train      | chunk2  | 68772696764      |
| train      | chunk3  | 68604762303      |
| train      | chunk4  | 68989066539      |
| train      | chunk5  | 69284076530      |
| train      | chunk6  | 68740396332      |
| train      | chunk7  | 68810004291      |
| train      | chunk8  | 69416021237      |
| train      | chunk9  | 68954916164      |
| train      | chunk10 | 69201529264      |
| validation | chunk1  | 117113469        |
| validation | chunk2  | 114167430        |
| validation | chunk3  | 118725653        |
| validation | chunk4  | 118551208        |
| validation | chunk5  | 117557738        |
| test       | chunk1  | 116135508        |
| test       | chunk2  | 121456829        |
| test       | chunk3  | 117992482        |
| test       | chunk4  | 115073005        |
| test       | chunk5  | 117061864        |