---
title: raw_v.01_parquet
canonical_url: "https://www.modelscope.cn/datasets/common-pile/raw_v.01_parquet"
md_url: "https://www.modelscope.cn/datasets/common-pile/raw_v.01_parquet.md"
repository: common-pile/raw_v.01_parquet
last_updated: 2025-07-18
license: "Apache License 2.0"
storage_size: "1.8 TB"
downloads: 302
stars: 0
---

# raw_v.01_parquet

> raw_v.01_parquet - common-pile 在 ModelScope 开源的数据集。Common Pile v0.1 — Parquet Consolidated

common-pile/raw_v.01_parquet 是 ModelScope 魔搭社区上的数据集，存储大小 1.8 TB，采用 Apache License 2.0 许可。

- **Repository**: common-pile/raw_v.01_parquet
- **License**: Apache License 2.0
- **Storage size**: 1.8 TB
- **Downloads**: 302
- **Stars**: 0
- **Last updated**: 2025-07-18

Source: https://www.modelscope.cn/datasets/common-pile/raw_v.01_parquet

---

# Common Pile v0.1 — Parquet Consolidated

## Description
This dataset bundles **all “raw” corpora from the Common Pile v0.1 Raw Data [collection](https://huggingface.co/collections/common-pile/common-pile-v01-raw-data-6826b454a5a6a445d0b51b37)**, converted to Apache Parquet and consolidated in a single repository.

Nothing has been filtered or modified; the only changes are:
* **Format:** original JSON → Parquet  
* **Layout:** many repositories → one consolidated dataset  
* **Extra column:** a `len_category` bucket for quick length-based filtering

Only the three original columns (`id`, `text`, `source`) are carried over; `len_category` is derived from `text` length.

## Dataset Schema

| Column | Type   | Notes                                                         |
|--------|--------|---------------------------------------------------------------|
| `id`           | string | Original document ID               |
| `text`         | string | UTF-8 plain text                                      |
| `source`       | string | Name of the originating corpus (e.g. `library_of_congress`) |
| `len_category` | string | Bucketed document length (bytes)|


## License Issues
Licensing follows the individual corpora.

While we aim to produce datasets with completely accurate licensing information, license laundering and inaccurate metadata can cause us to erroneously assign the incorrect license to some documents (for further discussion of this limitation, please see [our paper](https://huggingface.co/papers/2506.05209)). If you believe you have found an instance of incorrect licensing in this dataset, please [start a discussion](https://github.com/r-three/common-pile/discussions/new) on this repository. 


## Other Versions
* [Per-corpus “raw” datasets](https://huggingface.co/collections/common-pile/common-pile-v01-raw-data-6826b454a5a6a445d0b51b37)  
* [Filtered set used for Comma v0.1 training](https://huggingface.co/datasets/common-pile/common-pile-filtered)

## Citation
If you use this dataset, please cite:

```bibtex
@article{kandpal2025common,
  title       = {{The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text}},
  author      = {Nikhil Kandpal and Brian Lester and Colin Raffel and Sebastian Majstorovic and
                 Stella Biderman and Baber Abbasi and Luca Soldaini and Enrico Shippole and
                 A. Feder Cooper and Aviya Skowron and Shayne Longpre and Lintang Sutawika and
                 Alon Albalak and Zhenlin Xu and Guilherme Penedo and Loubna Ben and Elie Bakouch and
                 John David and Honglu Fan and Dashiell Stander and Guangyu Song and Aaron Gokaslan and
                 John Kirchenbauer and Tom Goldstein and Brian R and Bhavya Kailkhura and Tyler Murray},
  journal     = {arXiv preprint},
  year        = {2025}
}