---
title: Puro-2B
canonical_url: "https://www.modelscope.cn/datasets/thu-pacman/Puro-2B"
md_url: "https://www.modelscope.cn/datasets/thu-pacman/Puro-2B.md"
repository: thu-pacman/Puro-2B
last_updated: 2026-09-01
license: other
storage_size: "677 GB"
downloads: 2014
stars: 1
---
# Puro-2B
> Puro-2B - thu-pacman 在 ModelScope 开源的数据集。Puro-2B Pretraining Data: The Recipe Behind a 2B Model
thu-pacman/Puro-2B 是 ModelScope 魔搭社区上的数据集，存储大小 677 GB，采用 other 许可。
- **Repository**: thu-pacman/Puro-2B
- **License**: other
- **Storage size**: 677 GB
- **Downloads**: 2014
- **Stars**: 1
- **Last updated**: 2026-09-01
Source: https://www.modelscope.cn/datasets/thu-pacman/Puro-2B
---
# Puro-2B Pretraining Data: The Recipe Behind a 2B Model
**This is the materialized pretraining data release for
[Puro-2B-Base](https://huggingface.co/thu-pacman/Puro-2B-Base), a 2B base model
trained from scratch on consumer-grade RTX 5090 GPUs.**
[](https://huggingface.co/thu-pacman/Puro-2B-Base)
[](https://huggingface.co/datasets/thu-pacman/Puro-2B)
[](#license-and-upstream-terms)
[](https://arxiv.org/abs/2608.27370)
The repository contains the component-level data pools used to construct the
two Puro-2B pretraining phases, together with the tokenizer used for token
accounting. It is organized for inspection, selective streaming, and recipe
reconstruction rather than as a small train/test benchmark.
## Dataset Summary
| Domain | Phase 1 | Phase 1 share | Phase 2 | Phase 2 share |
| --- | ---: | ---: | ---: | ---: |
| English | 321B | 73.2% | 558B | 59.4% |
| Mathematics | 31.6B | 7.2% | 171B | 18.3% |
| Chinese | 51.3B | 11.7% | 88.5B | 9.4% |
| Code | 34.6B | 7.9% | 108B | 11.5% |
| SFT / instruction-formatted | 0 | 0.0% | 12.7B | 1.3% |
| **Materialized pool** | **439B** | **100.0%** | **938B** | **100.0%** |
Token counts are computed with the bundled tokenizer and measure materialized
token exposure, not unique upstream content. Dataset-family rows may aggregate
overlapping configurations.
The production model consumed approximately 439B tokens in Phase 1 and 961B
additional tokens in Phase 2. The Phase 2 training count is larger than the
938B stationary Phase 2 pool because the opening transition replays Phase 1
data while introducing the Phase 2 stream. Replay is a training-schedule
operation, not additional independent source data.
## Repository Structure
```text
.
├── phase1/ # Component data used to build the Phase 1 mixture
├── phase2/ # Component data used to build the Phase 2 mixture
├── qwen2_tokenizer/ # Tokenizer files used for token accounting
└── README.md
```
The release defines two Hugging Face configurations, `phase1` and `phase2`.
Each configuration exposes one `train` split because both phases are
pretraining corpora. Directory names such as `train`, `validation`, `test`, or
`benchmark` inside individual source components preserve upstream organization;
they are not Puro-2B evaluation splits.
Files are grouped by source component. Their filesystem order is not the final
curriculum order used by the production run. The transition, replay, curriculum
bucket construction, deterministic seeds, and global uniform reshuffle are
recipe operations documented in the technical report and implemented in the
data-processing pipeline.
## Data Fields
Released Parquet records use a common schema:
| Field | Type | Description |
| --- | --- | --- |
| `text` | string | Normalized text presented to the tokenizer. |
| `token_count` | int64 | Number of tokens under the bundled tokenizer. |
| `domain_category` | string | Semantic domain, such as English, Chinese, math, code, or instruction-formatted data. |
| `source_component_id` | string | Stable identifier for the materialized source component. |
Source-specific metadata not represented by these fields should be recovered
from the upstream dataset and the Puro data-processing manifests.
## Loading the Data
The full repository is hundreds of gigabytes. Streaming or selecting individual
components is recommended for exploration.
```python
from datasets import load_dataset
phase1 = load_dataset(
"thu-pacman/Puro-2B",
"phase1",
split="train",
streaming=True,
)
example = next(iter(phase1))
print(example["source_component_id"], example["token_count"])
print(example["text"][:500])
```
Load Phase 2 by changing the configuration name:
```python
phase2 = load_dataset(
"thu-pacman/Puro-2B",
"phase2",
split="train",
streaming=True,
)
```
For a single component, load its Parquet files directly:
```python
from datasets import load_dataset
open_web_math = load_dataset(
"parquet",
data_files={
"train": (
"hf://datasets/thu-pacman/Puro-2B/"
"phase2/open-web-math/*.parquet"
)
},
split="train",
streaming=True,
)
```
For reproducible experiments, pin the dataset `revision` to a commit hash and
record the selected component paths, filtering rules, and shuffle seed.
## How the Recipe Was Built
The Puro-2B pipeline separates recipe selection from shard materialization:
1. **Source acquisition.** Candidate English, Chinese, mathematics, code, and
instruction-formatted datasets are collected from public sources.
2. **Within-source processing.** Large web components are deduplicated within
source. The project does not claim global cross-source deduplication.
3. **Proxy benchmarking.** A shared Qwen3-0.6B proxy checkpoint is continued on
candidate sources or score slices and evaluated on a fixed 15-benchmark
suite. These results guide source and within-source selection.
4. **Materialization.** Selected revisions, filters, token budgets, mixture
weights, and random seeds are frozen into component-level Parquet files.
5. **Training order.** Phase 2 uses component-local ranks to build an ordered
curriculum while approximately preserving the mixture. Scores are never
compared numerically across different source datasets.
Because the proxy study has no matched base-mixture-only continuation, its
candidate profiles should be interpreted as recipe signals rather than isolated
causal effects.
## Sources at a Glance
| Domain | Representative source families |
| --- | --- |
| English | Nemotron-CC-v2, FineWeb-Edu, Cosmopedia-v2, DCLM, ArXiv, FineWiki |
| Chinese | FineWeb-Edu-CN, ChineseWebText2.0, merged Chinese web corpora, Baidu Baike, FineWiki-CN, UN documents, Alpaca-Zh |
| Mathematics | UltraData-Math, SwallowMath-v2, Nemotron-CC-Math, MegaMath-Web-Pro, OpenWebMath, FineMath, AutoMathText, NuminaMath-CoT, FineProofs |
| Code | MegaMath-Code, Nemotron Synthetic Code, Swallow-Code-v2, CoderForge, StackExchange, Python and GitHub corpora, Codeforces-CoTs, Jupyter Agent |
| Instruction-formatted | Nemotron Terminal Corpus, JiuZhang3.0, Tulu 3 SFT, ToolMind, ToolBench, SlimOrca, LongAlpaca, OpenThoughts Agent |
See the [technical report source](https://github.com/thu-yao-01-luo/Prom-Technical-Report)
for the component-level token counts, materialization modes, upstream
identifiers, proxy protocol, and license audit.
## License and Upstream Terms
The dataset card uses the **`other`** license designation because the recipe
combines components with different upstream terms. Apache 2.0 applies only to
project-authored documentation, metadata, manifests, and processing artifacts
explicitly released under that license. It does not replace the licenses,
copyrights, privacy rights, or terms attached to third-party source content.
Important boundaries include:
- NVIDIA-controlled sources such as Nemotron-CC-v2 and the Synthetic-Code
partition of Nemotron-Pretraining-Code-v1 have data terms that prohibit raw
redistribution. Their corresponding directories provide reconstruction or
provenance notices instead of republished raw samples where required.
- Web and code corpora may retain terms from Common Crawl, websites, authors,
or original software repositories even when dataset metadata uses a
permissive license.
- Some source cards do not declare a dataset-level license, and some mixtures
contain subsets with stricter terms. Absence of a declared license should not
be interpreted as permission to redistribute or use the content without
restriction.
Users are responsible for checking the upstream terms of the components they
download and for determining whether their intended use is permitted. The
technical report's component inventory is the starting point for that review.
## Known Limitations and Responsible Use
This is large-scale pretraining data collected from heterogeneous public and
synthetic sources. It may contain personal information, copyrighted material,
incorrect claims, duplicated content, offensive language, security-sensitive
code, or social and cultural biases. Filtering reduces some problems but does
not eliminate them.
The release does not claim exhaustive benchmark decontamination. Some source
families provide decontaminated subsets, but that property should not be
generalized to every component. Users should run task-specific contamination,
privacy, safety, and licensing checks before training or deploying a model.
## Related Artifacts
- Model: [thu-pacman/Puro-2B-Base](https://huggingface.co/thu-pacman/Puro-2B-Base)
- Training code: [thu-pacman/Puro-Megatron](https://github.com/thu-pacman/Puro-Megatron)
- Data processing: [thu-pacman/Kaiyuan-Spark](https://github.com/thu-pacman/Kaiyuan-Spark)
- Technical report: pending upload to arXiv.
## Citation
Please cite our technical report if you find our work useful:
```bibtex
@misc{luo2026puro2b,
title={Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090},
author={Kairong Luo and Jiarui Cui and Yaorui Yin and Shengqi Chen and Yiming Yang and Linxiang Gao and Yanmohan Wang and Mingzhe Zhang and Kaiyue Wen and Kaifeng Lyu and Wenguang Chen},
year={2026},
eprint={2608.27370},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2608.27370},
}
```