---
title: chempile-education
canonical_url: "https://www.modelscope.cn/datasets/jablonkagroup/chempile-education"
md_url: "https://www.modelscope.cn/datasets/jablonkagroup/chempile-education.md"
repository: jablonkagroup/chempile-education
last_updated: 2025-06-23
license: "Apache License 2.0"
storage_size: "247 MB"
downloads: 666
stars: 1
---

# chempile-education

> chempile-education - jablonkagroup 在 ModelScope 开源的数据集。A comprehensive chemistry education dataset containing 129.26M+ tokens from diverse educational sources

jablonkagroup/chempile-education 是 ModelScope 魔搭社区上的数据集，存储大小 247 MB，采用 Apache License 2.0 许可。

- **Repository**: jablonkagroup/chempile-education
- **License**: Apache License 2.0
- **Storage size**: 247 MB
- **Downloads**: 666
- **Stars**: 1
- **Last updated**: 2025-06-23

Source: https://www.modelscope.cn/datasets/jablonkagroup/chempile-education

---

# ChemPile-Education

<div align="center">

![ChemPile Logo](CHEMPILE_LOGO.png)
  
[![Dataset](https://img.shields.io/badge/🤗%20Hugging%20Face-Dataset-yellow)](https://huggingface.co/datasets/jablonkagroup/chempile-education)
[![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
[![Paper](https://img.shields.io/badge/📄-Paper-red)](https://arxiv.org/abs/2505.12534)
[![Website](https://img.shields.io/badge/🌐-Website-green)](https://chempile.lamalab.org/)

*A comprehensive chemistry education dataset containing 129.26M+ tokens from diverse educational sources*

</div>

## 📋 Dataset Summary

ChemPile-Education is a large-scale chemistry natural language dataset extracted from diverse educational resources, including open-source textbooks, lecture notes, transcripts, and other educational materials. This dataset captures fundamental chemistry knowledge and concepts as students would encounter them in textbooks and classroom settings.

### 📊 Dataset Statistics

| Subset | Tokens | Documents | Description |
|--------|--------|-----------|-------------|
| LibreText Chemistry | 114M | 58,9K | Open-source chemistry textbooks |
| MIT OCW Lecture Transcripts | 10M | 58K | University-level lecture transcripts |
| US Olympiad Problems | 260K | 1,22K | Competition chemistry problems with solutions |
| YouTube Transcripts | 5M | 5,09K | Educational chemistry video transcripts |
| **Total** | **~129.26M** | **~123K** | Chemical educational content |

## 🗂️ Dataset Configurations

The dataset includes four distinct subsets available as Hugging Face configurations:

- `LibreText_Chemistry-default`
- `mit-ocw-lecture-transcripts-default` 
- `us-olympiad-problems-default`
- `youtube-transcripts-as-lectures-default`

## 📜 License

All content is released under the **CC BY-NC-SA 4.0** license, which allows for:
- ✅ Non-commercial use
- ✅ Sharing and redistribution
- ✅ Adaptation and modification
- ⚠️ Attribution required
- ⚠️ Share-alike (derivatives must use same license)

## 📖 Dataset Details

### 🧪 LibreText Chemistry

**Source**: [LibreTexts Chemistry](https://chem.libretexts.org/) - A comprehensive open-source chemistry education platform

**Coverage**: General chemistry, organic chemistry, inorganic chemistry, physical chemistry, and biochemistry

**Extraction Method**: Custom web scraper for HTML content extraction

**Fields**:
- `url`: Source URL for transparency and verification
- `text`: Educational content about chemistry concepts

**Statistics**: 114M tokens across 58,946 documents covering undergraduate to graduate-level chemistry topics

### 🎓 MIT OCW Lecture Transcripts

**Source**: [MIT OpenCourseWare](https://ocw.mit.edu/) - Free online course materials from MIT

**Coverage**: Chemistry, biology, chemical engineering, and physics courses

**Extraction Method**: Automated download using keyword-based course identification

**Fields**:
- `course`: Course name and identifier
- `url`: Original source URL for reference
- `topic`: Specific lecture topic
- `text`: Lecture transcript content
- `index`: Document identifier

**Statistics**: 10M tokens from approximately 500 university-level lectures

### 🏆 US Olympiad Problems

**Source**: [American Chemical Society (ACS)](https://www.acs.org/) US Chemistry Olympiad materials

**Coverage**: Competition-level chemistry problems with detailed solutions

**Extraction Method**: PDF processing with LLM-generated Q&A pairs using *Gemini 2.0 Flash Thinking Experimental 01-21*

**Quality Control**: 
- Minimum 250 characters per answer
- Expert manual review by chemistry professionals
- Filtered for educational value

**Fields**:
- `metadata`: Problem details (year, number, topic)
- `problem_statement`: Original olympiad problem
- `options`: Multiple choice options
- `solution`: Detailed solution explanation
- `correct_answer`: Correct option identifier
- `text`: Combined problem and solution content
- `filter`: Quality control flag
- `index`: Document identifier

**Statistics**: 260K+ tokens from ~400 competition problems with comprehensive solutions

### 📺 YouTube Transcripts as Lectures

**Source**: Educational chemistry videos from YouTube with permissive CC licenses

**Coverage**: Diverse chemistry topics from educational content creators

**Extraction Method**: 
- LLM-generated search keywords for educational chemistry content
- Automated transcript extraction from CC-licensed videos
- Content cleaning and structuring using *GPT-4.1*

**Fields**:
- `id`: Unique YouTube video identifier
- `title`: Video title
- `text`: Cleaned and structured transcript content
- `index`: Document identifier

**Statistics**: 5M+ tokens from ~1,000 educational chemistry videos

## 🚀 Quick Start

```python
from datasets import load_dataset, get_dataset_config_names

# List all available configurations
configs = get_dataset_config_names("jablonkagroup/chempile-education")
print(f"Available configs: {configs}")
# ['LibreText_Chemistry-default', 'mit-ocw-lecture-transcripts-default', 
#  'us-olympiad-problems-default', 'youtube-transcripts-as-lectures-default']

# Load a specific subset
dataset = load_dataset("jablonkagroup/chempile-education", name="LibreText_Chemistry-default")

print(dataset)
# DatasetDict({
#     train: Dataset({
#         features: ['url', 'text'],
#         num_rows: 53051
#     })
#     test: Dataset({
#         features: ['url', 'text'], 
#         num_rows: 2948
#     })
#     val: Dataset({
#         features: ['url', 'text'],
#         num_rows: 2947
#     })
# })

# Access a sample
sample = dataset['train'][0]
print(f"Sample URL: {sample['url'][:50]}...")
print(f"Sample text: {sample['text'][:200]}...")
```

## 🎯 Use Cases

- **🤖 Language Model Training**: Pre-training or fine-tuning models for chemistry domain
- **📚 Educational AI**: Building chemistry tutoring and Q&A systems  
- **🔍 Information Retrieval**: Chemistry knowledge base construction
- **📝 Content Generation**: Automated chemistry educational content creation
- **🧠 Domain Adaptation**: Adapting general models to chemistry domain

## ⚠️ Limitations & Considerations

- **Language**: English only (monolingual dataset)
- **Scope**: Focused on educational content; may not cover cutting-edge research and some knowledge might be missing
- **Quality**: Variable quality across sources; some automated transcripts may contain errors
- **Bias**: Reflects biases present in educational materials and sources
- **License**: Non-commercial use only due to CC BY-NC-SA 4.0 license

## 🛠️ Data Processing Pipeline

1. **Collection**: Automated scraping and API-based collection from sources
2. **Extraction**: Text extraction from various formats (HTML, PDF, video transcripts)
3. **Cleaning**: LLM-assisted content cleaning and structuring
4. **Quality Control**: Expert review and filtering
5. **Standardization**: Consistent formatting and metadata addition
6. **Validation**: Split creation and final quality checks
    
## 🏗️ ChemPile Collection

This dataset is part of the **ChemPile** collection, a comprehensive open dataset containing over 75 billion tokens of curated chemical data for training and evaluating general-purpose models in the chemical sciences. 

### Collection Overview
- **📊 Scale**: 75+ billion tokens across multiple modalities
- **🧬 Modalities**: Structured representations (SMILES, SELFIES, IUPAC, InChI), scientific text, executable code, and molecular images
- **🎯 Design**: Integrates foundational educational knowledge with specialized scientific literature
- **🔬 Curation**: Extensive expert curation and validation
- **📈 Benchmarking**: Standardized train/validation/test splits for robust evaluation
- **🌐 Availability**: Openly released via Hugging Face

## 📄 Citation

If you use this dataset in your research, please cite:

```bibtex
@article{mirza2025chempile0,
  title   = {ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models},
  author  = {Adrian Mirza and Nawaf Alampara and Martiño Ríos-García and others},
  year    = {2025},
  journal = {arXiv preprint arXiv:2505.12534}
}
```

## 👥 Contact & Support

- **Paper**: [arXiv:2505.12534](https://arxiv.org/abs/2505.12534)
- **Website**: [ChemPile Project](https://chempile.lamalab.org/)
- **Dataset**: [Hugging Face](https://huggingface.co/datasets/jablonkagroup/chempile-education)
- **Issues**: Please report data issues or questions via the Hugging Face dataset page

---

<div align="center">

![ChemPile Logo](CHEMPILE_LOGO.png)

<i>Part of the ChemPile project - Advancing AI for Chemical Sciences</i>

</div>