---
title: biodiversity_heritage_library
canonical_url: "https://www.modelscope.cn/datasets/common-pile/biodiversity_heritage_library"
md_url: "https://www.modelscope.cn/datasets/common-pile/biodiversity_heritage_library.md"
repository: common-pile/biodiversity_heritage_library
last_updated: 2025-06-11
license: "Apache License 2.0"
storage_size: "37 GB"
downloads: 482
stars: 0
---

# biodiversity_heritage_library

> biodiversity_heritage_library - common-pile 在 ModelScope 开源的数据集。Biodiversity Heritage Library

common-pile/biodiversity_heritage_library 是 ModelScope 魔搭社区上的数据集，存储大小 37 GB，采用 Apache License 2.0 许可。

- **Repository**: common-pile/biodiversity_heritage_library
- **License**: Apache License 2.0
- **Storage size**: 37 GB
- **Downloads**: 482
- **Stars**: 0
- **Last updated**: 2025-06-11

Source: https://www.modelscope.cn/datasets/common-pile/biodiversity_heritage_library

---

# Biodiversity Heritage Library

## Description 
The [Biodiversity Heritage Library](https://www.biodiversitylibrary.org) (BHL) is an open-access digital library for biodiversity literature and archives. 
This dataset contains over 42 million public domain books and documents from the BHL collection. 
These works were collected using the [bulk data download interface](https://about.biodiversitylibrary.org/tools-and-services/developer-and-data-tools/#Data%20Exports) provided by the BHL and were filtered based on their associated license metadata. 
We use the optical character recognition (OCR)-generated text distributed by BHL.
Code for collecting, processing, and preparing this dataset is available in the [common-pile GitHub repo](https://github.com/r-three/common-pile).

## Dataset Statistics
| Documents | UTF-8 GB |
|-----------|----------|
| 42,418,498 | 96 |

## License Issues
While we aim to produce datasets with completely accurate licensing information, license laundering and inaccurate metadata can cause us to erroneously assign the incorrect license to some documents (for further discussion of this limitation, please see [our paper](https://huggingface.co/papers/2506.05209)). If you believe you have found an instance of incorrect licensing in this dataset, please [start a discussion](https://github.com/r-three/common-pile/discussions/new) on this repository. 

## Other Versions
This is the "raw" version of the Biodiversity Heritage Library dataset. 
If you are looking for the filtered version used to train [Comma v0.1](https://huggingface.co/common-pile/comma-v0.1), you can find it [here](https://huggingface.co/datasets/common-pile/biodiversity_heritage_library_filtered).

## Citation
If you use this dataset, please cite:
```bibtex
@article{kandpal2025common,
  title={{The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text}},
  author={Nikhil Kandpal and Brian Lester and Colin Raffel and Sebastian Majstorovic and Stella Biderman and Baber Abbasi and Luca Soldaini and Enrico Shippole and A. Feder Cooper and Aviya Skowron and Shayne Longpre and Lintang Sutawika and Alon Albalak and Zhenlin Xu and Guilherme Penedo and Loubna Ben  and Elie Bakouch and John David  and Honglu Fan and Dashiell Stander and Guangyu Song and Aaron Gokaslan and John Kirchenbauer and Tom Goldstein and Brian R and Bhavya Kailkhura and Tyler Murray},
  journal={arXiv preprint},
  year={2025}
}
```