---
title: starcoderdata
canonical_url: "https://www.modelscope.cn/datasets/AI-ModelScope/starcoderdata"
md_url: "https://www.modelscope.cn/datasets/AI-ModelScope/starcoderdata.md"
repository: AI-ModelScope/starcoderdata
chinese_name: starcoderdata
last_updated: 2023-12-07
license: "Apache License 2.0"
storage_size: "289 GB"
domain:
  - multilinguality
  - task_categories
  - pretty_name
tasks:
  - multilingual
  - text-generation
  - The-Stack
downloads: 78431
stars: 5
---

# starcoderdata

> starcoderdata - AI-ModelScope 在 ModelScope 开源的数据集。StarCoder Training Dataset

AI-ModelScope/starcoderdata 是 ModelScope 魔搭社区上的multilingual、text-generation、The-Stack数据集，涉及 multilinguality、task_categories、pretty_name 领域，存储大小 289 GB，采用 Apache License 2.0 许可。

- **Repository**: AI-ModelScope/starcoderdata
- **License**: Apache License 2.0
- **Tasks**: multilingual, text-generation, The-Stack
- **Domain**: multilinguality, task_categories, pretty_name
- **Storage size**: 289 GB
- **Downloads**: 78431
- **Stars**: 5
- **Last updated**: 2023-12-07

Source: https://www.modelscope.cn/datasets/AI-ModelScope/starcoderdata

---

# StarCoder Training Dataset

## Dataset description
This is the dataset used for training [StarCoder](https://huggingface.co/bigcode/starcoder) and [StarCoderBase](https://huggingface.co/bigcode/starcoderbase). It contains 783GB of code in 86 programming languages, and includes 54GB GitHub Issues + 13GB Jupyter notebooks in scripts and text-code pairs,
and 32GB of GitHub commits, which is approximately 250 Billion tokens.

## Dataset creation
The creation and filtering of The Stack is explained in the [original dataset](https://huggingface.co/datasets/bigcode/the-stack-dedup), we additionally decontaminate and clean all 86 programming
languages in the dataset, in addition to GitHub issues, Jupyter Notebooks and GitHub commits. We also apply near-deduplication and remove PII, all details are mentionned in our [Paper: 💫 StarCoder, May The Source Be With You](https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view)

## How to use the dataset
```python
from datasets import load_dataset

# to load python for example
ds = load_dataset("bigcode/starcoderdata", data_dir="python", split="train")
```

GitHub issues, GitHub commits and Jupyter notebooks subsets have different columns from the rest so loading the entire dataset at once may fail, we suggest loading programming languages separatly from these categories. 
````
jupyter-scripts-dedup-filtered
jupyter-structured-clean-dedup
github-issues-filtered-structured
git-commits-cleaned
````
