---
title: the-pile-freelaw-refined-by-data-juicer
canonical_url: "https://www.modelscope.cn/datasets/Data-Juicer/the-pile-freelaw-refined-by-data-juicer"
md_url: "https://www.modelscope.cn/datasets/Data-Juicer/the-pile-freelaw-refined-by-data-juicer.md"
repository: Data-Juicer/the-pile-freelaw-refined-by-data-juicer
last_updated: 2026-10-26
license: "Apache License 2.0"
storage_size: "1.6 MB"
domain:
  - text
tasks:
  - text-generation
language:
  - en
downloads: 169
stars: 1
---

# the-pile-freelaw-refined-by-data-juicer

> the-pile-freelaw-refined-by-data-juicer - Data-Juicer 在 ModelScope 开源的数据集。A refined version of FreeLaw dataset in The Pile by Data-Juicer. Removing some "bad" samples from the original dataset to make it higher-quality. This dataset is usually used to…

Data-Juicer/the-pile-freelaw-refined-by-data-juicer 是 ModelScope 魔搭社区上的text-generation数据集，涉及 text 领域，存储大小 1.6 MB，采用 Apache License 2.0 许可。

- **Repository**: Data-Juicer/the-pile-freelaw-refined-by-data-juicer
- **License**: Apache License 2.0
- **Tasks**: text-generation
- **Domain**: text
- **Storage size**: 1.6 MB
- **Downloads**: 169
- **Stars**: 1
- **Last updated**: 2026-10-26

Source: https://www.modelscope.cn/datasets/Data-Juicer/the-pile-freelaw-refined-by-data-juicer

---

# The Pile -- FreeLaw (refined by Data-Juicer)

A refined version of FreeLaw dataset in The Pile by [Data-Juicer](https://github.com/alibaba/data-juicer). Removing some "bad" samples from the original dataset to make it higher-quality.

This dataset is usually used to pretrain a Large Language Model.

**Notice**: Here is a small subset for previewing. The whole dataset is available [here](https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/LLM_data/our_refined_datasets/pretraining/the-pile-freelaw-refine-result.jsonl) (About 45GB).

## Dataset Information

- Number of samples: 2,942,612 (Keep ~82.61% from the original dataset)

## Refining Recipe
```yaml
# global parameters
project_name: 'Data-Juicer-recipes-freelaw'
dataset_path: '/path/to/your/dataset'  # path to your dataset directory or file
export_path: '/path/to/your/dataset.jsonl'

np: 50  # number of subprocess to process your dataset
open_tracer: true

# process schedule
# a list of several process operators with their arguments
process:
  - clean_email_mapper:
  - clean_links_mapper:
  - fix_unicode_mapper:
  - punctuation_normalization_mapper:
  - whitespace_normalization_mapper:

  - alphanumeric_filter:
      tokenization: false
      min_ratio: 0.3  # <3sigma (0.436)
  - average_line_length_filter:  # for code
      max_len: 697  # 3sigma TBD
  - character_repetition_filter:
      rep_len: 10
      max_ratio: 0.4  # >3sigma (0.350)
  - flagged_words_filter:
      lang: en
      tokenization: true
      max_ratio: 0.0053  # 3sigma
  - language_id_score_filter:
      min_score: 0.5 # < 3sigma (0.583)
  - maximum_line_length_filter:  # for code
      max_len: 4229  # 3sigma
  - perplexity_filter:
      lang: en
      max_ppl: 5322  # 3sigma
  - special_characters_filter:
      max_ratio: 0.7  # > 3sigma (0.626)
  - stopwords_filter:  # not use
      lang: en
      tokenization: true
      min_ratio: 0.1 # > 3sigma (0.07)
  - text_length_filter:
      max_len: 84026 # 3sigma
  - words_num_filter:
      lang: en
      tokenization: true
      min_num: 100
      max_num: 15208  # 3sigma
  - word_repetition_filter:
      lang: en
      tokenization: true
      rep_len: 10
      max_ratio: 0.155  # 3sigma

  - document_simhash_deduplicator:
      tokenization: space
      window_size: 6
      lowercase: true
      ignore_pattern: '\p{P}'
      num_blocks: 6
      hamming_distance: 4
```
