---
title: middle_training
canonical_url: "https://www.modelscope.cn/datasets/wyhewy/middle_training"
md_url: "https://www.modelscope.cn/datasets/wyhewy/middle_training.md"
repository: wyhewy/middle_training
last_updated: 2026-05-10
license: "Apache License 2.0"
storage_size: "3.7 GB"
downloads: 19
stars: 0
---

# middle_training

> middle_training - wyhewy 在 ModelScope 开源的数据集。This dataset is a globally deduplicated middle-training raw-text subset converted from BAAI IndustryCorpus2. It is intended for continued pretraining / middle training of small language models, especially when the…

wyhewy/middle_training 是 ModelScope 魔搭社区上的数据集，存储大小 3.7 GB，采用 Apache License 2.0 许可。

- **Repository**: wyhewy/middle_training
- **License**: Apache License 2.0
- **Storage size**: 3.7 GB
- **Downloads**: 19
- **Stars**: 0
- **Last updated**: 2026-05-10

Source: https://www.modelscope.cn/datasets/wyhewy/middle_training

---

# middle_training

This dataset is a globally deduplicated middle-training raw-text subset converted from BAAI IndustryCorpus2. It is intended for continued pretraining / middle training of small language models, especially when the goal is to improve general text understanding before downstream instruction tuning.

## Summary

- Final records after global deduplication: **1,021,694**
- LLM-kept source records before global deduplication: **1,077,123**
- Removed duplicates: **55,429**
- Duplicate by source `_id`: **26,013**
- Duplicate by normalized text hash: **29,416**
- Empty text removed: **0**
- Languages: Chinese **837,467** / English **184,227**
- LLM quality buckets: high **471,064** / medium **550,630**

## Source And Pipeline

The source data comes from multiple BAAI IndustryCorpus2 sub-datasets on ModelScope. The processing pipeline was:

1. Sample raw candidates from IndustryCorpus2 parquet shards by industry category and language.
2. Apply metadata and text-quality filters before LLM filtering.
3. Use a local vLLM service with `Qwen3-4B-Instruct-2507` to judge whether each candidate is suitable for 0.6B-scale middle training.
4. Keep only LLM-approved `high` and `medium` quality rows.
5. Run global deduplication across all kept rows using source `_id` and normalized text hash.
6. Export both normalized raw-text files and ms-swift pretraining files.

## Cleaning Rules

The first-stage deterministic cleaning used IndustryCorpus2 metadata and simple text statistics.

Default Chinese rules:

- `perplexity <= 800`
- `quality_score >= 3.5`
- text length between `80` and `900` characters
- `word_rep_ratio <= 0.10`
- `char_rep_ratio <= 0.04`
- `special_char_ratio <= 0.20`
- `alnum_ratio >= 0.78`

Default English rules:

- `quality_score >= 3.5`
- text length between `80` and `2500` characters
- output text truncated to at most `700` characters when configured
- `word_rep_ratio <= 0.15`
- `char_rep_ratio <= 0.10`
- `special_char_ratio <= 0.35`
- `alnum_ratio >= 0.70`
- English perplexity was not used because many English shards do not provide reliable `perplexity`.

Some categories used stricter overrides. For example, Film & Entertainment Chinese data used `perplexity <= 900`, `max_chars <= 800`, `char_rep_ratio <= 0.03`, `special_char_ratio <= 0.18`, and `alnum_ratio >= 0.80`.

## LLM Filtering Rules

The second-stage semantic filter used a local vLLM deployment of `Qwen3-4B-Instruct-2507`.

Rows were kept if they were:

- natural, fluent, and coherent;
- relevant to the target industry category;
- suitable as ordinary raw text for continued pretraining;
- article-like, explanatory, descriptive, review-like, or service-information-like text;
- not overly noisy or specialized for a small model.

Rows were dropped if they were:

- download pages, pirated resource pages, cloud-disk pages, BT/magnet/free-full-version pages;
- job posts, resumes, or job-duty pages unless the category explicitly benefits from employment information;
- overly sensitive gossip, pornographic, violent, suicide-related, or crime-detail text;
- large lists, ads, marketing spam, SEO stitching, garbled text, severe truncation, or heavy repetition;
- clearly off-topic for the industry;
- too specialized or strange and likely to increase perplexity for a 0.6B model.

The LLM returned a structured judgment with `keep`, `quality`, `content_type`, `reason`, and `issue_tags`. Only kept rows were exported.

## Global Deduplication

Deduplication was applied after all category-level LLM filtering:

- First key: stable source ID from `raw_id` or `meta._id`.
- Second key: SHA1 of normalized text.
- Text normalization: collapse whitespace, trim, lowercase, then hash.
- If either source ID or normalized text hash was already seen, the row was removed.

This removed **55,429** duplicated rows from **1,077,123** LLM-kept rows, leaving **1,021,694** final records.

## Data Modules

| Module | Records |
| --- | ---: |
| `raw_text` | 1,021,694 |
| `ms_swift_pretrain` | 1,021,694 |

`raw_text/*.jsonl` contains normalized records with metadata and LLM filter explanations.

`ms_swift_pretrain/*.jsonl` contains ms-swift pretraining format:

```json
{"messages": [{"role": "assistant", "content": "..."}]}
```

## Category Counts

| Category | Slug | Records |
| --- | --- | ---: |
| Subject Education | `subject_education` | 97,620 |
| Finance & Economics | `finance_economics` | 88,902 |
| Health & Medicine | `health_medicine` | 87,891 |
| Transportation | `transportation` | 84,145 |
| Film & Entertainment | `film_entertainment` | 81,443 |
| Travel & Geography | `tourism_geography` | 72,322 |
| Hospitality & Catering | `hospitality_catering` | 59,742 |
| Agriculture & Fisheries | `agriculture_fisheries` | 51,590 |
| Automobiles | `automobile` | 50,633 |
| Sports | `sports` | 50,158 |
| Politics & Administration | `politics_administration` | 45,343 |
| Manufacturing | `manufacturing` | 44,656 |
| Computing & Telecommunications | `computing_telecom` | 44,306 |
| Real Estate & Construction | `real_estate_construction` | 42,647 |
| Media & Journalism | `media_journalism` | 38,715 |
| Law & Justice | `law_judiciary` | 36,124 |
| Safety Management | `safety_management` | 13,554 |
| Information Services | `information_services` | 9,572 |
| Literature & Emotions | `literature_emotions` | 9,271 |
| Water Resources & Marine | `water_marine` | 4,772 |
| Power & Energy | `power_energy` | 4,573 |
| Gaming | `gaming` | 3,715 |

## Files

- `raw_text/raw_text_*.jsonl`: normalized raw text records.
- `ms_swift_pretrain/pretrain_*.jsonl`: ms-swift pretraining records.
- `dataset_info.json`: full counts by language, category, slug, source file, quality, and dedupe statistics.
- `DATA_CLEANING.md`: detailed cleaning and filtering documentation.

## Raw Text Row Schema

Raw text rows contain:

```json
{
  "type": "raw_text",
  "source": "modelscope:BAAI/IndustryCorpus2_xxx",
  "category": "...",
  "industry_type": "...",
  "slug": "...",
  "language": "chinese|english",
  "quality_bucket": "high|middle",
  "llm_quality": "high|medium",
  "content_type": "...",
  "text": "...",
  "meta": {
    "_id": "...",
    "perplexity": 123.4,
    "quality_score": 4.0,
    "num_words": 100,
    "alnum_ratio": 0.9,
    "special_char_ratio": 0.1,
    "char_rep_ratio": 0.0,
    "word_rep_ratio": 0.0
  },
  "filter_reason": "...",
  "issue_tags": []
}
```

## Example ms-swift Usage

```bash
swift sft --dataset /path/to/ms_swift_pretrain/*.jsonl ...
```