---
title: GneissWeb.Quality_annotator
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/GneissWeb.Quality_annotator"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/GneissWeb.Quality_annotator.md"
repository: AI-ModelScope/GneissWeb.Quality_annotator
last_updated: 2025-02-15
license: apache-2.0
library_name:
  - pytorch
frameworks:
  - pytorch
downloads: 29
stars: 0
---

# GneissWeb.Quality_annotator

> GneissWeb.Quality_annotator - AI-ModelScope 在 ModelScope 开源的模型。Model Summary This fastText model is used as part of the ensemble filter in GneissWeb to detect and remove low-quality documents.

AI-ModelScope/GneissWeb.Quality_annotator 是 ModelScope 魔搭社区上的机器学习模型，采用 apache-2.0 许可。

- **Repository**: AI-ModelScope/GneissWeb.Quality_annotator
- **License**: apache-2.0
- **Downloads**: 29
- **Stars**: 0
- **Last updated**: 2025-02-15

Source: https://www.modelscope.cn/models/AI-ModelScope/GneissWeb.Quality_annotator

---

**Model Summary** 
 
This fastText model is used as part of the ensemble filter in GneissWeb to detect and remove low-quality documents. 

Please refer to the [GneissWeb](https://huggingface.co/datasets/ibm-granite/GneissWeb) dataset page for more details.

- **Developers**: IBM Research
- **Release Date**: Feb 10th, 2025
- **License**: Apache 2.0

**Training Data**

The model is trained on 400k documents, equality split between positive (i.e., high-quality) and negative (i.e., low-quality) classes. Please refer to [fasttext text classification tutorial](https://fasttext.cc/docs/en/python-module.html) for details. 
Training data is selected as follows. 

- *Positive documents*: 190k synthetic documents randomly sampled from the [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) dataset, and
  10k documents with high educational value selected as follows: first, 600k random documents from [FineWeb-V1.1.0](https://huggingface.co/datasets/HuggingFaceFW/fineweb)
  are annotated asking Mixtral-8x22B-Instruct to score each document between 1 to 5  for its educational quality (with 5 being the highest quality), using a prompt similar to the one used by [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).
  Then, 10k random documents are selected from documents with scores greater than or equal to 4.
- *Negative documents*: 200k random documents out of the 600k Mixtral-annotated documents with scores less than or equal to 2.