--- title: embeddings-pre-training canonical_url: "https://www.modelscope.cn/datasets/lightonai/embeddings-pre-training" md_url: "https://www.modelscope.cn/datasets/lightonai/embeddings-pre-training.md" repository: lightonai/embeddings-pre-training last_updated: 2026-10-15 license: "Apache License 2.0" storage_size: "1.9 TB" downloads: 2390 stars: 1 --- # embeddings-pre-training > embeddings-pre-training - lightonai 在 ModelScope 开源的数据集。This large-scale dataset is designed for pre-training state-of-the-art text embedding models. Its goal is to reproduce and build upon the data recipe described in the mGTE technical report (Zhang et… lightonai/embeddings-pre-training 是 ModelScope 魔搭社区上的数据集，存储大小 1.9 TB，采用 Apache License 2.0 许可。 - **Repository**: lightonai/embeddings-pre-training - **License**: Apache License 2.0 - **Storage size**: 1.9 TB - **Downloads**: 2390 - **Stars**: 1 - **Last updated**: 2026-10-15 Source: https://www.modelscope.cn/datasets/lightonai/embeddings-pre-training --- ## Overview This large-scale dataset is designed for pre-training state-of-the-art text embedding models. Its goal is to reproduce and build upon the data recipe described in the [mGTE technical report](https://arxiv.org/abs/2407.19669) (Zhang et al., 2024), which details the data sources used to train the GTE family of embedding models but does not release the data itself. We assembled this dataset as part of a research effort to understand how data composition affects retrieval model quality. Our experiments confirmed that this type of mixture produces significantly stronger models than alternatives (see our [blog post](https://huggingface.co/blog/lightonai/lateon) for details). ## Licensing **Each source in this dataset retains its original license.** We do not relicense any data. Users are responsible for verifying that their intended use complies with the license terms of each individual source. The "Source" column in the table below links to the original dataset, where license information can be found. *** ## Dataset Structure The dataset includes the following columns: * `query`: The input text. * `document`: The corresponding document text. * `index`: A unique identifier for each row. * `drop`: A boolean value indicating whether a row should be excluded during pre-training. * `duplicate`: If not `null`, this contains the `index` of a row with a duplicate query and document. If a row has multiple duplicate, the min(indexes duplicates) should be used. *** ## Recommended Usage For optimal model pre-training, it is recommended to use the subset of data where the `drop` column is `False` and the `duplicate` column is `null`. The complete dataset, including rows marked for dropping and duplication, is provided to allow for the incremental improvement and analysis of the data cleaning process. Work in progress. ```sql SELECT index, query, document FROM lightonai/embeddings-pre-training WHERE NOT DROP AND DUPLICATE IS NULL ``` Each dataset is a distinct configuration within `lightonai/embeddings-pre-training`. To load a specific dataset you will need to specify the configuration and the split: ```python from datasets import load_dataset dataset load_dataset( "lightonai/embeddings-pre-training", "wikihow", split="train", ) ``` | Dataset | MGTE Training | Language | Source | | :---------------------------------- | :-----------: | :------------ | :----- | | agnews | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/agnews) | | altlex | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/altlex) | | amazon_qa | ✅ | English | [nomic](https://huggingface.co/datasets/nomic-ai/nomic-embed-unsupervised-data) | | amazon_reviews | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/amazon-reviews) | | arxiv_title_abstract | ✅ | English | [universetdb](https://huggingface.co/datasets/UniverseTBD/arxiv-abstracts-large) | | beir_dbpedia | ✅ | English | [beir](https://huggingface.co/datasets/BeIR/dbpedia-entity) | | biorxiv_title_abstract | ✅ | English | [laion](https://huggingface.co/datasets/laion/biorXiv_metadata) | | cnn_dailymail | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/embedding-training-data) | | codesearchnet | | English | [st](https://huggingface.co/datasets/sentence-transformers/codesearchnet) | | msmarco | ✅ | English | [microsoft](https://huggingface.co/datasets/microsoft/ms_marco) | | mtp | ✅ | English | [mgte](https://arxiv.org/pdf/2407.19669) | | cc_news_fr | ✅ | French | [intfloat](https://huggingface.co/datasets/intfloat/multilingual_cc_news) | | cc_news_en | ✅ | English | [nomic](https://huggingface.co/datasets/nomic-ai/nomic-embed-unsupervised-data) | | eli5 | | English | [st](https://huggingface.co/datasets/sentence-transformers/eli5) | | gooaq_qa | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/embedding-training-data) | | hermes | | English | [teknium](https://huggingface.co/datasets/teknium/OpenHermes-2.5) | | medrxiv_title_abstract | ✅ | English | [mteb](https://huggingface.co/datasets/mteb/raw_medrxiv) | | nllb_eng_fra | | Cross lingual | [allenai](https://huggingface.co/datasets/allenai/nllb) | | npr | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/npr) | | paq | | English | [st](https://huggingface.co/datasets/sentence-transformers/paq) | | quora | ✅ | English | [nomic](https://huggingface.co/datasets/nomic-ai/nomic-embed-unsupervised-data) | | reddit | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/reddit) | | reddit_body_comment | ✅ | English | [hf](https://huggingface.co/datasets/HuggingFaceGECLM/REDDIT_submissions), [pushshift](https://huggingface.co/datasets/fddemarco/pushshift-reddit-comments) | | s2orc_abstract_citation | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/s2orc) | | s2orc_citation_titles | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/s2orc) | | s2orc_title_abstract | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/s2orc) | | sentence_compression | | English | [st](https://huggingface.co/datasets/sentence-transformers/sentence-compression) | | simplewiki | | English | [st](https://huggingface.co/datasets/sentence-transformers/simple-wiki) | | stackexchange_body_body | | English | [st](https://huggingface.co/datasets/sentence-transformers/stackexchange-duplicates) | | stackexchange_duplicate_questions | | English | [st](https://huggingface.co/datasets/sentence-transformers/stackexchange-duplicates) | | stackexchange_qa | ✅ | English | [flax](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_title_best_voted_answer_jsonl) | | stackexchange_title_body | ✅ | English | [flax](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_title_best_voted_answer_jsonl) | | stackoverflow_title_body | ✅ | English | [flax](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_title_best_voted_answer_jsonl) | | webfaq_eng | | English | [padas-lab](https://huggingface.co/datasets/PaDaS-Lab/webfaq) | | webfaq_fra | | French | [padas-lab](https://huggingface.co/datasets/PaDaS-Lab/webfaq) | | wikihow | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/embedding-training-data) | | wikianswers | ✅ | English | [nomic](https://huggingface.co/datasets/nomic-ai/nomic-embed-unsupervised-data) | | wikianswers | ✅ | English | [nomic](https://huggingface.co/datasets/nomic-ai/nomic-embed-unsupervised-data) | | wikipedia_hlp_cm | ✅ | English | [facebookresearch/atlas](https://github.com/facebookresearch/atlas/tree/main) | | wikipedia_hlp_dl | ✅ | English | [facebookresearch/atlas](https://github.com/facebookresearch/atlas/tree/main) | | wikipedia_en_mgte | ✅ | English | [facebookresearch/atlas](https://github.com/facebookresearch/atlas/tree/main) | | wikipedia | | English | [wikimedia](https://huggingface.co/datasets/wikimedia/wikipedia) | | yahoo_answer | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/embedding-training-data) | | yahoo_qa | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/yahoo-answers/viewer/title-answer-pair) | | yahoo_question_body | ✅ | English | [st](https://huggingface.co/datasets/sentence-transformers/embedding-training-data) | | wikipedia_hlp | ✅ | English | [HLP](https://github.com/jzhoubu/HLP) (dl_10m.jsonl/cm_10m.jsonl) | ### FineWeb Data The following subsets are query/document pairs derived from [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2), processed through the [retrieval-common-crawl](https://github.com/orionweller/retrieval-common-crawl) pipeline by [Orion Weller](https://huggingface.co/orionweller). The intermediate processed version is available at [`orionweller/contrastive-pretraining`](https://huggingface.co/datasets/orionweller/contrastive-pretraining). | Dataset | Language | Source | | :---------------------------------- | :------------ | :----- | | fw-edu | English | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) | | fw2-arb_Arab | Arabic | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-ces_Latn | Czech | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-cmn_Hani | Chinese | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-dan_Latn | Danish | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-deu_Latn | German | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-ell_Grek | Greek | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-fas_Arab | Persian | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-fra_Latn | French | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-hun_Latn | Hungarian | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-ind_Latn | Indonesian | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-ita_Latn | Italian | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-jpn_Jpan | Japanese | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-nld_Latn | Dutch | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-pol_Latn | Polish | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-por_Latn | Portuguese | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-rus_Cyrl | Russian | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-spa_Latn | Spanish | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-swe_Latn | Swedish | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-tur_Latn | Turkish | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | | fw2-vie_Latn | Vietnamese | [orionweller/contrastive-pretraining](https://huggingface.co/datasets/orionweller/contrastive-pretraining) (from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb2)) | --- ## Citation If you use this dataset, please cite: ```bibtex @misc{sourty2025denseonlateon, title={DenseOn with LateOn: Open State-of-the-Art Single and Multi-Vector Models}, author={Sourty, Raphael and Chaffin, Antoine and Weller, Orion and Demoura, Paulo and Chatelain, Amélie}, year={2026}, howpublished={\url{https://huggingface.co/blog/lightonai/denseon-lateon}}, } ```