---
title: FACTORY
canonical_url: "https://www.modelscope.cn/datasets/facebook/FACTORY"
md_url: "https://www.modelscope.cn/datasets/facebook/FACTORY.md"
repository: facebook/FACTORY
last_updated: 2025-08-07
license: cc-by-nc-4.0
storage_size: "21 MB"
downloads: 262
stars: 0
---

# FACTORY

> FACTORY - facebook 在 ModelScope 开源的数据集。FACTORY is a large-scale, human-verified, and challenging prompt set. We employ a model-in-the-loop approach to ensure quality and address the complexities of evaluating long-form generation. Starting with seed topics…

facebook/FACTORY 是 ModelScope 魔搭社区上的数据集，存储大小 21 MB，采用 cc-by-nc-4.0 许可。

- **Repository**: facebook/FACTORY
- **License**: cc-by-nc-4.0
- **Storage size**: 21 MB
- **Downloads**: 262
- **Stars**: 0
- **Last updated**: 2025-08-07

Source: https://www.modelscope.cn/datasets/facebook/FACTORY

---

# Overview

FACTORY is a large-scale, human-verified, and challenging prompt set. We employ a model-in-the-loop approach to ensure quality and address the complexities of evaluating long-form generation. Starting with seed topics from Wikipedia, we expand each topic into a diverse set of prompts using large language models (LLMs). We then apply the model-in-the-loop method to filter out simpler prompts, maintaining a high level of difficulty. Human annotators further refine the prompts to ensure they are fact-seeking, answerable, unambiguous, not time-sensitive, and safe. To push the boundaries of long-form factuality evaluation, we identify a “hard” split of FACTORY that presents significant challenges to current state-of-the-art LLMs, with their outputs containing approximately 40% of claims for which humans cannot find supportive information online.

This dataset is stored in the JSON Lines (.jsonl) format, where each line contains a single JSON object representing one data entry.

# Structure

Each line in the dataset file has the following keys:

- question: (string) — A natural language question requiring a long-form answer.
- url: (string) — One or more URLs to resources that provide relevant information for answering the question.



![image/png](https://cdn-uploads.huggingface.co/production/uploads/66a79b27909a525bcbc0708f/3IQGboL5_1zhqJhDfRzIB.png)

Figure 1. Factual precision as evaluated by human annotators on 100 sentences per model for each benchmark. All the models are retrieval-augmented.



**We have also released the human annotations collected during the evaluation of factual precision, available [here](https://huggingface.co/datasets/facebook/FACTORY/blob/main/fact_checking/human_annotations.jsonl)**

# Structure for the Human Annotations

Each line in the file is a valid JSON object containing the following keys for each annotated claim:

- Claim 1, Claim 2, ..., Claim N:
  The text of each claim.

- Claim 1 Tag, Claim 2 Tag, ..., Claim N Tag:
  Factuality label for the corresponding claim. The label indicates the annotator's assessment of the claim's factuality and can be one of the following:
  + "Factual"
  + "NonFactual"
  + "Inconclusive"
  + "No Verifiable Fact"

- Source Claim 1, Source Claim 2, ..., Source Claim N:
  A string of URLs containing sources or evidence that support/refute the claim. This field may be empty if the claim's tag is "Inconclusive".

- Claim 1 Snippet, Claim 2 Snippet, ..., Claim N Snippet:
  Text snippets copied from the sources above, providing direct evidences for the associated claim and its assigned factuality label.


See our [technical report](https://arxiv.org/abs/2508.00109) for more details
# Reference

```
@article{chen2025factory,
  title={FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality},
  author={Chen, Mingda and Li, Yang and Chen, Xilun and Williams, Adina and Ghosh, Gargi and Yih, Scott},
  journal={arXiv preprint arXiv:2508.00109},
  year={2025}
}
```