---
title: hotpotqa-full-wiki-refine-12k
canonical_url: "https://www.modelscope.cn/datasets/twinkle-kit/hotpotqa-full-wiki-refine-12k"
md_url: "https://www.modelscope.cn/datasets/twinkle-kit/hotpotqa-full-wiki-refine-12k.md"
repository: twinkle-kit/hotpotqa-full-wiki-refine-12k
chinese_name: "HotPotQA fullwiki重优化数据集"
last_updated: 2026-05-19
license: "Apache License 2.0"
downloads: 15
stars: 1
---

# hotpotqa-full-wiki-refine-12k

> hotpotqa-full-wiki-refine-12k - twinkle-kit 在 ModelScope 开源的数据集。HotpotQA Fullwiki — Re-annotated Ground Truth (12K)

twinkle-kit/hotpotqa-full-wiki-refine-12k 是 ModelScope 魔搭社区上的数据集，采用 Apache License 2.0 许可。

- **Repository**: twinkle-kit/hotpotqa-full-wiki-refine-12k
- **License**: Apache License 2.0
- **Downloads**: 15
- **Stars**: 1
- **Last updated**: 2026-05-19

Source: https://www.modelscope.cn/datasets/twinkle-kit/hotpotqa-full-wiki-refine-12k

---

# HotpotQA Fullwiki — Re-annotated Ground Truth (12K)

A super-LLM re-annotated overlay of the **HotpotQA fullwiki train split**,
fixing the well-known ground-truth issues that plague the original release
(wrong question word, partial / single-form answers, presupposition
mismatch, unrepairable rows). Each line is a JSON object that supersedes
the original GT for a single HotpotQA `id`, with an explicit `verdict`
that downstream pipelines filter by.

Used as a **label-fix overlay** in the Twinkle RL stack to clean rewards
for [`cookbook/rl/grpo_condensed.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/grpo_condensed.py)
and as the audited GT source for the cold-start SFT builder
([`cookbook/rl/make_condensed_sft.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/make_condensed_sft.py)).

## Why re-annotate

Auditing original HotpotQA against its own supporting passages reveals
four recurring failure modes:

- **GT type mismatch** — question asks "where" but GT is a person's name.
- **Partial / incomplete** — multi-hop GT collapses to one of the two facts.
- **Single surface form** — `"2"` only, when both `"2"` and `"two"` are valid.
- **Malformed question** — wrong question word, truncated, or
  presupposition-mismatched against the answer type.

These all silently corrupt F1-based RL rewards. This dataset records a
strict-JSON verdict per row so callers can either keep / repair / drop
each example without re-running the audit.

## What is in the dataset

Each line is a JSON object. Core fields:

| Field | Type | Description |
| --- | --- | --- |
| `id` | str | Original HotpotQA row id (matches upstream). |
| `verdict` | str | One of `keep` / `fix_answer` / `fix_question` / `drop`. |
| `question` | str | Original HotpotQA question (verbatim). |
| `question_fixed` | str \| null | Repaired question when `verdict == 'fix_question'`; `null` otherwise. |
| `original_answer` | str | Original single-form GT, kept for trace/debug. |
| `answers` | list[str] | Multi-form gold answer list (empty for `drop`). |
| `reasoning` | str | One-sentence audit rationale. |
| `level`, `type` | str | Original difficulty / question type. |
| `supporting_facts` | dict | Original HotpotQA supporting-facts structure (titles + sentence ids). |
| `context` | dict | Original HotpotQA context (titles + sentences). |

### Verdict semantics

| Verdict | Meaning | `answers` | `question_fixed` |
| --- | --- | --- | --- |
| `keep` | Original Q + A both correct. | multi-form expansion of original A | `null` |
| `fix_answer` | Q is fine; A was wrong / incomplete. | corrected multi-form list | `null` |
| `fix_question` | Q is malformed but repairable into a Q the SAME passages answer with the SAME gold facts. | multi-form list | repaired Q |
| `drop` | Q cannot be repaired without changing the fact, OR passages don't support any answer. | `[]` | `null` |

### Multi-form answer rules

The auditor expands each accepted GT to all valid surface forms before
emitting `answers`:

- **Number variants** — `"3"`, `"three"`, `"three-door"`, `"3-door"`.
- **Range variants** — `"1901"`, `"1902"`, `"1901-1902"`, `"1901-2"`.
- **Location variants** — `"Everett"`, `"Washington"`, `"WA"`, `"United States"`.
- **Person variants** — legal name / nickname / full name.
- **Entity-role pairs** — both the role and the entity for role-of-X questions.
- **Show-vs-character pairs** — both the show and the character for
  best-known-for questions.
- **Abbreviations** — `"NYC"` / `"New York City"` / `"New York"`.
- **Title variants** — `"Dr. Smith"` / `"Smith"`.
- **Date formats** — `"July 4, 1776"` / `"4 July 1776"`.

Every answer is **short** (a name, entity, number, date, or yes/no) and
**grounded** in the supporting passages — no hallucinated forms.

## How it was built

Generated by
[`cookbook/rl/reannotate_groundtruth.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/reannotate_groundtruth.py)
against `hotpotqa/hotpot_qa:fullwiki:train`:

1. **Sampling** — stratified per-level sampling (`easy` / `medium` /
   `hard`) to the requested totals, with a `wrong_ids.txt` allow-list of
   known-bad rows force-included.
2. **Audit prompt** — each row is sent to a super-LLM (`qwen-max` or
   equivalent) together with its question, full context passages, and
   original GT. The system prompt locks the auditor to the four-verdict
   taxonomy and the multi-form / question-rewrite rules above.
3. **Strict-JSON parsing** — the response is parsed as a single JSON
   object; markdown fences are stripped, and a regex fallback recovers
   loose responses. Up to 3 retries per row on parse / API failure.
4. **Schema validation** — `verdict` must be in
   `{keep, fix_answer, fix_question, drop}`; `drop` rows must carry
   empty `answers` and `null` `question_fixed`; `fix_question` rows must
   carry a non-empty `question_fixed` distinct from the original.
5. **Resume-friendly write** — output is appended to a single JSONL; on
   re-run, already-written ids are skipped automatically.

The result is a **12K-row overlay** stratified across HotpotQA levels,
each row stamped with a verdict and a multi-form gold list grounded in
its own passages.

## Usage

### As a label-fix overlay (RL training)

```python
import json

overrides: dict = {}
drop_ids: set = set()
with open('hotpotqa_fullwiki_reannotated_12k.jsonl', 'r', encoding='utf-8') as fh:
    for line in fh:
        obj = json.loads(line)
        rid = obj['id']
        if obj['verdict'] == 'drop':
            drop_ids.add(rid)
        else:
            overrides[rid] = obj

# Filter dropped rows; overlay fixed question / multi-form answers on the rest.
def apply_overlay(row):
    if row['id'] in drop_ids:
        return None
    ov = overrides.get(row['id'])
    if ov is None:
        return {**row, 'answers': [(row.get('answer') or '').strip()]}
    return {
        **row,
        'question': (ov.get('question_fixed') or row['question']),
        'answers': ov['answers'],
    }
```

### As an audited GT source (SFT data builder)

The cold-start SFT builder
[`cookbook/rl/make_condensed_sft.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/make_condensed_sft.py)
consumes this overlay directly: `drop` rows are skipped, `fix_*` rows
have their question / answers replaced before condensation and oracle
rollout.

### Download

```python
from modelscope import MsDataset
ds = MsDataset.load('twinkle-kit/hotpotqa-full-wiki-refine-12k', split='train')
```

## Reproducing

```bash
python cookbook/rl/reannotate_groundtruth.py \
    --output hotpotqa_fullwiki_reannotated_12k.jsonl \
    --model qwen-max \
    --api-key $OPENAI_API_KEY \
    --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
    --total 12000 --easy 2000 --medium 4000 --hard 6000 \
    --concurrency 16 --seed 42
```

For the smaller "fix the known-bad cases only" run:

```bash
python cookbook/rl/reannotate_groundtruth.py \
    --output hotpotqa_reannotated_wrong.jsonl \
    --model qwen-max \
    --only-forced --wrong-ids cookbook/rl/wrong_ids.txt \
    --concurrency 16
```

## License

Apache License 2.0. The underlying HotpotQA passages and original
questions remain under their original CC BY-SA 4.0 license; this
dataset only adds an audit overlay (verdict + multi-form answers +
optional repaired question).