---
title: FannOrFlop-MCQ_verified
canonical_url: "https://www.modelscope.cn/datasets/IFM/FannOrFlop-MCQ_verified"
md_url: "https://www.modelscope.cn/datasets/IFM/FannOrFlop-MCQ_verified.md"
repository: IFM/FannOrFlop-MCQ_verified
last_updated: 2026-10-16
license: apache-2.0
storage_size: "6.8 MB"
downloads: 47
stars: 0
---

# FannOrFlop-MCQ_verified

> FannOrFlop-MCQ_verified - IFM 在 ModelScope 开源的数据集。Fann or Flop: Arabic poetry multiple-choice tasks (verified)

IFM/FannOrFlop-MCQ_verified 是 ModelScope 魔搭社区上的数据集，存储大小 6.8 MB，采用 apache-2.0 许可。

- **Repository**: IFM/FannOrFlop-MCQ_verified
- **License**: apache-2.0
- **Storage size**: 6.8 MB
- **Downloads**: 47
- **Stars**: 0
- **Last updated**: 2026-10-16

Source: https://www.modelscope.cn/datasets/IFM/FannOrFlop-MCQ_verified

---

# Fann or Flop: Arabic poetry multiple-choice tasks (verified)

A filtered version of multiple-choice questions on Arabic poetry **derived from the poems of the Fann or Flop benchmark**, with samples flagged by a multi-judge audit removed. There is one config per task (14 configs): `poem_text__genre`, `poem_text__meter`, `poem_text__poet_name`, `poet_name__genre`, and so on, where the names before and after `__` describe what is given in the question and what is asked.

> **What "verified" means.** Samples were removed when all three LLM judges flagged a significant defect. This is not a human review, and only samples that baseline models answered incorrectly were audited, so some defects may remain in the kept data.

## Source
- The poems and their metadata (poet, era, genre, meter) come from [`omkarthawakar/FannOrFlop`](https://huggingface.co/datasets/omkarthawakar/FannOrFlop) (Apache-2.0), revision `874b3f82d066e786369158a8ca3967585712486e` (6,984 poems with explanations).
- The upstream dataset contains **no questions, options or answer keys**. The multiple-choice questions here (6,984 rows across the 14 tasks, the `50_verses` version, which uses up to 50 verses of a poem) were generated from those poems for evaluation, and the generation script with its distractors is not available. For 5,719 of the 6,984 questions the verses shown were located in the upstream poems, and the answers checked against the upstream genre, meter, era and poet metadata (about 2,960 answers checked, no disagreement). Questions that give only a poet's name have no verses to locate.
- This is therefore a **derived dataset**, not the original Fann or Flop release.

## Columns
`question` (prompt text including the options), `options` (option letters) and `correct_answer` (gold letter). Tasks that include a poem show its verses in the question.

## Audit and filtering

Samples were screened with the multi-judge audit described in [`IFM/discarded_samples`](https://huggingface.co/datasets/IFM/discarded_samples) (revision `3a3c1339`, config `FannOrFlop_MCQ`). Three language-model judges (GPT-5.6 Sol, Gemini 3.8 Flash, Claude Opus 5) assessed candidate samples that strong baselines answered incorrectly. A sample is excluded only when **all three** judges report a *significant* issue (a problem with the question, the options or the gold label that prevents reliable answering or scoring). Excluded samples were matched to this benchmark by the row index (`doc_id`) assigned by lm-evaluation-harness, which equals the 0-based row index within that task's question file in the evaluation copy (the 14 configs here; the upstream poem file has no questions); every matched row was additionally checked against the discarded record's question text and gold label. All other rows are kept in their original order, with the column names and values of the evaluation copy.

| Config | Original rows | Excluded | Kept |
|---|---:|---:|---:|
| `genre__poem_text__poet_name__meter` | 200 | 1 | 199 |
| `poem_text__genre` | 550 | 36 | 514 |
| `poem_text__keywords` | 621 | 2 | 619 |
| `poem_text__meter` | 745 | 1 | 744 |
| `poem_text__poem_title` | 876 | 0 | 876 |
| `poem_text__poet_era` | 638 | 4 | 634 |
| `poem_text__poet_name` | 1042 | 0 | 1042 |
| `poem_text__poet_name__genre` | 321 | 28 | 293 |
| `poem_text__poet_name__meter` | 284 | 1 | 283 |
| `poem_text__poet_name__poet_era` | 225 | 12 | 213 |
| `poem_text__poet_name__rhyme` | 226 | 7 | 219 |
| `poet_name__genre` | 416 | 102 | 314 |
| `poet_name__meter` | 364 | 12 | 352 |
| `poet_name__poet_era` | 476 | 24 | 452 |
| **Total** | **6984** | **230** | **6754** |

## License
The poems come from an Apache-2.0 dataset, which allows derived works with attribution. Please keep the attribution below.

## Citation
Please cite the original Fann or Flop authors:

```bibtex
@misc{alghallabi2025fannflopmultigenremultiera,
  title         = {Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs},
  author        = {Wafa Alghallabi and Ritesh Thawkar and Sara Ghaboura and Ketan More and Omkar Thawakar and Hisham Cholakkal and Salman Khan and Rao Muhammad Anwer},
  year          = {2025},
  eprint        = {2505.18152},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2505.18152}
}

@misc{jais3_2026,
  title  = {Jais 3: Introducing Arabic-centric Agentic AI},
  author = {{IFM Team, Inception42 Team, and Cerebras Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/jais3/},
}
```