---
title: alyah-emirati-benchmark_verified
canonical_url: "https://www.modelscope.cn/datasets/IFM/alyah-emirati-benchmark_verified"
md_url: "https://www.modelscope.cn/datasets/IFM/alyah-emirati-benchmark_verified.md"
repository: IFM/alyah-emirati-benchmark_verified
last_updated: 2026-10-16
license: cc-by-4.0
storage_size: "66 KB"
downloads: 44
stars: 0
---

# alyah-emirati-benchmark_verified

> alyah-emirati-benchmark_verified - IFM 在 ModelScope 开源的数据集。Alyah Emirati Benchmark (verified)

IFM/alyah-emirati-benchmark_verified 是 ModelScope 魔搭社区上的数据集，存储大小 66 KB，采用 cc-by-4.0 许可。

- **Repository**: IFM/alyah-emirati-benchmark_verified
- **License**: cc-by-4.0
- **Storage size**: 66 KB
- **Downloads**: 44
- **Stars**: 0
- **Last updated**: 2026-10-16

Source: https://www.modelscope.cn/datasets/IFM/alyah-emirati-benchmark_verified

---

# Alyah Emirati Benchmark (verified)

A filtered version of **Alyah**, the Emirati-dialect multiple-choice benchmark from TII, with samples flagged by a multi-judge audit removed.

> **What "verified" means.** Samples were removed when all three LLM judges flagged a significant defect. This is not a human review, and only samples that baseline models answered incorrectly were audited, so some defects may remain in the kept data.

**This repository does not contain the Alyah questions.** The source dataset is public but states no license, so we publish the filter instead: which source rows to keep, plus a script that applies it to the source.

## How to use

```bash
pip install pandas pyarrow huggingface_hub
python apply_filter.py alyah_verified.parquet
```

[`apply_filter.py`](apply_filter.py) downloads the source at the revision below, checks every row against the fingerprints in `ids.csv`, and writes the 1,158-row verified test set. It stops with an error if the source has changed.

Or filter it yourself:

```python
import pandas as pd

ids = pd.read_csv("hf://datasets/IFM/alyah-emirati-benchmark_verified/ids.csv")
source = pd.read_parquet(
    "hf://datasets/tiiuae/alyah-emirati-benchmark@e404e3e7745f3401d3c13eb2e97be36bff898b77/data/test-00000-of-00001.parquet"
)
verified = source[ids["status"].eq("kept").to_numpy()].reset_index(drop=True)  # 1,158 rows
```

## Files
- `ids.csv`: one row per source row (1,173 rows):
  - `row`: the 0-based row index in the source file.
  - `category`: the source `category`.
  - `status`: `kept` (1,158) or `removed` (15).
  - `fingerprint`: the first 16 hex characters of a SHA-256 over the row's `query`, `option_1`–`option_4`, `correct_answer` and `category`, used to detect changes in the source.
- `apply_filter.py`: rebuilds the verified set from the source.

## Source
- Original dataset: [`tiiuae/alyah-emirati-benchmark`](https://huggingface.co/datasets/tiiuae/alyah-emirati-benchmark), revision `e404e3e7745f3401d3c13eb2e97be36bff898b77`, file `data/test-00000-of-00001.parquet` (1,173 rows). The data file is unchanged since the upload commit `134f969e`; later commits only edit the README.
- Columns of the rebuilt set, as in the source: `query`, `option_1`–`option_4`, `correct_answer` (the correct option number, 1–4) and `category`.

## Audit and filtering

Samples were screened with the multi-judge audit described in [`IFM/discarded_samples`](https://huggingface.co/datasets/IFM/discarded_samples) (config `Alyah (Emirati)`). Three language-model judges (GPT-5.6 Sol, Gemini 3.8 Flash, Claude Opus 5) assessed candidate samples that strong baselines answered incorrectly. A sample is excluded only when **all three** judges report a *significant* issue (a problem with the question, the options or the gold label that prevents reliable answering or scoring). Each discarded record's `doc_id` is the 0-based position of the sample within its category in the source file; `ids.csv` gives the matching source row (`row`) for each excluded sample. All other rows are kept unchanged and in their original order.

| Category | Original rows | Excluded | Kept |
|---|---:|---:|---:|
| Language & Dialect | 619 | 10 | 609 |
| Imagery & Figurative Meaning | 121 | 1 | 120 |
| Etiquette & Values | 173 | 0 | 173 |
| Religious & Social Sensitivity | 78 | 0 | 78 |
| Historical & Heritage Knowledge | 89 | 0 | 89 |
| Greetings & Daily Expressions | 61 | 4 | 57 |
| Poetry & Creative Expression | 32 | 0 | 32 |
| **Total** | **1,173** | **15** | **1,158** |

## License
The source dataset states no license, so this repository redistributes none of its text. The files here (`ids.csv`, `apply_filter.py` and this card) are released under CC BY 4.0. The verified set you build from the source remains subject to the source's terms.

## Citation
Please cite the original authors:

```bibtex
@misc{emirati_dialect_benchmark_2026,
    title = {Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models},
    author = {Omar Alkaabi and Ahmed Alzubaidi and Hamza Alobeidli and Shaikha Alsuwaidi and Mohammed Alyafeai and Leen AlQadi and Basma El Amel Boussaha and Hakim Hacid},
    year = {2026},
    month = {january},
    url = {https://huggingface.co/datasets/tiiuae/alyah-emirati-benchmark}
}

@misc{jais3_2026,
  title  = {Jais 3: Introducing Arabic-centric Agentic AI},
  author = {{IFM Team, Inception42 Team, and Cerebras Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/jais3/},
}
```