---
title: ArGuard-Task1
canonical_url: "https://www.modelscope.cn/datasets/QCRI/ArGuard-Task1"
md_url: "https://www.modelscope.cn/datasets/QCRI/ArGuard-Task1.md"
repository: QCRI/ArGuard-Task1
last_updated: 2026-08-11
license: cc-by-nc-4.0
storage_size: "332 MB"
downloads: 276
stars: 0
---

# ArGuard-Task1

> ArGuard-Task1 - QCRI 在 ModelScope 开源的数据集。ArGuard – Track A: Arabic Hateful Memes

QCRI/ArGuard-Task1 是 ModelScope 魔搭社区上的数据集，存储大小 332 MB，采用 cc-by-nc-4.0 许可。

- **Repository**: QCRI/ArGuard-Task1
- **License**: cc-by-nc-4.0
- **Storage size**: 332 MB
- **Downloads**: 276
- **Stars**: 0
- **Last updated**: 2026-08-11

Source: https://www.modelscope.cn/datasets/QCRI/ArGuard-Task1

---

# ArGuard – Track A: Arabic Hateful Memes

This repository hosts the official dataset for **Track A** of the
**ArGuard** shared task: multimodal hateful-meme detection in Arabic.
Each instance is an Arabic meme (image + OCR-extracted overlaid text)
manually annotated for hatefulness and fine-grained sub-types.

> **Content warning.** The dataset contains text and imagery that is
> offensive, discriminatory, or otherwise harmful by design. Handle
> with care.

## Track A subtasks

Given a meme (image + Arabic text):

- **Subtask A1 – Binary.** Classify the meme as `Hateful` or `Not Hateful`.
- **Subtask A2 – Fine-grained category prediction (multi-label).** Predict
  the applicable fine-grained sub-type(s) from a unified taxonomy that
  covers both hateful and non-hateful categories. Hateful memes draw
  labels from the hateful sub-type set (Mocking, Incitement,
  Dehumanization, Slurs, Contempt, Inferiority, Exclusion, …); non-hateful
  memes draw from `Humor`, `Sarcasm`, plus the shared `Other`. Both
  subtasks are evaluated on every meme.

## Splits

| Split        | Records | Labels       | Source                                  | Released                        |
|--------------|---------|--------------|-----------------------------------------|---------------------------------|
| `train`      |  3,500  | full         | single-annotated bulk                   | development phase               |
| `dev`        |    500  | full         | single-annotated bulk                   | development phase               |
| `dev_test`   |    500  | full         | single-annotated test sample            | development phase (leaderboard); **labels released** |
| `test`       |    500  | full         | **triple-annotated gold** (calibration) | final-evaluation phase; **labels released** |

- `dev_test` was the **leaderboard set** for the development phase. That
  phase is now closed and its gold labels have been **released**, so
  `dev_test` is usable as additional labelled data. It is disjoint from
  `test`.
- `test` was the **blind test set** for the final-evaluation phase. That
  phase has now closed and the gold labels are **released**, so the full
  dataset is labelled and the benchmark can be evaluated offline. All 500
  records are triple-annotated with majority voting, which makes `test`
  the highest-quality split and the one to report results on.

### Binary label distribution

| Split      | Hateful | Not Hateful | % Hateful |
|------------|--------:|------------:|----------:|
| train      |  1,324  |       2,176 |     37.8% |
| dev        |      189 |         311 |     37.8% |
| dev_test   |      189 |         311 |     37.8% |
| test       |      148 |         352 |     29.6% |
| **Total**  | **1,850** |   **3,150** |     37.0% |

Note that `test` is deliberately **not** distribution-matched to the
other splits: it was drawn from the triple-annotated calibration pool,
where majority voting resolves more borderline cases to `Not Hateful`.

### Fine-grained sub-types (Subtask A2)

The Subtask A2 label space is **one unified multi-label vocabulary**
that covers both hateful and non-hateful sub-types:

- **Hateful sub-types** (active in the released data): Mocking,
  Incitement, Dehumanization, Slurs, Contempt, Inferiority, Exclusion.
- **Non-hateful sub-types**: Humor, Sarcasm.
- **Shared**: Other (used by both Hateful and Not-Hateful memes).

Five additional hateful classes appear in the annotation taxonomy but
have **zero training support** in the released data: Extremism,
Historical, Insults, Stereotyping, Threat. They are documented for
completeness, accepted by the format checker, and ignored by the scorer.

In practice each meme's fine-grained labels are drawn from its own
binary class: a Hateful meme will only carry hateful sub-types (and/or
`Other`); a Not-Hateful meme will only carry `Humor` / `Sarcasm` /
`Other`. Sub-types are multi-label, so per-class counts sum to more
than the meme counts.

## Record schema

```python
{
    "id": "f9a8…b1.jpg",            # str – original image filename, unique
    "image": <PIL.Image.Image>,     # embedded bytes, decoded on access
    "text": "…",                    # str – OCR-extracted Arabic meme text
    "label": "Hateful" | "Not Hateful",  # every split is now labelled
    "fine_grained_label": [...],         # list[str], possibly empty
}
```

## Usage

```python
from datasets import load_dataset

ds = load_dataset("QCRI/ArGuard-Task1")
print(ds)

train_ex = ds["train"][0]
train_ex["image"].show()
print(train_ex["text"], train_ex["label"], train_ex["fine_grained_label"])

test_ex = ds["test"][0]
print(test_ex["label"], test_ex["fine_grained_label"])
```

**Recommended benchmark protocol.** Train on `train` + `dev` + `dev_test`
(4,500 memes, single-annotator labels) and report on `test` (500 memes,
triple-annotated majority vote). That is the split the shared-task
leaderboard was scored on, so results are comparable to the published
rankings.

## Shared-task resources

- **Website:** https://araieval.github.io/ArGuard2026/
- **Starter kit / baselines / scorers (GitHub):** https://github.com/araieval/ArGuard-2026-tasks
- **Submission — Subtask A1:** https://www.codabench.org/competitions/16909/
- **Submission — Subtask A2:** https://www.codabench.org/competitions/16910/
- **Contact organisers:** arguard2026-organizers@googlegroups.com

## Annotation

- All memes are manually annotated following the ArGuard guidelines.
- **train**, **dev**, **dev_test**: single-annotator labels (bulk
  annotation).
- **test**: triple-annotated. Binary label is the majority vote; the
  fine-grained label set is the union of sub-types selected by
  annotators whose binary label matches the majority.
- Inter-annotator agreement on the calibration subset is above 0.81.

## Intended use and limitations

- **Intended use.** Research on Arabic multimodal hate speech detection,
  including binary classification, fine-grained sub-type prediction,
  and vision-language modelling.
- **Limitations.** Memes reflect online discourse and contain offensive
  and harmful content. Annotations on `train` / `dev` / `dev_test` are
  single-annotator and may contain noise; only the held-out `test` split
  uses triple-annotated majority-voted labels.
- **Not for deployment.** This dataset is for research and benchmarking;
  it is not a moderation tool.

## License

Released under **CC BY-NC 4.0** for non-commercial research use only.
Not to be used for commercial purposes or for training systems that
generate harmful content.

## Citation

A citation will be provided when the shared-task overview paper is
released. Until then, please cite this repository URL.

## Contact

- **Email:** arguard2026-organizers@googlegroups.com
- **Website:** https://araieval.github.io/ArGuard2026/
- **GitHub:** https://github.com/araieval/ArGuard-2026-tasks