---
title: Android-World-Eval
canonical_url: "https://www.modelscope.cn/datasets/SeerRay-Lab/Android-World-Eval"
md_url: "https://www.modelscope.cn/datasets/SeerRay-Lab/Android-World-Eval.md"
repository: SeerRay-Lab/Android-World-Eval
last_updated: 2026-09-30
license: mit
downloads: 48
stars: 0
---

# Android-World-Eval

> Android-World-Eval - SeerRay-Lab 在 ModelScope 开源的数据集。Android-World-Eval: RealGUI Evaluation Results

SeerRay-Lab/Android-World-Eval 是 ModelScope 魔搭社区上的数据集，采用 mit 许可。

- **Repository**: SeerRay-Lab/Android-World-Eval
- **License**: mit
- **Downloads**: 48
- **Stars**: 0
- **Last updated**: 2026-09-30

Source: https://www.modelscope.cn/datasets/SeerRay-Lab/Android-World-Eval

---

# Android-World-Eval: RealGUI Evaluation Results

This dataset contains the evaluation results of **Xiaomi-GUI-0** method on the [Android World](https://github.com/google-research/android_world) benchmark.

## Overview

- **Method**: Xiaomi-GUI-0
- **Benchmark**: Android World (116 tasks)
- **Rollouts**: 4 independent evaluation runs per task (464 total samples)

## Results

| Rollout | Samples | Success | Accuracy |
|---------|---------|---------|----------|
| 0       | 116     | 94      | 81.03%   |
| 1       | 116     | 91      | 78.45%   |
| 2       | 116     | 91      | 78.45%   |
| 3       | 116     | 90      | 77.59%   |
| **Overall** | **464** | **366** | **78.88%** |

## Data Format

Each JSON file represents one task rollout with the following structure:

### Top-level Fields

| Field | Description |
|-------|-------------|
| `trace_id` | Unique identifier for this evaluation trace |
| `step` | Training step of the evaluated checkpoint |
| `sample_index` | Task index (0-115) |
| `rollout_n` | Rollout number (0-3) |
| `experiment_name` | Name of the experiment configuration |
| `task` | Human-readable task description |
| `task_name` | Canonical task identifier |
| `initial_prompt` | The instruction given to the agent |
| `start_time` / `end_time` | Timestamps for the evaluation run |
| `total_spans` | Total number of interaction spans |
| `status` | Evaluation status (`completed`) |
| `phase` | Evaluation phase (`validate`) |
| `final_reward` | **1.0 = task completed successfully, 0.0 = task failed** |
| `attributes` | Task metadata including `task_complexity` and `task_max_steps` |

### Turn-level Fields (in `turns` array)

Each turn represents one step of agent-environment interaction:

| Field | Description |
|-------|-------------|
| `turn_index` | Sequential index of this turn |
| `screenshot_base64` | Base64-encoded PNG screenshot of the device screen at this step |
| `llm_response` | Raw LLM output including thought process and action |
| `action` | Parsed action object with `action` type, `arguments`, `thought`, and `answer` |
| `reward` | Intermediate reward signal (0.0 for intermediate steps) |
| `timestamp` | Timestamp of this turn |
| `messages` | Full message history sent to the LLM |

## File Naming Convention

```
sample_{task_index}_rollout_{rollout_number}.json
```

- `task_index`: 0-115 (116 unique Android World tasks)
- `rollout_number`: 0-3 (4 independent runs per task)

## Usage

```python
import json
from huggingface_hub import hf_hub_download

file = hf_hub_download(
    repo_id="SeerRay-Lab/Android-World-Eval",
    filename="sample_0_rollout_0.json",
    repo_type="dataset"
)

with open(file, "r") as f:
    data = json.load(f)

print(f"Task: {data['task']}")
print(f"Success: {data['final_reward'] == 1.0}")
print(f"Steps taken: {len(data['turns'])}")
```

## Citation

If you use this dataset, please cite the RealGUI paper and Android World benchmark.