---
title: AgentJudgeBench
canonical_url: "https://www.modelscope.cn/datasets/ServiceNow-AI/AgentJudgeBench"
md_url: "https://www.modelscope.cn/datasets/ServiceNow-AI/AgentJudgeBench.md"
repository: ServiceNow-AI/AgentJudgeBench
last_updated: 2026-10-10
license: apache-2.0
storage_size: "1.4 GB"
downloads: 327
stars: 0
---

# AgentJudgeBench

> AgentJudgeBench - ServiceNow-AI 在 ModelScope 开源的数据集。AgentJudgeBench: Evaluating LLM Judge Reliability on Agentic Tool-Calling

ServiceNow-AI/AgentJudgeBench 是 ModelScope 魔搭社区上的数据集，存储大小 1.4 GB，采用 apache-2.0 许可。

- **Repository**: ServiceNow-AI/AgentJudgeBench
- **License**: apache-2.0
- **Storage size**: 1.4 GB
- **Downloads**: 327
- **Stars**: 0
- **Last updated**: 2026-10-10

Source: https://www.modelscope.cn/datasets/ServiceNow-AI/AgentJudgeBench

---

<div align="center">

<h1>
  <img src="banner.png" alt="AgentJudgeBench Logo" width="48" style="vertical-align:middle; margin-right:8px;" />
  AgentJudgeBench: Evaluating LLM Judge Reliability on Agentic Tool-Calling
</h1>

<p>
  <a href="https://arxiv.org/abs/2608.26623">
    <img src="https://img.shields.io/badge/Paper-red?logo=arxiv&logoColor=white" />
  </a>
  <a href="https://github.com/ServiceNow/SyGra/tree/scratch/agent_judge_bench/tasks/agentic_bfcl_judge_eval">
    <img src="https://img.shields.io/badge/GitHub-black?logo=github" />
  </a>
  <a href="https://huggingface.co/datasets/ServiceNow-AI/AgentJudgeBench">
    <img src="https://img.shields.io/badge/Dataset-yellow?logo=huggingface&logoColor=black" />
  </a>
</p>

<p>
  <i>
    A benchmark for systematically evaluating how reliably LLM judges assess
    agentic tool-calling workflows across structured, dependency-driven tasks.
  </i>
</p>

</div>

<div align="center">
  <img src="banner.png" alt="AgentJudgeBench Overview" width="90%" />
</div>

## Why this benchmark?
AgentJudgeBench measures how reliably LLM judges assess agentic tool-calling outputs. It provides 3,808 benchmark records spanning six DAG topologies and three difficulty tiers, paired with five generator models' outputs, deterministic programmatic evaluation scores, and verdicts from six general-purpose LLM judges plus one judge-specialised baseline (Prometheus-2) — enabling systematic study of judge alignment, failure modes, and the effect of ground-truth availability.

AgentJudgeBench fills this gap with four design principles:

| Feature | Description |
|---|---|
| **Per-metric decomposition** | Four metrics: tool selection, parameter structure, sequence accuracy, query coverage |
| **Difficulty tiers** | Each record has easy / medium / hard variants via controlled query rewriting |
| **GT ablation** | Paired with-GT and without-GT conditions isolate the effect of showing the judge a reference |
| **Deterministic reference** | A programmatic scorer provides a bias-free ground truth for alignment measurement |

---

## Dataset Structure

### `main` config — 3,808 records

Each row is a unique benchmark record with all three difficulty variants of the user query.

| Column | Type | Description |
|---|---|---|
| `id` | string | Unique record identifier |
| `dag_type` | string | Workflow topology: `linear`, `fan_out`, `fan_in`, `diamond`, `optional_enrichment`, `loop_like` |
| `query_easy` | string | User query — explicit, unambiguous |
| `query_medium` | string | User query — partially implicit |
| `query_hard` | string | User query — fully implicit, high ambiguity |
| `system_prompt` | string | System prompt for the generator model |
| `available_tools` | list[dict] | JSON schemas of all available tools (8–19 per record) |
| `expected_tool_calls` | list[dict] | Ground-truth ordered tool-call sequence |

### Per-generator configs — `llama_3_3_70b_instruct` · `llama_3_1_8b_instruct` · `qwen3_32b` · `smollm3_3b` · `gpt5_4`

Each row is a (record × difficulty) pair with the generator's predicted tool calls and programmatic scores.

| Column | Type | Description |
|---|---|---|
| `id` | string | Record identifier (joins to `main`) |
| `dag_type` | string | Workflow topology |
| `difficulty` | string | `easy`, `medium`, or `hard` |
| `user_query` | string | The query for this difficulty tier |
| `available_tools` | list[dict] | Tool schemas |
| `generated_tool_calls` | list[dict] | Model's predicted tool-call sequence |
| `expected_tool_calls` | list[dict] | Ground-truth tool-call sequence |
| `score_tool_selection` | float | Programmatic score ∈ {0, 0.5, 1} |
| `score_parameter_structure` | float | Programmatic score ∈ {0, 0.5, 1} |
| `score_sequence_accuracy` | float | Programmatic score ∈ {0, 0.5, 1} |
| `score_query_coverage` | float | Programmatic score ∈ {0, 0.5, 1} |
| `overall_programmatic_score` | float | Mean of four metric scores ∈ [0, 1] |

### Judge-verdict configs — `judge_gpt_5_4` · `judge_claude_large` · `judge_gemini_2_5_pro` · `judge_qwq_32b` · `judge_gpt_oss_20b` · `judge_gpt_oss_120b` · `judge_prometheus2_7b`

Each row is a (record × generator × difficulty) triple with this judge's verdict under both the with-GT and without-GT prompt conditions. Join on `id` + `generator` + `difficulty` against the corresponding per-generator config to recover the query, tools, and predicted tool calls being judged.

| Column | Type | Description |
|---|---|---|
| `id` | string | Record identifier |
| `generator` | string | Which generator's output was judged (`llama_3_3_70b_instruct`, `llama_3_1_8b_instruct`, `qwen3_32b`, `smollm3_3b`, or `gpt_5_4`) |
| `difficulty` | string | `easy`, `medium`, or `hard` |
| `dag_type` | string | Workflow topology |
| `with_gt` | dict | Verdict when the judge is shown the ground-truth tool-call sequence |
| `without_gt` | dict | Verdict when the judge sees only the query and tool schemas |

Each of `with_gt` / `without_gt` contains one block per metric (`tool_selection`, `parameter_structure`, `sequence_accuracy`, `query_coverage`), each with `llm_score` (the judge's verdict, ∈ {0, 0.5, 1}), `justification` (the judge's one-sentence rationale), `programmatic_score` (the reference value), and `match_score` (1 − |llm_score − programmatic_score|); plus a top-level `overall_alignment_pct`.

`judge_prometheus2_7b` follows the fine-tuned model's own absolute-grading protocol (one call per metric) rather than the shared single-call JSON schema used by the other six judges, but is normalised into the identical row schema above.

---

## Dataset Statistics

### Records by DAG topology

| DAG Type | Records | % |
|---|---|---|
| Linear | 812 | 21.3% |
| Fan-Out | 399 | 10.5% |
| Fan-In | 1,049 | 27.5% |
| Diamond | 501 | 13.2% |
| Optional Enrichment | 823 | 21.6% |
| Loop-Like | 224 | 5.9% |
| **Total** | **3,808** | **100%** |

### Structural properties

| Metric | Min | Max | Mean | Median |
|---|---|---|---|---|
| Available tools per record | 8 | 19 | 12.9 | 13 |
| Parameters per tool | 0 | 19 | 2.4 | 2 |
| Expected tool calls per record | 2 | 5 | 3.3 | 3 |

Records span **15 enterprise domains** including IT Service Management, Contract Lifecycle Management, and Energy Grid Operations. Each record contains 5–14 tools with typed JSON schemas, typed arguments, and verified ground-truth execution traces.

---

## DAG Topologies

The six topologies capture progressively complex dependency structures found in real enterprise agentic workflows:

| Topology | Description |
|---|---|
| **Linear** | Sequential chain — output of each step feeds the next |
| **Fan-Out** | One call triggers multiple independent parallel branches |
| **Fan-In** | Multiple branches converge into a single aggregation step |
| **Diamond** | Fan-out followed by fan-in — parallel execution with merge |
| **Optional Enrichment** | Core path with optional conditional branch |
| **Loop-Like** | Iterative or conditional repeat pattern |

---

## Difficulty Tiers

Each record has three query variants that preserve the ground-truth tool-call sequence while increasing query ambiguity:

- **Easy** — explicit parameter values, clear tool intents, direct phrasing
- **Medium** — some values implicit or paraphrased, requires inference
- **Hard** — fully implicit, high linguistic ambiguity, requires contextual reasoning

Three frontier LLM meta-judges (Claude Sonnet 4.5, GPT-5.4, Gemini-2.5-Pro) independently validated the rewrites over a 198-triplet stratified sample. Ground-truth tool-call preservation was unanimous on 94.9% of medium rewrites and 76.8% of hard rewrites, with zero unanimous rejections on either. On the complementary "strictly harder than predecessor" criterion, unanimity was 58.1% for medium-vs-easy and 93.9% for hard-vs-medium — the latter is the paper's primary difficulty-degradation evidence; medium should be read as a robustness check rather than a fully independent, calibrated tier. See the paper's Appendix (Rewriting-Preservation Validation Study) for the full per-criterion breakdown.

---

## Evaluation Metrics

The programmatic judge scores four independent dimensions per record:

| Metric | What it measures |
|---|---|
| **Tool Selection** | Are the right tools present? No extraneous tools? |
| **Parameter Structure** | Are required argument keys present and correctly structured? |
| **Sequence Accuracy** | Are calls ordered to respect task dependencies? |
| **Query Coverage** | Does the plan address all parts of the user request? |

Each metric returns a score in {0, 0.5, 1.0}. The `overall_programmatic_score` / `overall_alignment_pct` is the unweighted mean across all four.

---

## LLM Judges

| Judge | Role | Notes |
|---|---|---|
| GPT-5.4 | General-purpose, frontier | Also used as a generator; non-reproducible Azure preview snapshot |
| Claude Sonnet 4.5 | General-purpose, frontier | |
| Gemini-2.5-Pro | General-purpose, frontier | |
| QwQ-32B | General-purpose, open-weight (reasoning) | `enable_thinking=true` |
| GPT-OSS-20B | General-purpose, open-weight | |
| GPT-OSS-120B | General-purpose, open-weight | |
| Prometheus-2 (7B) | Judge-specialised, open-weight | Baseline; does not cluster with the six general-purpose judges (20–30pp lower alignment) — see paper Limitations |

---

## Usage

```python
from datasets import load_dataset

# Load benchmark records (queries + ground truth)
ds = load_dataset("ServiceNow-AI/AgentJudgeBench", "main")

# Load Llama-3.3-70B generator outputs with programmatic scores
ds_gen = load_dataset("ServiceNow-AI/AgentJudgeBench", "llama_3_3_70b_instruct")

# Load GPT-OSS-120B's judge verdicts across all generators
ds_judge = load_dataset("ServiceNow-AI/AgentJudgeBench", "judge_gpt_oss_120b")

# Example: filter to hard records where the generator scored below 0.5
hard_failures = ds_gen["train"].filter(
    lambda r: r["difficulty"] == "hard" and r["overall_programmatic_score"] < 0.5
)

# Example: compare a judge's with-GT vs without-GT verdict on the same records
sample = ds_judge["train"][0]
print(sample["with_gt"]["tool_selection"]["llm_score"], sample["without_gt"]["tool_selection"]["llm_score"])
```

---

## Citation

```bibtex
@inproceedings{agentjudgebench2026,
  title     = {{AgentJudgeBench}: A Multi-Difficulty Benchmark for Evaluating {LLM} Judges on Agentic Tool-Calling},
  author    = {Verma, Abhigya and Saha, Amit Kumar and Subramanian, Seganrasan and Aluru, Sai Harshitha},
  year      = {2026},
}
```

Built with [SyGra](https://github.com/ServiceNow/SyGra), an open-source graph-oriented synthetic-data-generation framework.

---

## License

Apache 2.0