---
title: JevOCRBench
canonical_url: "https://www.modelscope.cn/datasets/OpenBMB/JevOCRBench"
md_url: "https://www.modelscope.cn/datasets/OpenBMB/JevOCRBench.md"
repository: OpenBMB/JevOCRBench
last_updated: 2026-11-01
license: other
storage_size: "1.3 GB"
downloads: 27005
stars: 0
---

# JevOCRBench

> JevOCRBench - OpenBMB 在 ModelScope 开源的数据集。JevOCRBench is a benchmark for OCR correctness decisions. It tests whether a decision model can tell whether an OCR result is correct. JevOCRBench contains 2,267 Chinese and English records covering text, formulas,…

OpenBMB/JevOCRBench 是 ModelScope 魔搭社区上的数据集，存储大小 1.3 GB，采用 other 许可。

- **Repository**: OpenBMB/JevOCRBench
- **License**: other
- **Storage size**: 1.3 GB
- **Downloads**: 27005
- **Stars**: 0
- **Last updated**: 2026-11-01

Source: https://www.modelscope.cn/datasets/OpenBMB/JevOCRBench

---

# JevOCRBench

[中文](README.zh-CN.md) · [Data terms](DATA_LICENSE.md)

JevOCRBench is a benchmark for **OCR correctness decisions**. It tests whether a decision model can tell whether an OCR result is correct. JevOCRBench contains **2,267** Chinese and English records covering text, formulas, and tables. Each record mainly contains the source image, the OCR candidate text, a rendering of that candidate text, and a human-reviewed correctness label. Negatives come from real error outputs of frontier OCR models.

## Data information

### Distribution statistics

![Category, language, and source distribution](assets/distribution-en.png)

### Data construction

Images come from three public benchmarks of mainstream OCR models: [OmniDocBench v1.6](https://huggingface.co/datasets/opendatalab/OmniDocBench), [Real5-OmniDocBench](https://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench), and [olmOCR-bench](https://huggingface.co/datasets/allenai/olmOCR-bench). Candidates were sampled from frontier OCR parsers such as MiniCPM-OCR 1.0 and PaddleOCR-VL-1.6, labeled by humans for correctness, then lightly filtered and balanced by category.

## Download and read

Install and access the dataset with huggingface_hub:

```bash
python -m pip install -U huggingface_hub
```

```python
import json
from pathlib import Path

from huggingface_hub import snapshot_download

root = Path(snapshot_download(
    repo_id="openbmb/JevOCRBench",
    repo_type="dataset",
))

with (root / "data/jevocrbench.jsonl").open(encoding="utf-8") as f:
    sample = json.loads(next(line for line in f if line.strip()))

source_image = root / sample["image_path"]
candidate_render = root / sample["render_image_path"]

assert source_image.is_file()
assert candidate_render.is_file()
print(sample["item_id"], sample["ocr_text"])
```

## Data schema

`data/jevocrbench.jsonl` contains one JSON object per line. Every row has exactly 14 fields in the order below.

| Field | Type | Meaning |
| --- | --- | --- |
| `item_id` | integer | Sequential record ID from 0 to 2,266 and prediction join key. |
| `label` | string | `pos` for a faithful candidate; `neg` for a candidate with a defect. |
| `category` | string | `text`, `formula`, or `table`. |
| `lang` | string | `en` or `zh`. |
| `source` | string | One of 14 source-subset identifiers. |
| `scenario` | string / null | Real5 acquisition condition; null outside Real5. |
| `model` | string | Public display name of the OCR model/checkpoint that produced `ocr_text`. |
| `page_id` | string | Source-local page identifier. |
| `image_path` | string | Source-crop path relative to the repository root. |
| `render_image_path` | string | Candidate-render path relative to the repository root. |
| `ocr_text` | string | OCR candidate being verified. |
| `error_type` | string / null | Primary error category for a classified negative. |
| `error_types_secondary` | array / null | Zero to three additional error categories. |
| `error_evidence` | string / null | Short Chinese description of the observed error. |

Example:

```json
{
  "item_id": 678,
  "label": "neg",
  "category": "text",
  "lang": "zh",
  "source": "omnidocbench_v1.6",
  "scenario": null,
  "model": "PaddleOCR-VL-1.6",
  "page_id": "newspaper_35bdaa11dec104ccbdc9e19ac8527d21_1.jpg",
  "image_path": "images/00654.png",
  "render_image_path": "renders/00679.png",
  "ocr_text": "## 莱集标准化",
  "error_type": "char_misrecognition",
  "error_types_secondary": [],
  "error_evidence": "聚焦二字被错识为莱集"
}
```

All three error fields are null for positives.

Definitions and boundary rules for all 11 error categories are in [`taxonomy/error_types.json`](taxonomy/error_types.json).

## Evaluation

For every record, present these three inputs in order and evaluate them with our [rubic prompt](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/assets/minicpm-ocr/rubic_prompt.py):

1. source document crop;
2. raw OCR candidate text;
3. candidate rendering.

A higher score from the decision model indicates a stronger tendency toward `pos`. All published results use these three inputs. Do not use labels, error types, sources, candidate models, or other metadata as inputs.

![JevOCRBench benchmark results](assets/benchmark-results.png)

Evaluation code:

```python
import json
from sklearn.metrics import roc_auc_score

recs = [json.loads(line) for line in open("data/jevocrbench.jsonl", encoding="utf-8")]  # 2,267 records

# Your verifier: one score per record (illustrative)
# scores = [your_verifier(r["image_path"], r["ocr_text"], r["render_image_path"]) for r in recs]

auroc = roc_auc_score([r["label"] == "pos" for r in recs], scores)
```

## Repository layout

```text
JevOCRBench/
├── data/jevocrbench.jsonl
├── images/                         # 2,188 source crops
├── renders/                        # 2,267 per-record renders
├── viewer/                         # Dataset Viewer Parquet with embedded images
├── taxonomy/error_types.json
├── assets/benchmark-results.png
├── assets/distribution-zh.png
├── assets/distribution-en.png
├── README.md
├── README.zh-CN.md
├── DATA_LICENSE.md
├── LICENSE                         # software only
└── NOTICE
```

## License and source terms

The combined package uses `license: other` because different files and records have different terms:

- JevOCRBench software: Apache License 2.0 under the repository-root [`LICENSE`](LICENSE);
- eligible JevOCRBench-added metadata: Apache-2.0 only to the extent the contributors have authority to license it;
- OmniDocBench-derived materials: upstream research-only, non-commercial terms;
- Real5-derived materials: Real5 declares Apache-2.0, while inherited OmniDocBench annotations, source-page correspondences, and source-document terms remain applicable;
- olmOCR-bench-derived database material: ODC-BY-1.0 with attribution.

Underlying documents may carry additional rights. Read [DATA_LICENSE.md](DATA_LICENSE.md) and [NOTICE](NOTICE); those files and the upstream terms control over this summary.