---
title: ether0-benchmark
canonical_url: "https://www.modelscope.cn/datasets/futurehouse/ether0-benchmark"
md_url: "https://www.modelscope.cn/datasets/futurehouse/ether0-benchmark.md"
repository: futurehouse/ether0-benchmark
last_updated: 2025-06-24
license: "Apache License 2.0"
storage_size: "451 KB"
downloads: 567
stars: 0
---

# ether0-benchmark

> ether0-benchmark - futurehouse 在 ModelScope 开源的数据集。QA benchmark (test set) for the ether0 reasoning language model: https://huggingface.co/futurehouse/ether0

futurehouse/ether0-benchmark 是 ModelScope 魔搭社区上的数据集，存储大小 451 KB，采用 Apache License 2.0 许可。

- **Repository**: futurehouse/ether0-benchmark
- **License**: Apache License 2.0
- **Storage size**: 451 KB
- **Downloads**: 567
- **Stars**: 0
- **Last updated**: 2025-06-24

Source: https://www.modelscope.cn/datasets/futurehouse/ether0-benchmark

---

[![Model on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/model-on-hf-md-dark.svg)](https://huggingface.co/futurehouse/ether0)

![ether0 logo](images/ether0_logo.svg)

# ether0-benchmark

QA benchmark (test set) for the ether0 reasoning language model:
https://huggingface.co/futurehouse/ether0

This benchmark is made from commonly used tasks - like reaction prediction in USPTO/ORD,
molecular captioning from PubChem, or predicting GHS classification.
It's unique from other benchmarks in that all answers are a molecule.
It's balanced so that each task is about 25 questions,
a reasonable amount for frontier model evaluations.
The tasks generally follow previously reported numbers -
e.g., a reaction prediction accuracy of 80% here would be approximately equivalent
to performance on a withheld split of the USPTO-50k dataset.

See our preprint [here](https://arxiv.org/abs/2506.17238) for more details on dataset construction and reward functions.
The tasks in this test set include:

- Completing SMILES fragments
- Designing molecules adhering to molecular formula and functional group constraints
- Predicting reaction outcomes
- Proposing one-step synthesis pathways
- Editing the solubility of a molecule
- Converting IUPAC name to SMILES
- Answering multiple-choice questions about safety, ADME properties, BBB permeability, toxicity, scent, and pKa

We have measured the performance of ether0 and several frontier LLMs (at time of writing) on this benchmark:

![ether0 benchmarking](images/ether0_benchmark.png)

## Licensing

This dataset repository is CC BY 4.0, copyright 2025 FutureHouse.

## Code

Reward functions and problem templates used for this dataset can be found at
https://github.com/Future-House/ether0.