---
title: LIBRA
canonical_url: "https://www.modelscope.cn/datasets/ai-forever/LIBRA"
md_url: "https://www.modelscope.cn/datasets/ai-forever/LIBRA.md"
repository: ai-forever/LIBRA
last_updated: 2026-06-12
license: mit
storage_size: "2.0 GB"
downloads: 2745
stars: 0
---

# LIBRA

> LIBRA - ai-forever 在 ModelScope 开源的数据集。LIBRA: Long Input Benchmark for Russian Analysis

ai-forever/LIBRA 是 ModelScope 魔搭社区上的数据集，存储大小 2.0 GB，采用 mit 许可。

- **Repository**: ai-forever/LIBRA
- **License**: mit
- **Storage size**: 2.0 GB
- **Downloads**: 2745
- **Stars**: 0
- **Last updated**: 2026-06-12

Source: https://www.modelscope.cn/datasets/ai-forever/LIBRA

---

# LIBRA: Long Input Benchmark for Russian Analysis
<p align="center">
<img src="logo.png" width="500" />
</p>

## LIBRA

LIBRA (Long Input Benchmark for Russian Analysis) is designed to evaluate the capabilities of large language models (LLMs) in understanding and processing long texts in Russian. This benchmark includes 18 datasets adapted for different tasks and complexities. The tasks are divided into four complexity groups and allow evaluation across various context lengths ranging from 4k up to 512k tokens.

For model comparison and results, see the [LIBRA Leaderboard](https://huggingface.co/spaces/ai-forever/LIBRA-Leaderboard). The benchmark is described in detail in our [paper](https://arxiv.org/abs/2408.02439).

**NOTE:** This is a new benchmark version released in May 2026. The original version (described in the paper) can be found [here](https://huggingface.co/datasets/ai-forever/LIBRA_old). We strongly encourage using the current version as it contains cleaned and extended datasets with ensured data quality.

## LIBRA Mini

Running a full LIBRA evaluation can be prohibitively expensive and time-consuming due to the large number of datasets and long context lengths involved. Moreover, some of the included tasks have become less informative as benchmarks, with modern models achieving near-saturated scores on them.

To address this, we introduce **LIBRA Mini** — a compact, curated subset of 6 datasets selected from the full benchmark. These datasets represent the most challenging and diagnostically informative tasks in LIBRA, covering diverse task types and complexity levels (see [Task Description](#task-description) for more information):

- **ruBABILongQA3** — multi-fact reasoning over long contexts
- **ruSciPassageCount** — counting unique paragraphs in extended scientific texts
- **LibrusecMHQA** — multi-hop QA with information spread across multiple text parts
- **LongContextMultiQ** — multi-hop QA based on Wikidata and Wikipedia
- **ru2WikiMultihopQA** — multi-hop reasoning across multiple Wikipedia articles
- **MatreshkaNames** — identifying persons in dialogues based on discussed topics

LIBRA Mini uses the same **Exact Match (EM)** metric and evaluation methodology as the full benchmark. Results for LIBRA Mini are reported in a dedicated section on the [leaderboard](https://huggingface.co/spaces/ai-forever/LIBRA-Leaderboard).

We recommend **using LIBRA Mini as the primary evaluation suite for model comparisons**, while the full LIBRA benchmark remains available for comprehensive analysis.

## Dataset Structure

<p align="center">
  <img src="libra_structure.svg" width="600" />
</p>

The datasets are divided into subsets based on context lengths. The table below shows the number of examples per context length for each dataset. Datasets included in **LIBRA Mini** are highlighted in bold. Note that not all datasets cover the full range of context lengths — some are designed for specific length ranges that best suit their task type.

| **Task** | **4k** | **8k** | **16k** | **32k** | **64k** | **128k** | **256k** | **512k** | | **Total** |
|:---|---:|---:|---:|---:|---:|---:|---:|---:|---|---:|
| *— Group I —* | | | | | | | | | | |
| passkey | 200 | 200 | 200 | 200 | 200 | 200 | 200 | 200 | | **1600** |
| passkey_with_librusec | 200 | 200 | 200 | 200 | 200 | 200 | 200 | 200 | | **1600** |
| *— Group II —* | | | | | | | | | | |
| librusec_history | - | 32 | 32 | 32 | 32 | - | - | - | | **128** |
| **matreshka_names** | 150 | 150 | 145 | 150 | 45 | - | - | - | | **640** |
| matreshka_yes_no | 300 | 300 | 300 | 300 | 290 | 280 | - | - | | **1770** |
| ru_quality | - | 18 | 184 | - | - | - | - | - | | **202** |
| ru_sci_abstract_retrieval | 209 | 210 | 210 | 206 | 185 | 200 | 200 | - | | **1420** |
| ru_sci_fi | - | - | - | 216 | 213 | - | - | - | | **429** |
| ru_tpo | - | 900 | - | - | - | - | - | - | | **900** |
| *— Group III —* | | | | | | | | | | |
| **librusec_mhqa** | - | 384 | - | - | - | - | - | - | | **384** |
| **long_context_multiq** | 158 | 121 | 83 | 109 | 41 | - | - | - | | **512** |
| **ru_2wikimultihopqa** | - | 147 | 384 | 369 | - | - | - | - | | **900** |
| ru_babilong_qa1 | 99 | 99 | 99 | 94 | 91 | 99 | 200 | 98 | | **879** |
| ru_babilong_qa2 | 85 | 76 | 77 | 73 | 65 | 99 | 200 | 98 | | **773** |
| **ru_babilong_qa3** | 60 | 68 | 69 | 65 | 65 | 100 | 198 | 98 | | **723** |
| ru_babilong_qa4 | 78 | 85 | 83 | 87 | 75 | 99 | 200 | 98 | | **805** |
| ru_babilong_qa5 | 99 | 99 | 98 | 96 | 96 | 99 | 200 | 98 | | **885** |
| *— Group IV —* | | | | | | | | | | |
| **ru_sci_passage_count** | 104 | 99 | 100 | 87 | 91 | 90 | 43 | 60 | | **674** |

## Task Description

The benchmark tasks are organized into four complexity groups, ranging from simple retrieval to complex reasoning. The grouping reflects both the cognitive difficulty of the task and the degree to which models must integrate information across the full context window. Group I serves as a basic sanity check, verifying that a model can process long inputs at all. Groups II and III progressively require deeper language understanding, multi-step reasoning, and the ability to locate and combine information from distant parts of the context. Group IV represents the most demanding tasks, requiring complex reasoning that goes beyond standard question answering formats. The total score on the leaderboard is computed across all four groups.

### Group I: Simple Information Retrieval (sanity check)
This group includes the most simple tasks which serve as a sanity check for models to work with such amount of tokens.
- **Passkey**: Extract a relevant piece of code number from a long text fragment. Based on the original [PassKey test](https://github.com/CStanKonrad/long_llama/blob/main/examples/passkey.py) from the LongLLaMA's GitHub repo.
- **PasskeyWithLibrusec**: Similar to Passkey but with added noise from Librusec texts.

### Group II: Question Answering and Multiple Choice
This group consists of standard QA and multiple choice tasks adapted for the long-context setting.
- **MatreshkaNames**: Identify the person in dialogues based on the discussed topic. We used [Matreshka](https://huggingface.co/datasets/zjkarina/matreshka) dataset and [Russian Names](https://www.kaggle.com/datasets/rai220/russian-cyrillic-names-and-sex/data) dataset to create this and the next task.
- **MatreshkaYesNo**: Indicate whether a specific topic was mentioned in the dialog.
- **LibrusecHistory**: Answer questions based on historical texts. Ideologically similar to the [PassageRetrieval dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/passage_retrieval_en) from LongBench.
- **ruSciFi**: Answer true/false based on context and general world knowledge. Translation of [SciFi dataset](https://huggingface.co/datasets/L4NLP/LEval/viewer/sci_f) from L-Eval which originally was based on [SF-Gram](https://github.com/nschaetti/SFGram-dataset).
- **ruSciAbstractRetrieval**: Retrieve relevant paragraphs from scientific abstracts.
- **ruTPO**: Multiple-choice questions similar to TOEFL exams. Translation of the [TPO dataset](https://huggingface.co/datasets/L4NLP/LEval/viewer/tpo) from L-Eval.
- **ruQuALITY**: Multiple-choice QA tasks based on detailed texts. Created by translating the [QuALITY dataset](https://huggingface.co/datasets/L4NLP/LEval/viewer/quality) from L-Eval.

### Group III: Multi-hop Question Answering
This group includes long-context multi-hop QA problems where the answer requires combining multiple pieces of information distributed across the context.
- **ruBABILongQA (1-5)**: 5 long-context reasoning tasks for QA using facts hidden among irrelevant information.
- **LongContextMultiQ**: Multi-hop QA based on Wikidata and Wikipedia.
- **LibrusecMHQA**: Multi-hop QA requiring information distributed across several text parts.
- **ru2WikiMultihopQA**: Translation of the [2WikiMultihopQA dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/2wikimqa_e) from LongBench.

### Group IV: Complex Reasoning and Mathematical Problems
This group includes the most complex long-context tasks, which span beyond multiple-choice and multi-hop QA. At this point Group IV comprises only one task and we invite the community to contribute to it.
- **ruSciPassageCount**: Count unique paragraphs in a long text. Uses the basic idea of the original [PassageCount dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/passage_count) from LongBench.

## Metrics

We use **Exact Match (EM)** as a primary metric for all tasks. **EM** is used to evaluate the accuracy of the model's responses by comparing the predicted answers to the ground truth.

## Changes from the Original Version

This version of LIBRA includes both automatic and manual improvements over the original release. All datasets underwent automatic quality filtering to ensure consistency and reliability of annotations. In addition, several datasets were manually revised and extended with the help of human annotators: **LibrusecMHQA**, **LongContextMultiQ**, **MatreshkaNames**, **MatreshkaYesNo**, **ru2WikiMultihopQA**, **ruSciFi**, and **ruTPO** received targeted corrections and additional examples. The datasets **ruGSM100** and **ruQasper** were removed from the benchmark as they did not meet the updated quality criteria.

The maximum supported context length has been extended from 128k to 512k tokens. The following datasets now include examples at longer context lengths not present in the original version: **ruBABILongQA (1–5)**, **ruSciAbstractRetrieval**, **ruSciPassageCount**, **LongContextMultiQ**, **MatreshkaYesNo**, **Passkey**, and **PasskeyWithLibrusec**.

## Evaluation

Starting from this version, LIBRA supports evaluation via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/pull/3751) — a widely adopted framework for standardized LLM evaluation. Both full **LIBRA** and compact **LIBRA Mini** evaluations are supported.

To get started:

```bash
pip install lm-eval[vllm]
```

Run evaluation on the full LIBRA benchmark:

```bash
lm_eval --model vllm \
        --model_args pretrained=Qwen/Qwen3-30B-A3B,max_model_len=262144 \
        --tasks libra \
        --apply_chat_template \
        --device cuda:0
```

Run evaluation on LIBRA Mini only:

```bash
lm_eval --model vllm \
        --model_args pretrained=Qwen/Qwen3-30B-A3B,max_model_len=262144 \
        --tasks libra_mini \
        --apply_chat_template \
        --device cuda:0
```

For the full list of configuration options and instructions on adding new models, please refer to the [lm-evaluation-harness documentation](https://github.com/EleutherAI/lm-evaluation-harness/pull/3751).

## Citation

```
@misc{churin2024longinputbenchmarkrussian,
      title={Long Input Benchmark for Russian Analysis}, 
      author={Igor Churin and Murat Apishev and Maria Tikhonova and Denis Shevelev and Aydar Bulatov and Yuri Kuratov and Sergei Averkiev and Alena Fenogenova},
      year={2024},
      eprint={2408.02439},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2408.02439}, 
}
```