---
title: "Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning"
canonical_url: "https://www.modelscope.cn/papers/125042"
md_url: "https://www.modelscope.cn/papers/125042.md"
arxiv_id: 2503.07572
published: 2025-03-10
last_updated: 2025-03-10
authors:
  - "Yuxiao Qu"
  - "Matthew Y. R. Yang"
  - "Amrith Setlur"
  - "Lewis Tunstall"
  - "Edward Emanuel Beeching"
  - "Ruslan Salakhutdinov"
  - "Aviral Kumar"
model_name: MRT
model_developer: "Carnegie Mellon University、Hugging Face"
domain:
  - "自然语言处理"
  - "强化学习"
  - "大语言模型推理"
  - "元强化学习"
  - "测试时计算优化"
type:
  - "自然语言处理"
  - "强化学习"
  - "大语言模型推理"
  - "元强化学习"
  - "测试时计算优化"
  - "Machine Learning"
  - "Artificial Intelligence"
  - "Computation and Language"
arxiv_url: "https://arxiv.org/abs/2503.07572"
pdf_url: "https://arxiv.org/pdf/2503.07572"
code_link: "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
---

# Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning

> Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs. Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize…

「Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning」是 ModelScope 魔搭社区收录的论文，arXiv 2503.07572，作者为 Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur et al.，发表于 2025-03-10，属于 自然语言处理、强化学习、大语言模型推理 领域。

- **ArXiv**: 2503.07572
- **Published**: 2025-03-10
- **Authors**: Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, Aviral Kumar
- **Model**: MRT
- **Developer**: Carnegie Mellon University、Hugging Face
- **Domain**: 自然语言处理, 强化学习, 大语言模型推理, 元强化学习, 测试时计算优化
- **ArXiv URL**: https://arxiv.org/abs/2503.07572
- **PDF**: https://arxiv.org/pdf/2503.07572
- **Code**: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

Source: https://www.modelscope.cn/papers/125042

---

> 优化测试时间计算：元强化微调引领大语言模型推理新突破

## 摘要

本文探讨了如何优化大语言模型（LLM）在推理过程中对测试时间计算资源的使用。现有的方法主要通过微调搜索轨迹或运行带有0/1结果奖励的强化学习来实现这一目标，但这些方法未能高效利用测试时间计算资源。为了解决这一问题，作者提出了一种新的微调方法——元强化微调（Meta Reinforcement Fine-Tuning，简称MRT）。MRT将优化测试时间计算的问题形式化为元强化学习问题，并引入累积遗憾作为衡量标准。具体而言，MRT通过最大化密集奖励和结果奖励来最小化累积遗憾，从而平衡探索与利用。实验表明，MRT相比传统的基于结果奖励的RL方法，在数学推理任务上取得了2-3倍的性能提升和约1.5倍的令牌效率提升。此外，MRT在处理更复杂的、未见过的问题时表现出了更好的适应性。

## Abstract

Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs. Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget improves? In this paper, we try to answer these questions. We formalize the problem of optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a principled perspective on spending test-time compute. This perspective enables us to view the long output stream from the LLM as consisting of several episodes run at test time and leads us to use a notion of cumulative regret over output tokens as a way to measure the efficacy of test-time compute. Akin to how RL algorithms can best tradeoff exploration and exploitation over training, minimizing cumulative regret would also provide the best balance between exploration and exploitation in the token stream. While we show that state-of-the-art models do not minimize regret, one can do so by maximizing a dense reward bonus in conjunction with the outcome 0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in the output stream, quantified by the change in the likelihood of eventual success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL.
