---
title: "Expect the Unexpected: FailSafe Long Context QA for Finance"
canonical_url: "https://www.modelscope.cn/papers/114717"
md_url: "https://www.modelscope.cn/papers/114717.md"
arxiv_id: 2502.06329
published: 2025-02-10
last_updated: 2025-02-10
authors:
  - "Kiran Kamble"
  - "Melisa Russak"
  - "Dmytro Mozolevskyi"
  - "Muayad Ali"
  - "Mateusz Russak"
  - "Waseem AlShikh"
model_name: FailSafeQA
model_developer: "Writer、Inc."
domain:
  - "自然语言处理"
  - "金融人工智能"
  - "长上下文理解"
  - "大语言模型评测"
  - "鲁棒性分析"
type:
  - "自然语言处理"
  - "金融人工智能"
  - "长上下文理解"
  - "大语言模型评测"
  - "鲁棒性分析"
  - "Computation and Language"
arxiv_url: "https://arxiv.org/abs/2502.06329"
pdf_url: "https://arxiv.org/pdf/2502.06329"
---

# Expect the Unexpected: FailSafe Long Context QA for Finance

> We propose a new long-context financial benchmark, FailSafeQA, designed to test the robustness and context-awareness of LLMs against six variations in human-interface interactions in LLM-based query-answer systems within finance. We concentrate on two case…

「Expect the Unexpected: FailSafe Long Context QA for Finance」是 ModelScope 魔搭社区收录的论文，arXiv 2502.06329，作者为 Kiran Kamble, Melisa Russak, Dmytro Mozolevskyi et al.，发表于 2025-02-10，属于 自然语言处理、金融人工智能、长上下文理解 领域。

- **ArXiv**: 2502.06329
- **Published**: 2025-02-10
- **Authors**: Kiran Kamble, Melisa Russak, Dmytro Mozolevskyi, Muayad Ali, Mateusz Russak, Waseem AlShikh
- **Model**: FailSafeQA
- **Developer**: Writer、Inc.
- **Domain**: 自然语言处理, 金融人工智能, 长上下文理解, 大语言模型评测, 鲁棒性分析
- **ArXiv URL**: https://arxiv.org/abs/2502.06329
- **PDF**: https://arxiv.org/pdf/2502.06329

Source: https://www.modelscope.cn/papers/114717

---

> 应对不确定性：FailSafeQA为金融领域量身打造的长上下文问答系统

## 摘要

本文提出了一种新的长上下文金融基准测试工具——FailSafeQA，旨在评估大型语言模型（LLM）在金融领域的鲁棒性和上下文感知能力。研究背景方面，随着金融服务业对大型语言模型的依赖日益增加，确保这些模型在面对输入扰动时的可靠性和准确性变得至关重要。FailSafeQA通过模拟六种不同的人机交互变化，特别是在查询失败和上下文失败两种场景下进行测试。在查询失败场景中，研究人员通过引入拼写错误、不完整的查询和领域外的查询来考察模型的表现；在上下文失败场景中，则模拟了上传损坏、无关或空文档的情况。该研究采用了LLM-as-a-Judge的方法，并使用Qwen2.5-72B-Instruct模型进行细粒度评分，定义并计算了鲁棒性、上下文锚定和合规性得分。实验结果表明，尽管一些模型在应对输入扰动方面表现出色，但它们必须在提供准确回答和避免幻觉之间取得平衡。例如，Palmyra-Fin-128k-Instruct模型虽然在合规性上表现最佳，但在17%的测试案例中难以维持稳健预测；而OpenAI o3-mini模型虽然最稳健，却在41%的测试案例中制造了虚假信息。这一研究表明，即使高性能的模型也有很大的改进空间，强调了FailSafeQA作为开发更可靠的金融应用LLM工具的重要性。

## Abstract

We propose a new long-context financial benchmark, FailSafeQA, designed to test the robustness and context-awareness of LLMs against six variations in human-interface interactions in LLM-based query-answer systems within finance. We concentrate on two case studies: Query Failure and Context Failure. In the Query Failure scenario, we perturb the original query to vary in domain expertise, completeness, and linguistic accuracy. In the Context Failure case, we simulate the uploads of degraded, irrelevant, and empty documents. We employ the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained rating criteria to define and calculate Robustness, Context Grounding, and Compliance scores for 24 off-the-shelf models. The results suggest that although some models excel at mitigating input perturbations, they must balance robust answering with the ability to refrain from hallucinating. Notably, Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained strong baseline performance but encountered challenges in sustaining robust predictions in 17% of test cases. On the other hand, the most robust model, OpenAI o3-mini, fabricated information in 41% of tested cases. The results demonstrate that even high-performing models have significant room for improvement and highlight the role of FailSafeQA as a tool for developing LLMs optimized for dependability in financial applications. The dataset is available at: https://huggingface.co/datasets/Writer/FailSafeQA