---
title: "Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)"
canonical_url: "https://www.modelscope.cn/papers/62147"
md_url: "https://www.modelscope.cn/papers/62147.md"
arxiv_id: 2410.07507
published: 2024-10-10
last_updated: 2024-10-10
authors:
  - "Abhijit Mishra"
  - "Shreya Shukla"
  - "Jose Torres"
  - "Jacek Gwizdka"
  - "Shounak Roychowdhury"
model_name: Thought2Text
model_developer: "德克萨斯大学奥斯汀分校"
domain:
  - "自然语言处理"
  - "脑机接口"
  - "深度学习"
type:
  - "自然语言处理"
  - "脑机接口"
  - "深度学习"
  - "Computation and Language (cs.CL)"
arxiv_url: "https://arxiv.org/abs/2410.07507"
pdf_url: "https://arxiv.org/pdf/2410.07507.pdf"
---

# Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)

> Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves…

「Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)」是 ModelScope 魔搭社区收录的论文，arXiv 2410.07507，作者为 Abhijit Mishra, Shreya Shukla, Jose Torres et al.，发表于 2024-10-10，属于 自然语言处理、脑机接口、深度学习 领域。

- **ArXiv**: 2410.07507
- **Published**: 2024-10-10
- **Authors**: Abhijit Mishra, Shreya Shukla, Jose Torres, Jacek Gwizdka, Shounak Roychowdhury
- **Model**: Thought2Text
- **Developer**: 德克萨斯大学奥斯汀分校
- **Domain**: 自然语言处理, 脑机接口, 深度学习
- **ArXiv URL**: https://arxiv.org/abs/2410.07507
- **PDF**: https://arxiv.org/pdf/2410.07507.pdf

Source: https://www.modelscope.cn/papers/62147

---

> 从脑电波到文字：Thought2Text实现思想的直接表达

## 摘要

本文提出了一种名为Thought2Text的方法，旨在通过大规模语言模型（LLMs）从脑电图（EEG）信号生成文本。研究背景涉及脑机接口（BCI）系统的进展，特别是结合便携式和可穿戴的非侵入性EEG设备，实现大脑与外部设备的直接交互。本文的方法包括三个阶段：首先，训练EEG编码器以提取视觉特征；其次，利用图像和文本数据微调LLMs，以实现多模态描述生成；最后，在推理过程中进一步微调EEG嵌入以直接从EEG生成文本。实验使用了一个公共EEG数据集，验证了多模态LLMs（如LLAMA-V3, MISTRAL-V0.3, QWEN2.5）的有效性。该方法在便携、低成本的“思想到文本”技术方面取得了显著进展，具有在神经科学和自然语言处理中的潜在应用价值。论文的贡献包括将脑信号与LLMs集成，利用EEG信号的语言无关性增强LLM的交互性和跨语言的可扩展性，并在一个流行的公共数据集上验证了模型的有效性。

## Abstract

Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves three stages: (1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference. Experiments on a public EEG dataset collected for six subjects with image stimuli demonstrate the efficacy of multimodal LLMs (LLaMa-v3, Mistral-v0.3, Qwen2.5), validated using traditional language generation evaluation metrics, GPT-4 based assessments, and evaluations by human expert. This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing (NLP).