听得懂语气，也说得出语气的开源全双工语音模型
AI降临派（Adventists AI）· Apache-2.0
GitHub · Hugging Face · 魔搭组织页 · English
## 亮点 - **VoiceBench 82.8、Big Bench Audio 81.8%：开源全双工里最强的「脑子」。** 两项都是开源全双工模型第一，VoiceBench 与 GPT-4o-mini-Audio 持平。 - **副语言理解 48.6，综合第一：听得懂语气。** 在 LISTEN、StressTest、MSPB 公开基准上高于 Gemini 3.8 Flash（46.9）和 Qwen3-Omni（44.6）。它直接从声音里听出情绪、重音、反讽、笑声和叹气。 - **NVV-SuperBench 中文 0.74、英文 0.68：说得出语气。** 无表达能力的基线只有 0.37 / 0.27。回答时什么时候笑、什么时候叹气、嗓子干了要不要清一清，由模型根据对话自己决定。 - **约 0.4 秒开口：快。** 你说话的全程它都在听，回复边听边起草；你一停就接上，你插话它马上停下，回答你的新问题。 - **Full-Duplex-Bench v3 工具调用 Pass@1 48%，高于 Grok（43%）；接话成功率 100%：能干活。** 支持 Live 模式，可以和后端大模型联合工作；兼容 OpenAI GPT-Live 的接口格式。文字和声音同步流式生成，每句回答都有文字记录，可以审计。 - **单张显卡即可部署：能自己部署。** 量化版本可在单张消费级显卡上运行。代码和权重都是 Apache-2.0，可以商用。 ## 评测详情 ### 对话能力：VoiceBench（官方评测器，gpt-4o-mini 判分） | 模型 | 全双工 | 开放权重 | 总分 | |---|:-:|:-:|:-:| | GPT-4o-Audio | — | — | 86.8 | | **Sophon-Realtime v1** | ✅ | ✅ | **82.8** | | GPT-4o-mini-Audio | — | — | 82.8 | | MiniCPM-o 4.5 | ✅ | ✅ | 76.1 | | MiniCPM-o 2.6 | ✅ | ✅ | 71.2 | | Freeze-Omni | ✅ | ✅ | 55.2 | | NemotronLabs VoiceChat | ✅ | ✅ | 55.1 | | PersonaPlex | ✅ | ✅ | 30.6 | | Moshi | ✅ | ✅ | 29.5 | AlpacaEval 用 199 题标准子集。其他模型的分数来自 VoiceBench 榜单和 NemotronLabs VoiceChat 论文（arXiv:2609.21967）表 3，其中 MiniCPM-o 4.5 的分数来自第三方评测。 ### 语音推理：Big Bench Audio 1,000 道语音推理题，按它实际说出的话判分。 | 模型 | 开放权重 | 准确率 | |---|:-:|:-:| | GPT-Live-1 | — | 90% | | **Sophon-Realtime v1** | ✅ | **81.8%** | | Raon SpeechChat | ✅ | 58% | | Freeze-Omni | ✅ | 33% | | PersonaPlex | ✅ | 19% | | Moshi | ✅ | 4% | Sophon 为自测：回答转写后由 gpt-4.1 判分。其他模型的数字来自 Artificial Analysis 榜单。 ### 调用工具：Full-Duplex-Bench v3 真人不流利语音 + 多步工具调用，100 题，GPT-4o 判分。 | 模型 | 开放权重 | Pass@1 ↑ | 接话成功率 ↑ | |---|:-:|:-:|:-:| | GPT-Realtime | — | 60% | 96% | | Gemini Live 3.1 | — | 54% | — | | **Sophon-Realtime** | ✅ | **48%** | **100%** | | 级联方案 | — | 45% | — | | Grok | — | 43% | — | | Ultravox v0.7 | ✅ | 41% | — | Sophon 为多次运行中的最好成绩（内部基线配置）。其他模型的数字来自 Full-Duplex-Bench v3 官方结果。 ### 听懂语气：副语言理解 Sophon-Realtime 用 [DuplexJev](https://www.modelscope.cn/models/AdventistsAI/DuplexJev-32B-Para) 听声音，测的是情绪、反讽、重音和语调。 | 模型 | LISTEN（4 项）+ StressTest + MSPB 合计 | |---|:-:| | **DuplexJev（Sophon-Realtime 的听觉模型）** | **48.6** | | Gemini 3.8 Flash | 46.9 | | Qwen3-Omni | 44.6 | ### 说出语气：NVV-SuperBench 看模型说出来的声音里，该有的笑、咳嗽、叹气、呼吸有没有真的出现。官方评测脚本，每类 50 条，Gemini 3 Flash 判分，F1 越高越好。 | 模型 | 中文 | 英文 | |---|:-:|:-:| | **Sophon-Realtime v1** | **0.74** | **0.68** | | 无表达能力的内部基线 | 0.37 | 0.27 | ## 工作原理 - **一门藏在模型里的「向量语言」。** 我们发现，语音大模型在听声音时，内部用的那串连续向量本身就是一门完整的语言：冻结的大模型能像读文字一样直接读懂它，里面既有说了什么，也有怎么说的，包括语调、轻重、情绪。Sophon-Realtime v1 的发声端直接「写」这门向量语言，再把它还原成声音。听和说用的是同一套表示，所以模型听得懂的语气，它也说得出来。 - **一直在听。** 音频持续流入模型。每次停顿，同一个模型用一个 token 判断：你说完了、还在说、在附和，还是在想。 - **提前起草，准时开口。** 第一次停顿就开始起草回复，判断轮到自己时立刻说出来；你如果继续说，草稿在你听到之前就丢掉。 - **记得你真正听到了什么。** 历史里保存双方的真实音频。你打断时，只保留你已经听到的部分。 ## 部署 单张显卡即可部署；量化版本可在单张消费级显卡上运行。 ```bash pip install sophon-realtime modelscope download --model AdventistsAI/Sophon-Realtime-v1 --local_dir Sophon-Realtime-v1 # 启动命令见 GitHub 仓库 deploy/ 目录 ``` 服务兼容 OpenAI GPT-Live 的接口格式，现有的实时语音客户端可以直接接入。 ## 微调 可以用 ms-swift 微调思考器的 LLM 主干（LoRA），改人设、口吻、领域知识和工具调用习惯。听觉编码器和发声部分保持原样，接话判断不受影响。微调从自带的写标签 LoRA（`adapters/tag`）接着训，笑声、咳嗽等声音表现会保留。一张 80G 显卡即可。 ```bash pip install "ms-swift>=4.6" git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune modelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1 python -m sophon_swift.adapters to-swift Sophon-Realtime-v1/adapters/tag tag_swift swift sft --custom_register_path sophon_swift/swift_register.py \ --model Sophon-Realtime-v1 --adapters tag_swift --tuner_type lora \ --dataset my_data.jsonl --learning_rate 5e-5 --num_train_epochs 2 \ --gradient_accumulation_steps 8 --gradient_checkpointing true --output_dir output ``` 数据格式、多轮与工具调用、导出部署和回归检查见 GitHub 仓库的 [finetune 目录](https://github.com/adventists-ai/sophon-realtime/tree/main/finetune)。 ## 许可证 代码和权重都是 Apache-2.0。第三方组件保留各自的许可证，见 NOTICE。 ## 联系 商务合作：jiejin@adventists.ai --- # Sophon-Realtime v1 (English) **An open full-duplex voice model that hears your tone and speaks with one.** Adventists AI (AI降临派) · Apache-2.0 ## Highlights - **VoiceBench 82.8 and Big Bench Audio 81.8%: the strongest brain among open full-duplex models.** #1 among open full-duplex models on both; level with GPT-4o-mini-Audio on VoiceBench. - **Paralinguistic understanding 48.6, #1 overall: it hears your tone.** Ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on LISTEN, StressTest and MSPB. It hears emotion, emphasis, sarcasm, laughter and sighs straight from your voice. - **NVV-SuperBench 0.74 (Chinese) and 0.68 (English): it speaks with tone.** The baseline without expression scores 0.37 / 0.27. The model decides from the conversation when to laugh, when to sigh, and when to clear a dry throat. - **Starts talking about 0.4 s after you stop: it's fast.** It listens the whole time you talk and drafts its reply as you go. It picks up as soon as you stop, and when you cut in it stops and answers your new question. - **Full-Duplex-Bench v3 tool calling: Pass@1 48%, ahead of Grok (43%), and 100% response rate: it gets work done.** Live mode lets it work together with a larger backend model, and it is compatible with the OpenAI GPT-Live interface format. Text and voice stream together, so every reply comes with an auditable transcript. - **Runs on a single GPU: yours to deploy.** A quantized build runs on a single consumer GPU. Apache-2.0 code and weights, free for commercial use. ## Benchmark details ### Conversation: VoiceBench (official evaluators, gpt-4o-mini judge) | model | full-duplex | open weights | overall | |---|:-:|:-:|:-:| | GPT-4o-Audio | — | — | 86.8 | | **Sophon-Realtime v1** | ✅ | ✅ | **82.8** | | GPT-4o-mini-Audio | — | — | 82.8 | | MiniCPM-o 4.5 | ✅ | ✅ | 76.1 | | MiniCPM-o 2.6 | ✅ | ✅ | 71.2 | | Freeze-Omni | ✅ | ✅ | 55.2 | | NemotronLabs VoiceChat | ✅ | ✅ | 55.1 | | PersonaPlex | ✅ | ✅ | 30.6 | | Moshi | ✅ | ✅ | 29.5 | AlpacaEval uses the 199-question standard subset. Other scores come from the VoiceBench leaderboard and from Table 3 of the NemotronLabs VoiceChat paper (arXiv:2609.21967); the MiniCPM-o 4.5 score there is from an independent evaluation. ### Spoken reasoning: Big Bench Audio 1,000 spoken reasoning questions, graded on what the model actually says. | model | open weights | accuracy | |---|:-:|:-:| | GPT-Live-1 | — | 90% | | **Sophon-Realtime v1** | ✅ | **81.8%** | | Raon SpeechChat | ✅ | 58% | | Freeze-Omni | ✅ | 33% | | PersonaPlex | ✅ | 19% | | Moshi | ✅ | 4% | Sophon: self-run, with spoken answers transcribed and judged by gpt-4.1. Others: Artificial Analysis leaderboard. ### Tool calling: Full-Duplex-Bench v3 Disfluent real human speech with multi-step tool calls. 100 tasks, judged by GPT-4o. | model | open weights | Pass@1 ↑ | responds when it should ↑ | |---|:-:|:-:|:-:| | GPT-Realtime | — | 60% | 96% | | Gemini Live 3.1 | — | 54% | — | | **Sophon-Realtime** | ✅ | **48%** | **100%** | | cascaded pipeline | — | 45% | — | | Grok | — | 43% | — | | Ultravox v0.7 | ✅ | 41% | — | Sophon: best of several runs (internal baseline configuration). Others: official Full-Duplex-Bench v3 results. ### Hearing tone: paralinguistic understanding Sophon-Realtime listens with [DuplexJev](https://www.modelscope.cn/models/AdventistsAI/DuplexJev-32B-Para). These benchmarks test emotion, sarcasm, stress and intonation. | model | LISTEN (4 tasks) + StressTest + MSPB, combined | |---|:-:| | **DuplexJev (Sophon-Realtime's ears)** | **48.6** | | Gemini 3.8 Flash | 46.9 | | Qwen3-Omni | 44.6 | ### Speaking with tone: NVV-SuperBench Checks whether the laughs, coughs, sighs and breaths a reply calls for actually show up in the audio. Official scripts, 50 items per category, judged by Gemini 3 Flash; higher F1 is better. | model | Chinese | English | |---|:-:|:-:| | **Sophon-Realtime v1** | **0.74** | **0.68** | | internal baseline without expression | 0.37 | 0.27 | ## How it works - **A vector language hidden inside the model.** We found that the stream of continuous vectors a speech LLM builds while it listens is a complete language in its own right. A frozen LLM reads it as easily as text, and it carries both what was said and how it was said: intonation, emphasis and emotion. Sophon-Realtime v1's voice writes this vector language directly, then turns it back into sound. Hearing and speaking share one representation, so the tone the model can hear is the tone it can speak. - **Listens continuously.** At each pause, the same model decides in a single token whether you've finished, are still talking, are back-channeling or are thinking. - **Drafts early, speaks on time.** A reply is drafted from your first pause and released the moment the model decides it's its turn. If you keep talking, the draft is dropped before you hear it. - **Remembers what you actually heard.** History keeps the real audio of both sides. When you interrupt, only the part you heard is kept. ## Deployment Runs on a single GPU; a quantized build runs on a single consumer GPU. ```bash pip install sophon-realtime modelscope download --model AdventistsAI/Sophon-Realtime-v1 --local_dir Sophon-Realtime-v1 # launch commands: see deploy/ in the GitHub repository ``` The server is compatible with the OpenAI GPT-Live interface format, so existing realtime voice clients can connect directly. ## Fine-tuning Fine-tune the thinker's LLM backbone with LoRA in ms-swift to change persona, tone, domain knowledge and tool use. The audio encoder and the voice stay as released, and turn-taking is not affected. Training continues the bundled tag LoRA (`adapters/tag`), so laughter, coughs and other sound tags are kept. One 80 GB GPU is enough. ```bash pip install "ms-swift>=4.6" git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune modelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1 python -m sophon_swift.adapters to-swift Sophon-Realtime-v1/adapters/tag tag_swift swift sft --custom_register_path sophon_swift/swift_register.py \ --model Sophon-Realtime-v1 --adapters tag_swift --tuner_type lora \ --dataset my_data.jsonl --learning_rate 5e-5 --num_train_epochs 2 \ --gradient_accumulation_steps 8 --gradient_checkpointing true --output_dir output ``` Data format, multi-turn and tool calls, export, serving and regression checks: see [finetune/](https://github.com/adventists-ai/sophon-realtime/tree/main/finetune) in the GitHub repository. ## License Code and weights are released under Apache-2.0. Third-party components keep their own licences; see NOTICE. ## Contact Business enquiries: jiejin@adventists.ai