Sophon-Realtime v1 系列的端到端版：听和说用同一种向量语言
AI降临派（Adventists AI）· Apache-2.0
GitHub · Hugging Face · 魔搭 · English
> **研究预览。** 生产使用请选 [Sophon-Realtime-32B-Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag)（表现力最强，单卡）或 [Sophon-Realtime-32B-Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash)（最快、并发最高，单卡）。三个版本是同一个大脑，只有出声方式不同。 ## 这是什么 E2E 版不用单独的语音模型。32B 模型在听你说话时，内部会形成一串连续向量，里面既有说了什么，也有怎么说的（语调、轻重、情绪）。E2E 的说话头直接用这门「向量语言」写出回复，再由解码器还原成声音，所以模型听得懂的语气，它也能说出来。 ## 在 v1 系列里的位置 Sophon-Realtime v1 有三个版本，同一个大脑（VoiceBench 82.8、Big Bench Audio 81.8%，两项开源全双工第一；副语言理解综合第一），区别只在回复怎么变成声音： | | [Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag)（推荐） | [Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash) | **E2E（本仓库，研究预览）** | |---|---|---|---| | NVV-SuperBench（中 / 英） | **0.86 / 0.80** | 0.77 / 0.63 | 0.77 / 0.60 | | 单路首包 | 约 0.6 秒 | **0.31 秒** | 约 1 秒 | | 显卡 | 1 张 | 1 张 | 2 张 | E2E 适合研究端到端的语音表达；生产使用建议选 Tag 或 Flash。系列介绍和完整评测见 [v1 系列主页](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-v1)。 ## 本版更新（2026-11） - **解码器换代，4 步出声。** 每块音频的计算量与旧版相同。 - 同一说话头的回复上，与旧版解码器对比： | | 旧版（Sophon-Realtime-v1） | **本版** | |---|---|---| | 长回复错字率（150 条，中 / 英 / 日） | 1.78%（1.03 / 1.32 / 2.99） | **1.51%（0.97 / 0.71 / 2.86）** | | 未见说话人参考音色的相似度 | 0.33 | **0.52** | | 单路首包（发布包 bench，中位） | 0.57 s | **0.51 s** | - 旧版权重仍保留在 [Sophon-Realtime-v1](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-v1)。 ## 部署 两张 80 GB 显卡：思考器一张，说话引擎一张（说话引擎单卡建议 4–5 路并发）。需要 Linux、NVIDIA 驱动 580 及以上。 ```bash git clone https://github.com/adventists-ai/sophon-realtime cd sophon-realtime bash scripts/install.sh # 建两个虚拟环境，并从魔搭下载本模型 SOPHON_MODEL_DIR=$PWD/Sophon-Realtime-32B-E2E THINKER_GPU=0 TALKER_GPU=1 bash scripts/start.sh ``` 启动后打开 `http://<服务器>:8000/` 即可在网页里直接对话，也可以通过 WebSocket 接口（`/ws`）接入自己的客户端。说明见 GitHub 仓库的 [docs/deploy_CN.md](https://github.com/adventists-ai/sophon-realtime/blob/main/docs/deploy_CN.md)。 ## 文件 | 目录 | 内容 | |---|---| | `base/` | 32B 底座（思考器与说话头共用） | | `adapters/tag/` | 写回复的 LoRA | | `talker/` | 说话头 | | `e2/` | 解码器（4 步）与声码器 | | `voices/` | 默认音色 | ## 许可证 代码和权重都是 Apache-2.0。第三方组件保留各自的许可证，见 GitHub 仓库的 NOTICE。 ## 联系 商务合作：jiejin@adventists.ai --- # Sophon-Realtime-32B-E2E (research preview) **The end-to-end build of the Sophon-Realtime v1 family: hearing and speaking share one vector language.** Adventists AI · Apache-2.0 > **Research preview.** For production, use [Sophon-Realtime-32B-Tag](https://huggingface.co/adventists-ai/Sophon-Realtime-32B-Tag) (most expressive, one GPU) or [Sophon-Realtime-32B-Flash](https://huggingface.co/adventists-ai/Sophon-Realtime-32B-Flash) (fastest, most users per GPU, one GPU). All three share the same brain and differ only in how the reply becomes sound. ## What it is The E2E build speaks without a separate speech model. While the 32B model listens, it builds a stream of continuous vectors that carries both what was said and how (intonation, emphasis, emotion). The E2E talker writes replies directly in that vector language and a decoder turns it into sound, so the tone the model can hear is the tone it can speak. ## Where it fits in the v1 family Sophon-Realtime v1 has three builds that share one brain (VoiceBench 82.8 and Big Bench Audio 81.8%, #1 among open full-duplex models on both; #1 in paralinguistic understanding). They differ only in how the reply becomes sound: | | [Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag) (recommended) | [Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash) | **E2E (this repository, research preview)** | |---|---|---|---| | NVV-SuperBench (zh / en) | **0.86 / 0.80** | 0.77 / 0.63 | 0.77 / 0.60 | | First audio, one user | ~0.6 s | **0.31 s** | ~1 s | | GPUs | 1 | 1 | 2 | E2E is for research on end-to-end expressive speech; for production, use Tag or Flash. Family overview and full benchmarks: [v1 family home page](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-v1). ## What's new (Nov 2026) - **New decoder, 4 steps.** Same compute per audio chunk as before. - Against the previous decoder, on replies from the same talker: | | previous (Sophon-Realtime-v1) | **this release** | |---|---|---| | Long replies, error rate (150 replies, zh / en / ja) | 1.78% (1.03 / 1.32 / 2.99) | **1.51% (0.97 / 0.71 / 2.86)** | | Voice similarity to an unseen reference speaker | 0.33 | **0.52** | | First audio, one user (release-package bench, median) | 0.57 s | **0.51 s** | - The previous weights stay in [Sophon-Realtime-v1](https://huggingface.co/adventists-ai/Sophon-Realtime-v1). ## Deployment Two 80 GB GPUs: one for the thinker, one for the speaking engine (4–5 concurrent conversations recommended per engine GPU). Linux with NVIDIA driver 580 or newer. ```bash git clone https://github.com/adventists-ai/sophon-realtime cd sophon-realtime bash scripts/install.sh # two virtual environments + this model from ModelScope SOPHON_MODEL_DIR=$PWD/Sophon-Realtime-32B-E2E THINKER_GPU=0 TALKER_GPU=1 bash scripts/start.sh ``` Then open `http://