--- title: tag-tts canonical_url: "https://www.modelscope.cn/models/AdventistsAI/tag-tts" md_url: "https://www.modelscope.cn/models/AdventistsAI/tag-tts.md" repository: AdventistsAI/tag-tts chinese_name: "tag-tts：标签真正生效的 CosyVoice3" last_updated: 2026-10-29 license: apache-2.0 pipeline_tag: text-to-speech tasks: - text-to-speech base_model: - FunAudioLLM/Fun-CosyVoice3-0.5B-2512 base_model_relation: finetune parameters: 494.0M tensor_type: - BF16 library_name: - pytorch - safetensors - onnx frameworks: - Pytorch language: - zh - en downloads: 3 stars: 0 tags: - text-to-speech - cosyvoice - paralinguistic - reinforcement-learning - dpo - grpo - duplexjev --- # tag-tts > tag-tts - AdventistsAI 在 ModelScope 开源的模型。tag-tts makes CosyVoice3 actually do what its tags say. AdventistsAI/tag-tts 是 ModelScope 魔搭社区上的 494.0M 参数text-to-speech模型，采用 apache-2.0 许可，基于 FunAudioLLM/Fun-CosyVoice3-0.5B-2512 构建。 - **Repository**: AdventistsAI/tag-tts - **License**: apache-2.0 - **Tasks**: text-to-speech - **Parameters**: 494.0M - **Base model**: FunAudioLLM/Fun-CosyVoice3-0.5B-2512 - **Tags**: text-to-speech, cosyvoice, paralinguistic, reinforcement-learning, dpo, grpo, duplexjev - **Downloads**: 3 - **Stars**: 0 - **Last updated**: 2026-10-29 Source: https://www.modelscope.cn/models/AdventistsAI/tag-tts --- # tag-tts **tag-tts makes CosyVoice3 actually do what its tags say.** [Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) supports inline paralinguistic tags such as `[laughter]`, `[cough]`, `[sigh]`, `…` and `…`, but in real-time (bi-streaming) synthesis it ignores most of them. tag-tts is a drop-in replacement for its language model (`llm.pt`), trained with reinforcement learning so that the tag you write is the sound you hear. The judge in the RL loop is our paralinguistic speech-understanding model **[DuplexJev](https://github.com/adventists-ai/duplexjev)**: it listens to every candidate and decides whether the laugh, cough or sigh really happened and **which word carries the stress**. [中文说明](#中文说明) · [Audio examples](#audio-examples) · [Usage](#usage) · [Changelog](#changelog) > **v1.1 (2026-10-30)**: two more RL rounds (GRPO + DuplexJev-guided positives). Stress on the tagged word 35.0% → **40.7%** on held-out sentences, sigh 15.0% → **17.9%**, tongue-twister error back down (6.42 → **6.24**). ## Results Same text, same cloned voice, token-by-token bi-streaming, no style instruction. "Hit" = the event is clearly present. DuplexJev hit = DuplexJev-32B-Para picks the right answer in a multiple-choice question ("Which sound does the speaker make besides speaking?", options rotated); AST hit = AudioSet-AST event probability above a fixed threshold. 24 utterances per tag (tag at sentence start / middle / end, Chinese and English, 2 samples each). | Tag | CosyVoice3 | tag-tts v1.0 | **tag-tts v1.1** | |---|---|---|---| | `[laughter]` | 62% | 100% | **100%** | | `…` | 12% | 88% | **92%** | | `[cough]` | 42% | 79% | **71%** | | `[sigh]` | 4% | 17% | **17%** | | `[breath]` | 25% | 25% | 17% | | no tag (nothing extra heard) | 100% | 88% | 88% | 24 utterances per tag, so differences of one or two utterances (±8%) are noise. Stress, held-out sentences (70 sentences × 4 samples; one word wrapped in `…`; DuplexJev answers "Which word does the speaker emphasise?" among the target word, 2–3 other words and "no word is emphasised"): | | CosyVoice3 | tag-tts v1.0 | **tag-tts v1.1** | |---|---|---|---| | `` – stress lands on the tagged word | 28.6% | 35.0% | **40.7%** | | `[sigh]` – sigh heard (DuplexJev) | 4.3% | 15.0% | **17.9%** | Basic quality holds (zh, en and the tongue-twister set are all at or below the official model's error, except zh-hard which is 0.2 points worse). Seed-TTS-eval in the same deployment setting (bi-streaming, voice cloned from the prompt audio only, no prompt text; ASR: Paraformer-zh / Whisper-large-v3; speaker similarity with CAM++, so SIM is not comparable to numbers in papers): | | zh CER ↓ | en WER ↓ | zh-hard CER ↓ | SIM zh / en ↑ | |---|---|---|---|---| | CosyVoice3 (official `llm.pt`) | 1.53 | 2.58 | 6.00 | 0.776 / 0.814 | | tag-tts v1.0 | 1.29 | 2.02 | 6.42 | 0.776 / 0.815 | | **tag-tts v1.1** | **1.27** | **1.96** | 6.24 | 0.777 / 0.814 | tag-tts also inherits our earlier RL round for **sentence-level style instructions** ("say this happily / sadly / angrily / louder / slower …"). On 160 ESD sentences with an emotion instruction, the emotion recognised by emotion2vec matches the instruction in **66.2% / 71.9%** of cases with v1.1 (default voice / cloned ESD voice; v1.0 70.0% / 72.5%), vs **49.4% / 47.5%** for the official model. ## How it was trained 1. **Sampling.** A tag is inserted into each training sentence (start / middle / end, or one word wrapped in ``). The current model synthesises 8 candidates in the production setting (bi-streaming, 10 cloned voices, mostly without style instruction). 2. **Judging.** DuplexJev-32B answers multiple-choice questions about every candidate: *which word is stressed*, *which non-verbal sound do you hear*. AudioSet-AST is a second detector for laughter / cough / breath. Candidates that mispronounce (ASR error > 10%), drift in timbre or speaking rate are ranked last. 3. **Preference optimisation.** Best-vs-worst pairs within each group train the 0.5B language model with DPO (β = 0.1, lr 1e-6, chosen-NLL regulariser 0.1). Flow matching and vocoder are unchanged. Three on-policy rounds (2,100–2,800 preference pairs each) gave v1.0. 4. **v1.1: GRPO with guided positives.** Stress is the hardest target: for a third of the sentences the model never stresses the tagged word in 12 tries, so preference learning has nothing to work with. For those sentences we sample with a text hint (a short pause before the tagged word) and keep the candidates DuplexJev judges as stressed; training then sees the original text, so the model learns to stress from `` alone. These positives are added as an NLL term to on-policy GRPO (8 samples per prompt, group-normalised advantage, token-level clipping, KL to the previous round). Two rounds; the second did not improve further, so v1.1 is the first. The RL rounds use model-generated audio only; no external recordings are used for training. Lessons we learned on the way (they matter for anyone using CosyVoice3 tags): - With no instruction the prefix must be exactly `You are a helpful assistant.<|endofprompt|>`. A single extra space before `<|endofprompt|>` silently disables most tags (laughter detection 0.24 → 0.007). - In bi-streaming, send every tag as **one complete text chunk**. A tag split across chunks (`[laugh` + `ter]`) is read out loud. - Malformed or unknown tags (`[laugh]`, `(laughs)`, `[happy]`, `【笑声】`…) are read out loud. Whitelist your tags. - An LLM-based judge must be queried in exactly the format it was trained with; with a slightly different prompt DuplexJev's stress accuracy dropped from 90% to 5%. ## Usage Use the official [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) code. This repository is a complete model directory (only `llm.pt` differs from the official release), so it loads exactly like the official model: ```python import torchaudio from cosyvoice.cli.cosyvoice import AutoModel tts = AutoModel(model_dir="pretrained_models/tag-tts") # this repo prompt_wav = "prompt.wav" # 3–10 s of the target voice text = "You are a helpful assistant.<|endofprompt|>[laughter]哎呀，今天真是太累了，从早上忙到现在。" for i, out in enumerate(tts.inference_cross_lingual(text, prompt_wav, stream=False)): torchaudio.save(f"out_{i}.wav", out["tts_speech"], tts.sample_rate) # with a style instruction (note the space after the full stop) for out in tts.inference_instruct2("Others were hiding in fear for their lives.", "You are a helpful assistant. Please say this in a sad tone.<|endofprompt|>", prompt_wav, stream=False): ... ``` Already using the official model? Just replace `llm.pt` with the one from this repo. Recommended tags: `[laughter]`, `…`, `[cough]`, `…`, `[breath]`. `[sigh]` improved but is still unreliable. ## Audio examples Official CosyVoice3 vs tag-tts (v1.0 samples), same text and voice (in `examples/`). Numbers are DuplexJev's probability for the intended answer. | Text | CosyVoice3 | tag-tts | |---|---|---| | `[laughter]哎呀，今天真是太累了，从早上忙到现在。` | [▶ 0.00](examples/laughter_start_zh1_0_cosyvoice3.wav) | [▶ **1.00**](examples/laughter_start_zh1_0_tagtts.wav) | | `Okay, I see what you mean, let's take it step by step.` | [▶ 0.00](examples/span_laughter_mid_en0_1_cosyvoice3.wav) | [▶ **1.00**](examples/span_laughter_mid_en0_1_tagtts.wav) | | `[cough]Oh well, what a long day, I've been busy since this morning.` | [▶ 0.00](examples/cough_start_en1_1_cosyvoice3.wav) | [▶ **1.00**](examples/cough_start_en1_1_tagtts.wav) | | `Others were hiding in fear for their lives.` | [▶ 0.00](examples/u0478_en_3_cosyvoice3.wav) | [▶ **1.00**](examples/u0478_en_3_tagtts.wav) | | `[sigh]我支持你，他是需要重做` | [▶ 0.00](examples/u0470_zh_0_cosyvoice3.wav) | [▶ **0.99**](examples/u0470_zh_0_tagtts.wav) | | `Overcast skies are expected in the area, [sigh]though.` | [▶ 0.00](examples/u0447_en_1_cosyvoice3.wav) | [▶ **0.89**](examples/u0447_en_1_tagtts.wav) | ## Limitations - `[sigh]`, `[breath]` and `[quick_breath]` are still weak; `[lipsmack]`, `[mn]`, `[noise]`, `[accent]`, `[clucking]`, `[hissing]`, `[vocalized-noise]` were not trained (`[vocalized-noise]` can hurt pronunciation). - `` works in about 40% of cases. Stress is the hardest target and the judge is strict. - Tags work best where the sentence supports them (a laugh after a joke, a sigh after bad news). - Tongue twisters (Seed-TTS zh-hard) are slightly worse than the official model (CER 6.00 → 6.24). - `[cough]` and `[breath]` hit rates moved down by one or two of 24 bench utterances in v1.1 (within noise). - Evaluated on Chinese and English only. ## Related - **DuplexJev** – full-duplex speech-understanding models that hear emotion, stress, non-verbal sounds and turn-taking; the judge used to train tag-tts. [GitHub](https://github.com/adventists-ai/duplexjev) · [Hugging Face](https://huggingface.co/adventists-ai/DuplexJev-32B-Para) · [ModelScope](https://modelscope.cn/models/AdventistsAI/DuplexJev-32B-Para) - Base model: [Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) ## Changelog - **v1.1 (2026-10-30)** – two GRPO rounds with DuplexJev rewards and text-guided positives for stress. Stress 35.0% → 40.7%, sigh 15.0% → 17.9% (held-out), zh-hard 6.42 → 6.24. v1.0 weights stay available as the `v1.0` revision. - **v1.0 (2026-10-29)** – three DPO rounds judged by DuplexJev. ## License Apache-2.0, same as the base model. tag-tts only changes `llm.pt`; all other files are from Fun-CosyVoice3-0.5B-2512 by Tongyi Lab / FunAudioLLM. ## 中文说明 **tag-tts 让 CosyVoice3 的标签真正生效。** CosyVoice3 支持在文本里插入 `[laughter]`、`[cough]`、`[sigh]`、`…`、`…` 等副语言标签，但在实时逐字双流合成里，大部分标签会被忽略。tag-tts 只替换它的语言模型（`llm.pt`）， 用强化学习训练，让“写了笑就真的笑出来”。 强化学习的裁判是我们的副语言语音理解模型 **[DuplexJev](https://github.com/adventists-ai/duplexjev)**： 它逐条听合成结果，判断笑、咳嗽、叹气有没有真的出现，以及**重音落在哪个词上**。 - 笑声：DuplexJev 判定命中 62% → **100%**；成对写法 `…` 12% → **88%** - 咳嗽：42% → **71%**（v1.0 79%，在误差内）；叹气（留出句）：4.3% → **17.9%** - 重音（留出句，一个词加 ``）：重音落在目标词上 28.6% → **40.7%**（v1.0 为 35.0%） - 基础能力不退化：Seed-TTS-eval 中文字错率 1.53 → **1.27**，英文 2.58 → **1.96**（绕口令难句 6.00 → 6.24，略差） - 同时包含我们此前的整句情绪指令强化学习：情绪指令的命中率（emotion2vec）49.4% → **66.2%**（ESD 克隆音色 47.5% → **71.9%**） 训练方法：每句插入标签 → 当前模型在线上同款流式设置下采样 8 个候选 → DuplexJev-32B 做选择题打分 （AudioSet-AST 辅助）→ 念错字、音色或语速跑偏的直接判最差 → 组内好坏对做 DPO，三轮（v1.0）。v1.1 再做两轮 GRPO：对模型 12 次都读不出重音的句子，采样时先在目标词前加停顿作引导，挑出 DuplexJev 判为重读的样本，训练时换回原文本，让模型学会只看 `` 就重读。只更新 0.5B 语言模型， flow 与声码器不变；训练只用模型自己生成的音频。 使用注意： 1. 不带指令时前缀必须是 `You are a helpful assistant.<|endofprompt|>`，`<|endofprompt|>` 前多一个空格会让大部分标签失效； 2. 流式送字时，标签要作为一整块送入，拆开的标签会被念出来； 3. 只使用白名单标签，格式错误的标签会被念出来。 许可：Apache-2.0，与基座模型一致。除 `llm.pt` 外，其余文件均来自通义实验室 FunAudioLLM 的 Fun-CosyVoice3-0.5B-2512。