---
title: VocalNet-8B
canonical_url: "https://www.modelscope.cn/models/VocalNet/VocalNet-8B"
md_url: "https://www.modelscope.cn/models/VocalNet/VocalNet-8B.md"
repository: VocalNet/VocalNet-8B
last_updated: 2025-04-22
license: "Apache License 2.0"
model_type:
- omni_speech2s_llama
architectures:
- OmniSpeech2SLlamaForCausalLM
base_model:
- LLM-Research/Meta-Llama-3.1-8B-Instruct
base_model_relation: finetune
parameters: 11.8B
tensor_type:
- BF16
library_name:
- safetensors
- pytorch
frameworks:
- Pytorch
downloads: 62
stars: 0
---
# VocalNet-8B
> VocalNet-8B - VocalNet 在 ModelScope 开源的模型。🎧 VocalNet-8B Model Card
VocalNet/VocalNet-8B 是 ModelScope 魔搭社区上的 11.8B 参数机器学习模型，采用 Apache License 2.0 许可，基于 LLM-Research/Meta-Llama-3.1-8B-Instruct 构建。
- **Repository**: VocalNet/VocalNet-8B
- **License**: Apache License 2.0
- **Parameters**: 11.8B
- **Base model**: LLM-Research/Meta-Llama-3.1-8B-Instruct
- **Downloads**: 62
- **Stars**: 0
- **Last updated**: 2025-04-22
Source: https://www.modelscope.cn/models/VocalNet/VocalNet-8B
---
## 🎧 VocalNet-8B Model Card
**VocalNet-8B** is a high-performance, low-latency speech large language model (LLM) optimized for real-time voice interaction. Built upon [LLaMA-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), it employs **multi-token prediction (MTP)** to significantly enhance generation speed and quality, surpassing most mainstream speech and omni-modal LLMs. 🚀
### 📂 Code and Model Access
- **GitHub**: [VocalNet Repository](https://github.com/SJTU-OmniAgent/VocalNet) 🌐
- **HuggingFace**: [VocalNet/VocalNet-8B](https://huggingface.co/VocalNet/VocalNet-8B) 🤗
- **ModelScope**: [VocalNet/VocalNet-8B](https://www.modelscope.cn/models/VocalNet/VocalNet-8B) 🔮
### 🔧 Repository Download and Environment Setup
To get started with **VocalNet-8B**, clone the repository and set up the environment as follows. 🛠️
1. **Clone the Repository**:
```bash
git clone https://github.com/SJTU-OmniAgent/VocalNet.git
cd VocalNet
```
2. **Create and Activate Environment**:
```bash
conda create -n vocalnet python==3.10
conda activate vocalnet
```
3. **Install Dependencies**:
```bash
pip install --upgrade pip
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -e .
```
4. **Optional: Install Training Packages**:
If you plan to train the model, install additional packages:
```bash
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
```
### 📥 Download Instructions
**Via ModelScope SDK**:
```bash
pip install modelscope
```
```python
from modelscope import snapshot_download
model_dir = snapshot_download('VocalNet/VocalNet-8B')
```
**Via Git**:
```bash
git clone https://www.modelscope.cn/VocalNet/VocalNet-8B.git
```
### 🛠️ Dependencies
- **Speech Encoder**: [Whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 🎤
- **Vocoder**: [CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) for converting speech tokens to audio waveforms. 🔊
### 🔄 Local Inference
To perform inference with **VocalNet-8B**, follow these steps to set up and run the model locally. 📡
1. **Model Preparation**:
- Download **VocalNet-8B** from [HuggingFace](https://huggingface.co/VocalNet/VocalNet-8B) or [ModelScope](https://www.modelscope.cn/models/VocalNet/VocalNet-8B). 📦
- Download the **Whisper-large-v3** speech encoder from [HuggingFace](https://huggingface.co/openai/whisper-large-v3) and place it in the `./models/speech_encoder/` directory. 🎤
2. **CosyVoice Preparation**:
- VocalNet-8B uses **CosyVoice2-0.5B** to convert generated speech tokens into audio waveforms. Download it from [HuggingFace](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B). 🔊
3. **Path Modification**:
- Update the paths in `omni_speech/infer/vocalnet.py` to point to the downloaded models:
```python
COSYVOICE_MODEL="" # Path to CosyVoice2-0.5B, e.g., /workspace/CosyVoice/pretrained_models/CosyVoice2-0.5B-VocalNet
VOCALNET_MODEL="" # Path to VocalNet-8B, e.g., ./checkpoints/VocalNet-8B
```
4. **Run Inference**:
- For **speech-to-text (S2T)** inference:
```bash
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav
```
- For **speech-to-speech (S2S)** inference:
```bash
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav --s2s --save_dir ./
```
### 📊 Performance Evaluation
VocalNet-8B was evaluated on [OpenAudioBench](https://huggingface.co/datasets/baichuan-inc/OpenAudioBench), covering AlpacaEval, LLaMA Questions, TriviaQA, and Web Questions. **Bold** indicates the optimal result in each subgroup.
#### Overall Performance
| Model |
LLM Size |
Modality |
AlpacaEval |
LLaMA Questions |
TriviaQA |
Web Questions |
| Base Models |
| LLaMA-Omni |
8B |
s→t |
5.31 |
69.7 |
4.44 |
5.44 |
| s→s |
3.89 |
55.1 |
2.44 |
4.00 |
| Freeze-Omni |
7B |
s→t |
4.51 |
77.7 |
5.32 |
6.41 |
| s→s |
2.99 |
60.2 |
3.53 |
4.78 |
| GLM-4-Voice |
9B |
s→t |
5.86 |
77.4 |
4.95 |
5.56 |
| s→s |
5.27 |
64.3 |
4.63 |
5.40 |
| Baichuan-Omni-1.5 |
7B |
s→t |
5.20 |
77.6 |
5.72 |
6.12 |
| s→s |
4.10 |
61.2 |
4.13 |
5.18 |
| MiniCPM-o |
8B |
s→t |
6.13 |
77.2 |
6.43 |
7.16 |
| s→s |
4.95 |
65.8 |
4.99 |
6.22 |
| Minmo* |
8B |
s→t |
- |
78.9 |
4.83 |
5.50 |
| s→s |
6.48 |
64.1 |
3.75 |
3.99 |
| Qwen2.5-Omni |
8B |
s→t |
6.01 |
79.0 |
5.89 |
6.88 |
| s→s |
5.73 |
76.3 |
5.59 |
6.70 |
| VocalNet-8B (VA) |
8B |
s→t |
7.05 |
77.1 |
6.15 |
6.34 |
| s→s |
6.30 |
71.4 |
5.24 |
5.81 |
| VocalNet-8B |
8B |
s→t |
7.12 |
79.5 |
6.24 |
6.48 |
| s→s |
6.37 |
73.1 |
5.67 |
6.16 |
#### Response Alignment and Acoustic Quality
| Model |
AlpacaEval |
LLaMA Questions |
TriviaQA |
Web Questions |
Avg |
| WER |
UTMOS |
WER |
UTMOS |
WER |
UTMOS |
WER |
UTMOS |
WER |
UTMOS |
| Base Models |
| LLaMA-Omni |
6.00 |
3.942 |
10.00 |
4.003 |
20.93 |
3.965 |
14.60 |
3.935 |
15.90 |
3.956 |
| Freeze-Omni |
14.33 |
4.377 |
14.20 |
4.417 |
20.39 |
4.404 |
18.25 |
4.398 |
18.31 |
4.401 |
| GLM-4-Voice |
18.71 |
4.025 |
14.45 |
4.152 |
8.33 |
4.306 |
6.08 |
4.214 |
8.99 |
4.228 |
| Baichuan-Omni-1.5 |
20.84 |
4.082 |
22.82 |
4.332 |
22.36 |
4.401 |
23.29 |
4.350 |
22.67 |
4.347 |
| MiniCPM-o |
15.35 |
4.102 |
5.73 |
4.228 |
8.08 |
4.128 |
8.94 |
4.125 |
8.72 |
4.137 |
| Qwen2.5-Omni |
2.41 |
4.299 |
0.93 |
4.315 |
1.13 |
4.339 |
4.68 |
4.363 |
2.63 |
4.342 |
| VocalNet-8B (VA) |
2.65 |
4.490 |
3.00 |
4.503 |
5.02 |
4.499 |
4.21 |
4.485 |
4.26 |
4.493 |
| VocalNet-8B |
4.71 |
4.489 |
2.68 |
4.500 |
4.04 |
4.482 |
3.11 |
4.492 |
3.56 |
4.489 |
### ✍️ Citation
If you use VocalNet-8B, please cite:
```bib
@article{wang2025vocalnet,
title={VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation},
author={Wang, Yuhao and Liu, Heyang and Cheng, Ziyang and Wu, Ronghua and Gu, Qunshan and Wang, Yanfeng and Wang, Yu},
journal={arXiv preprint arXiv:2504.04060},
year={2025}
}
```