---
title: VocalNet-1B
canonical_url: "https://www.modelscope.cn/models/VocalNet/VocalNet-1B"
md_url: "https://www.modelscope.cn/models/VocalNet/VocalNet-1B.md"
repository: VocalNet/VocalNet-1B
last_updated: 2025-04-22
license: "Apache License 2.0"
model_type:
- omni_speech2s_llama
architectures:
- OmniSpeech2SLlamaForCausalLM
base_model:
- LLM-Research/Llama-3.2-1B-Instruct
base_model_relation: finetune
parameters: 2.7B
tensor_type:
- BF16
library_name:
- safetensors
- pytorch
frameworks:
- Pytorch
language:
- en
downloads: 47
stars: 0
---
# VocalNet-1B
> VocalNet-1B - VocalNet 在 ModelScope 开源的模型。🎧 VocalNet-1B Model Card
VocalNet/VocalNet-1B 是 ModelScope 魔搭社区上的 2.7B 参数机器学习模型，采用 Apache License 2.0 许可，基于 LLM-Research/Llama-3.2-1B-Instruct 构建。
- **Repository**: VocalNet/VocalNet-1B
- **License**: Apache License 2.0
- **Parameters**: 2.7B
- **Base model**: LLM-Research/Llama-3.2-1B-Instruct
- **Downloads**: 47
- **Stars**: 0
- **Last updated**: 2025-04-22
Source: https://www.modelscope.cn/models/VocalNet/VocalNet-1B
---
## 🎧 VocalNet-1B Model Card
**VocalNet-1B** is a high-performance, low-latency speech large language model (LLM) designed for real-time voice interaction. Built upon [LLaMA-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct), it leverages **multi-token prediction (MTP)** to boost generation speed and quality, matching mainstream speech LLMs with fewer parameters. 🚀
### 📂 Code and Model Access
- **GitHub**: [VocalNet Repository](https://github.com/SJTU-OmniAgent/VocalNet) 🌐
- **HuggingFace**: [VocalNet/VocalNet-1B](https://huggingface.co/VocalNet/VocalNet-1B) 🤗
- **ModelScope**: [VocalNet/VocalNet-1B](https://www.modelscope.cn/models/VocalNet/VocalNet-1B) 🔮
### 🔧 Repository Download and Environment Setup
To get started with **VocalNet-1B**, clone the repository and set up the environment as follows. 🛠️
1. **Clone the Repository**:
```bash
git clone https://github.com/SJTU-OmniAgent/VocalNet.git
cd VocalNet
```
2. **Create and Activate Environment**:
```bash
conda create -n vocalnet python==3.10
conda activate vocalnet
```
3. **Install Dependencies**:
```bash
pip install --upgrade pip
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -e .
```
4. **Optional: Install Training Packages**:
If you plan to train the model, install additional packages:
```bash
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
```
### 📥 Download Instructions
**Via ModelScope SDK**:
```bash
pip install modelscope
```
```python
from modelscope import snapshot_download
model_dir = snapshot_download('VocalNet/VocalNet-1B')
```
**Via Git**:
```bash
git clone https://www.modelscope.cn/VocalNet/VocalNet-1B.git
```
### 🛠️ Dependencies
- **Speech Encoder**: [Whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 🎤
- **Vocoder**: [CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) for converting speech tokens to audio waveforms. 🔊
### 🔄 Local Inference
To perform inference with **VocalNet-1B**, follow these steps to set up and run the model locally. 📡
1. **Model Preparation**:
- Download **VocalNet-1B** from [HuggingFace](https://huggingface.co/VocalNet/VocalNet-1B) or [ModelScope](https://www.modelscope.cn/models/VocalNet/VocalNet-1B). 📦
- Download the **Whisper-large-v3** speech encoder from [HuggingFace](https://huggingface.co/openai/whisper-large-v3) and place it in the `./models/speech_encoder/` directory. 🎤
2. **CosyVoice Preparation**:
- VocalNet-1B uses **CosyVoice2-0.5B** to convert generated speech tokens into audio waveforms. Download it from [HuggingFace](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B). 🔊
3. **Path Modification**:
- Update the paths in `omni_speech/infer/vocalnet.py` to point to the downloaded models:
```python
COSYVOICE_MODEL="" # Path to CosyVoice2-0.5B, e.g., /workspace/CosyVoice/pretrained_models/CosyVoice2-0.5B-VocalNet
VOCALNET_MODEL="" # Path to VocalNet-1B, e.g., ./checkpoints/VocalNet-1B
```
4. **Run Inference**:
- For **speech-to-text (S2T)** inference:
```bash
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav
```
- For **speech-to-speech (S2S)** inference:
```bash
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav --s2s --save_dir ./
```
### 📊 Performance Evaluation
VocalNet-1B was evaluated on [OpenAudioBench](https://huggingface.co/datasets/baichuan-inc/OpenAudioBench), covering AlpacaEval, LLaMA Questions, TriviaQA, and Web Questions. **Bold** indicates the optimal result in each subgroup.
#### Overall Performance
| Model |
LLM Size |
Modality |
AlpacaEval |
LLaMA Questions |
TriviaQA |
Web Questions |
| Tiny Models |
| Mini-Omni |
0.5B |
s→t |
1.84 |
2.7 |
0.12 |
0.22 |
| s→s |
1.80 |
2.7 |
0.08 |
0.20 |
| SLAM-Omni |
0.5B |
s→t |
3.50 |
29.4 |
0.39 |
0.84 |
| s→s |
3.01 |
26.7 |
0.34 |
0.69 |
| VocalNet-1B (VA) |
1B |
s→t |
5.38 |
70.3 |
3.38 |
4.93 |
| s→s |
4.83 |
61.0 |
2.78 |
4.47 |
| VocalNet-1B |
1B |
s→t |
5.79 |
71.7 |
3.60 |
5.16 |
| s→s |
5.03 |
63.7 |
3.06 |
4.68 |
#### Response Alignment and Acoustic Quality
| Model |
AlpacaEval |
LLaMA Questions |
TriviaQA |
Web Questions |
Avg |
| WER |
UTMOS |
WER |
UTMOS |
WER |
UTMOS |
WER |
UTMOS |
WER |
UTMOS |
| Tiny Models |
| Mini-Omni |
20.78 |
4.429 |
5.20 |
4.428 |
7.43 |
4.428 |
8.51 |
4.433 |
8.66 |
4.430 |
| SLAM-Omni |
5.52 |
4.439 |
5.55 |
4.467 |
6.16 |
4.470 |
6.50 |
4.461 |
6.17 |
4.464 |
| VocalNet-1B (VA) |
3.43 |
4.495 |
3.65 |
4.498 |
5.97 |
4.499 |
6.40 |
4.489 |
5.66 |
4.495 |
| VocalNet-1B |
3.43 |
4.491 |
3.27 |
4.497 |
6.73 |
4.486 |
4.88 |
4.493 |
5.31 |
4.491 |
### ✍️ Citation
If you use VocalNet-1B, please cite:
```bib
@article{wang2025vocalnet,
title={VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation},
author={Wang, Yuhao and Liu, Heyang and Cheng, Ziyang and Wu, Ronghua and Gu, Qunshan and Wang, Yanfeng and Wang, Yu},
journal={arXiv preprint arXiv:2504.04060},
year={2025}
}
```