---
title: zipformer-large-streaming
canonical_url: "https://www.modelscope.cn/models/pkufool/zipformer-large-streaming"
md_url: "https://www.modelscope.cn/models/pkufool/zipformer-large-streaming.md"
repository: pkufool/zipformer-large-streaming
last_updated: 2026-06-18
license: "Apache License 2.0"
pipeline_tag: auto-speech-recognition
tasks:
  - auto-speech-recognition
library_name:
  - onnx
  - pytorch
frameworks:
  - Pytorch
downloads: 1
stars: 0
---

# zipformer-large-streaming

> zipformer-large-streaming - pkufool 在 ModelScope 开源的模型。This is a large streaming zipformer model developed by Xiaomi AI Lab Next-gen-Kaldi team. The model was trained on around 20,0000 hours of open-sourced Chinese and English datasets. The number of…

pkufool/zipformer-large-streaming 是 ModelScope 魔搭社区上的auto-speech-recognition模型，采用 Apache License 2.0 许可。

- **Repository**: pkufool/zipformer-large-streaming
- **License**: Apache License 2.0
- **Tasks**: auto-speech-recognition
- **Downloads**: 1
- **Stars**: 0
- **Last updated**: 2026-06-18

Source: https://www.modelscope.cn/models/pkufool/zipformer-large-streaming

---

## Introduction

This is a large streaming [zipformer](https://arxiv.org/pdf/2310.11230) model developed by Xiaomi AI Lab Next-gen-Kaldi team. The model was trained on around 20,0000 hours of open-sourced
Chinese and English datasets. The number of parameters is around 150M.

The performance on some popular test sets (CER for Chinese, WER for English).

> The chunk-size=16 and left-context-frames=128

| Head | aishell test 1 / 2 |  wenetspeech test-net/meetting | Common Voice zh | kespeech test | librispeech test-clean / other | gigaspeech test | Common voice en | tedium test |
| -- | -- | -- | -- | -- | -- | -- | -- | -- |
| CTC        | 3.78 / 4.71 | 8.65 / 10.54 | 11.8 | 15.35 | 3.74 / 8.5 | 12.32 | 19.7 | 10.92 |
| Transducer | 3.53 / 4.48 | 8.31 / 10.27 | 11.99| 14.83 | 3.26 / 7.51 | 11.77| 17.53| 10.82 |

Please refer to [zipformer in github](https://github.com/pkufool/zipformer) for model details.

> Training set list: Librispeech, Gigaspeech, Commonvoice-2022(zh + en), Libriheavy, Emilia (zh+en), AIshell 2, Wenetspeech, Wenetspeech4tts, Kespeech, AIshell, aidatatang, aishell4, alimeeting, magicdata, primewords, stcmds, thchs30.


## Download model

### By SDK

```
# install modelscope
pip install modelscope

# download the model
from modelscope import snapshot_download
model_dir = snapshot_download('pkufool/zipformer-large-streaming')
```

### By git
```
git clone https://www.modelscope.cn/pkufool/zipformer-large-streaming.git
```


## Inference

### Install zipformer
```
pip install zipformer
```

### Infer with CTC head

#### Examples (with downloaded model)

> You have to download the model by youself and provide the model path.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --model zipformer-large-streaming/jit_model-chunk-32-left-128.pt \
    --ctc 1 \
    --model-type jit \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --model zipformer-large-streaming/ctc-chunk-32-left-128.onnx \
    --ctc 1 \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx fp16 model
zipformer inference \
    --model zipformer-large-streaming/ctc-chunk-32-left-128.fp16.onnx \
    --ctc 1 \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx int8 model
zipformer inference \
    --model zipformer-large-streaming/ctc-chunk-32-left-128.int8.onnx \
    --ctc 1 \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav
```

#### Example (with model name in modelscope)

> You don't have to download the model, the model will be downloaded automatically from modelscope.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --ctc 1 \
    --model-type jit \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --ctc 1 \
    --model-type onnx \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx fp16 model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --ctc 1 \
    --model-type onnx \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx int8 model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --ctc 1 \
    --model-type onnx \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav
```

### Infer with transducer head

#### Examples (with downloaded model)

> You have to download the model by youself and provide the model path.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --model zipformer-large-streaming/jit_model-chunk-32-left-128.pt \
    --model-type jit \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --encoder zipformer-large-streaming/encoder-chunk-32-left-128.onnx \
    --decoder zipformer-large-streaming/decoder-chunk-32-left-128.onnx \
    --joiner zipformer-large-streaming/joiner-chunk-32-left-128.onnx \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx fp16 model
Note: we don't use fp16 decoder model here.
zipformer inference \
    --encoder zipformer-large-streaming/encoder-chunk-32-left-128.fp16.onnx \
    --decoder zipformer-large-streaming/decoder-chunk-32-left-128.onnx \
    --joiner zipformer-large-streaming/joiner-chunk-32-left-128.fp16.onnx \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav
```

#### Example (with model name in modelscope)

> You don't have to download the model, the model will be downloaded automatically from modelscope.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --model-type jit \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --model-type onnx \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx fp16 model
zipformer inference \
    --ms-model pkufool/zipformer-large-streaming \
    --model-type onnx \
    --dtype fp16 \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav
```

## Training

### Install zipformer
```
pip install zipformer
```

```
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

zipformer train \
    --world-size 8 \
    --master-port 12361 \
    --exp-dir zipformer/exp_large \
    --num-encoder-layers 2,2,4,5,4,2 \
    --feedforward-dim 512,768,1536,2048,1536,768 \
    --encoder-dim 192,256,512,768,512,256 \
    --encoder-unmasked-dim 192,192,256,320,256,192 \
    --causal 1 \
    --bpe-model zh-en-8776 \
    --training-sets data/training_set.lst \
    --num-epochs 10 \
    --use-fp16 1 \
    --start-epoch 1 \
    --use-cr-ctc 1 \
    --use-ctc 1 \
    --base-lr 0.035 \
    --use-transducer 1 \
    --use-attention-decoder 0 \
    --enable-spec-aug 0 \
    --cr-loss-scale 0.02 \
    --time-mask-ratio 2.5 \
    --lr-hours 100000 \
    --num-workers 2 \
    --max-duration 200
```

## Export model

### Install zipformer
```
pip install zipformer
```


```
# The example uses chunk-size=32 and left-context-frames=128

zipformer export \
    --use-ctc 1 \
    --use-transducer 1 \
    --num-encoder-layers 2,2,4,5,4,2 \
    --feedforward-dim 512,768,1536,2048,1536,768 \
    --encoder-dim 192,256,512,768,512,256 \
    --encoder-unmasked-dim 192,192,256,320,256,192 \
    --causal \
    --chunk-size 32 \
    --left-context-frames 128 \
    --exp-dir zipformer/exp_large \
    --bpe-model zh-en-8776 \
    --iter ITER \
    --avg AVG
```

## Citation

```
@inproceedings{yao2024zipformer,
  title={Zipformer: A faster and better encoder for automatic speech recognition},
  author={Yao, Zengwei and Guo, Liyong and Yang, Xiaoyu and Kang, Wei and Kuang, Fangjun and Yang, Yifan and Jin, Zengrui and Lin, Long and Povey, Daniel},
  booktitle={International Conference on Learning Representations},
  volume={2024},
  pages={44440--44455},
  year={2024}
}

@inproceedings{yao2025cr,
  title={Cr-ctc: Consistency regularization on ctc for improved speech recognition},
  author={Yao, Zengwei and Kang, Wei and Yang, Xiaoyu and Kuang, Fangjun and Guo, Liyong and Zhu, Han and Jin, Zengrui and Li, Zhaoqing and Lin, Long and Povey, Daniel},
  booktitle={International Conference on Learning Representations},
  volume={2025},
  pages={26850--26868},
  year={2025}
}
```