---
title: zipformer-medium-streaming
canonical_url: "https://www.modelscope.cn/models/pkufool/zipformer-medium-streaming"
md_url: "https://www.modelscope.cn/models/pkufool/zipformer-medium-streaming.md"
repository: pkufool/zipformer-medium-streaming
last_updated: 2026-06-21
license: "Apache License 2.0"
pipeline_tag: auto-speech-recognition
tasks:
  - auto-speech-recognition
library_name:
  - onnx
  - pytorch
frameworks:
  - Pytorch
downloads: 4
stars: 0
---

# zipformer-medium-streaming

> zipformer-medium-streaming - pkufool 在 ModelScope 开源的模型。This is a medium streaming zipformer model developed by Xiaomi AI Lab Next-gen-Kaldi team. The model was trained on around 20,0000 hours of open-sourced Chinese and English datasets. The number of…

pkufool/zipformer-medium-streaming 是 ModelScope 魔搭社区上的auto-speech-recognition模型，采用 Apache License 2.0 许可。

- **Repository**: pkufool/zipformer-medium-streaming
- **License**: Apache License 2.0
- **Tasks**: auto-speech-recognition
- **Downloads**: 4
- **Stars**: 0
- **Last updated**: 2026-06-21

Source: https://www.modelscope.cn/models/pkufool/zipformer-medium-streaming

---

## Introduction

This is a medium streaming [zipformer](https://arxiv.org/pdf/2310.11230) model developed by Xiaomi AI Lab Next-gen-Kaldi team. The model was trained on around 20,0000 hours of open-sourced
Chinese and English datasets. The number of parameters is around 68M (for ctc head), 73M (for transducer head).

The performance on some popular test sets (CER for Chinese, WER for English).

> The chunk-size=16 and left-context-frames=128

| Head | aishell test 1 / 2 |  wenetspeech test-net/meetting | Common Voice zh | kespeech test | librispeech test-clean / other | gigaspeech test | Common voice en | tedium test |
| -- | -- | -- | -- | -- | -- | -- | -- | -- |
| CTC        | 4.46 / 5.09 | 9.74 / 11.21 | 12.68| 11.26 | 4.28 / 9.4 | 12.96 | 21.77| 11.26 |
| Transducer | 3.9 / 4.79  | 9.05 / 10.82 | 12.41| 17.89 | 3.64 / 8.08 | 12.13| 18.97| 10.9  |


Please refer to [zipformer in github](https://github.com/pkufool/zipformer) for model details.

> Training set list: Librispeech, Gigaspeech, Commonvoice-2022(zh + en), Libriheavy, Emilia (zh+en), AIshell 2, Wenetspeech, Wenetspeech4tts, Kespeech, AIshell, aidatatang, aishell4, alimeeting, magicdata, primewords, stcmds, thchs30.


## Download model

### By SDK

```
# install modelscope
pip install modelscope

# download the model
from modelscope import snapshot_download
model_dir = snapshot_download('pkufool/zipformer-medium-streaming')
```

### By git
```
git clone https://www.modelscope.cn/pkufool/zipformer-medium-streaming.git
```


## Inference

### Install zipformer
```
pip install zipformer
```

### Infer with CTC head

#### Examples (with downloaded model)

> You have to download the model by youself and provide the model path.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --model zipformer-medium-streaming/jit_model-chunk-32-left-128.pt \
    --ctc 1 \
    --model-type jit \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --model zipformer-medium-streaming/ctc-chunk-32-left-128.onnx \
    --ctc 1 \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx fp16 model
zipformer inference \
    --model zipformer-medium-streaming/ctc-chunk-32-left-128.fp16.onnx \
    --ctc 1 \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx int8 model
zipformer inference \
    --model zipformer-medium-streaming/ctc-chunk-32-left-128.int8.onnx \
    --ctc 1 \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav
```

#### Example (with model name in modelscope)

> You don't have to download the model, the model will be downloaded automatically from modelscope.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --ctc 1 \
    --model-type jit \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --ctc 1 \
    --model-type onnx \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx fp16 model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --ctc 1 \
    --model-type onnx \
    --dtype fp16 \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx int8 model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --ctc 1 \
    --model-type onnx \
    --dtype fp16 \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav
```

### Infer with transducer head

#### Examples (with downloaded model)

> You have to download the model by youself and provide the model path.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --model zipformer-medium-streaming/jit_model-chunk-32-left-128.pt \
    --model-type jit \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --encoder zipformer-medium-streaming/encoder-chunk-32-left-128.onnx \
    --decoder zipformer-medium-streaming/decoder-chunk-32-left-128.onnx \
    --joiner zipformer-medium-streaming/joiner-chunk-32-left-128.onnx \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav

# onnx fp16 model
# We don't use fp16 decoder model here.
zipformer inference \
    --encoder zipformer-medium-streaming/encoder-chunk-32-left-128.fp16.onnx \
    --decoder zipformer-medium-streaming/decoder-chunk-32-left-128.onnx \
    --joiner zipformer-medium-streaming/joiner-chunk-32-left-128.fp16.onnx \
    --model-type onnx \
    --tokens data/tokens.txt \
    data/en.wav data/zh.wav
```

#### Example (with model name in modelscope)

> You don't have to download the model, the model will be downloaded automatically from modelscope.

```
# The example uses chunk-size=32 and left-context-frames=128

# jit script model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --model-type jit \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --model-type onnx \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav

# onnx fp16 model
zipformer inference \
    --ms-model pkufool/zipformer-medium-streaming \
    --model-type onnx \
    --dtype fp16 \
    --chunk-size 32 \
    --left-context-frames 128 \
    data/en.wav data/zh.wav
```

## Training

### Install zipformer
```
pip install zipformer
```

```
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

zipformer train \
    --world-size 8 \
    --master-port 12361 \
    --exp-dir zipformer/exp_medium \
    --num-encoder-layers 2,2,3,4,3,2 \
    --feedforward-dim 512,768,1024,1536,1024,768 \
    --encoder-dim 192,256,384,512,384,256 \
    --encoder-unmasked-dim 192,192,256,256,256,192 \
    --causal 1 \
    --bpe-model zh-en-8776 \
    --training-sets data/training_set.lst \
    --num-epochs 10 \
    --use-fp16 1 \
    --start-epoch 1 \
    --use-cr-ctc 1 \
    --use-ctc 1 \
    --base-lr 0.035 \
    --use-transducer 1 \
    --use-attention-decoder 0 \
    --enable-spec-aug 0 \
    --cr-loss-scale 0.02 \
    --time-mask-ratio 2.5 \
    --lr-hours 100000 \
    --num-workers 2 \
    --max-duration 200
```

## Export model

### Install zipformer
```
pip install zipformer
```


```
# The example uses chunk-size=32 and left-context-frames=128

zipformer export \
    --use-ctc 1 \
    --use-transducer 1 \
    --num-encoder-layers 2,2,3,4,3,2 \
    --feedforward-dim 512,768,1024,1536,1024,768 \
    --encoder-dim 192,256,384,512,384,256 \
    --encoder-unmasked-dim 192,192,256,256,256,192 \
    --causal 1 \
    --chunk-size 32 \
    --left-context-frames 128 \
    --exp-dir zipformer/exp_medium \
    --bpe-model zh-en-8776 \
    --iter ITER \
    --avg AVG
```

## Citation

```
@inproceedings{yao2024zipformer,
  title={Zipformer: A faster and better encoder for automatic speech recognition},
  author={Yao, Zengwei and Guo, Liyong and Yang, Xiaoyu and Kang, Wei and Kuang, Fangjun and Yang, Yifan and Jin, Zengrui and Lin, Long and Povey, Daniel},
  booktitle={International Conference on Learning Representations},
  volume={2024},
  pages={44440--44455},
  year={2024}
}

@inproceedings{yao2025cr,
  title={Cr-ctc: Consistency regularization on ctc for improved speech recognition},
  author={Yao, Zengwei and Kang, Wei and Yang, Xiaoyu and Kuang, Fangjun and Guo, Liyong and Zhu, Han and Jin, Zengrui and Li, Zhaoqing and Lin, Long and Povey, Daniel},
  booktitle={International Conference on Learning Representations},
  volume={2025},
  pages={26850--26868},
  year={2025}
}
```