--- title: Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN canonical_url: "https://www.modelscope.cn/models/yangdi/Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN" md_url: "https://www.modelscope.cn/models/yangdi/Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN.md" repository: yangdi/Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN chinese_name: Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN last_updated: 2026-11-10 pipeline_tag: auto-speech-recognition tasks: - auto-speech-recognition model_type: - hy_v3 architectures: - HYV3SpeechForCausalLM parameters: 30.1B tensor_type: - I64 - F32 - I32 - BF16 library_name: - safetensors downloads: 16 stars: 0 --- # Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN > Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN - yangdi 在 ModelScope 开源的模型。Hy-SimulMT-30B-A3B-Interleave模型的W4A16量化 - **Repository**: yangdi/Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN - **Tasks**: auto-speech-recognition - **Parameters**: 30.1B - **Downloads**: 16 - **Stars**: 0 - **Last updated**: 2026-11-10 Source: https://www.modelscope.cn/models/yangdi/Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN --- # 量化转换过程 本次量化基于 LLM Compressor 工具链实现，采用无校准数据的 RTN（Round to Nearest）量化方案，将语言模型主干线性层权重压缩至 4bit，激活值保持 16bit；同时保留 MoE 路由层、语言模型输出头与音频编码器不量化，在压缩显存的同时最小化精度损失。 ## 1. 执行量化转换脚本 核心思路：在 CPU 侧加载原始 BF16 模型，修正 MoE 中间层维度配置后，调用 LLM Compressor 的 oneshot 接口完成权重量化，最后导出量化 checkpoint。 完整转换脚本如下： ```python import os os.environ["CUDA_VISIBLE_DEVICES"] = "" os.environ["TOKENIZERS_PARALLELISM"] = "false" from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier MODEL_ID = "./Hy-SimulMT-30B-A3B-Interleave" SAVE_DIR = "./Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN" # ========== 配置修正 ========== # 修正MoE中间层维度，匹配模型实际结构 config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True) config.moe_intermediate_size = 768 # ========== 加载模型 ========== model = AutoModelForCausalLM.from_pretrained( MODEL_ID, config=config, trust_remote_code=True, device_map="cpu", torch_dtype="auto" ) tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True) # ========== 量化配置 ========== # W4A16权重量化，跳过路由层、门控层与输出头，避免MoE调度精度下降 recipe = QuantizationModifier( targets="Linear", scheme="W4A16", ignore=[ "lm_head", "re:.*mlp.gate$", "re:.*gate.weight$", "re:.*router.*" ] ) oneshot(model=model, recipe=recipe, tokenizer=tokenizer) # 清理生成配置默认值 if hasattr(model, "generation_config"): model.generation_config.temperature = None model.generation_config.top_p = None model.generation_config.top_k = None # 保存量化模型 model.save_pretrained(SAVE_DIR) tokenizer.save_pretrained(SAVE_DIR) print(f"量化模型已保存至: {SAVE_DIR}") ``` ### 关键日志说明 执行过程中核心输出如下，包含几处可预期的提示： ```bash # 无GPU环境提示，CPU侧量化为预期行为 2026-11-10T02:54:55.9271 | get_main_device | WARNING - No accelerator available! Compressing model on CPU instead # 音频塔权重被标记为UNEXPECTED，属正常现象：量化工具仅处理语言模型，多模态组件不参与量化 [transformers] HYV3ForCausalLM LOAD REPORT from: ./Hy-SimulMT-30B-A3B-Interleave audio_tower.* | UNEXPECTED # MoE自动线性化：工具自动将融合的3D专家权重展开为2D线性层以支持量化 2026-11-10T02:55:22.9655 | apply_recipe_modifiers | WARNING - Detected an MoE model which has not been linearized. Linearizing experts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 47/47 [00:13<00:00, 3.53it/s] # 量化完成并导出 Compressing model: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18384/18384 [00:21<00:00, 858.18it/s] 量化模型已保存至: ./Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN ``` > 注：脚本末尾的 `ResourceTracker.__del__` 报错为 Python 多进程库的已知兼容问题，不影响模型文件导出结果，可直接忽略。 ## 2. 补全原始模型组件 LLM Compressor 仅对语言模型主干进行量化处理，音频编码器、分词器等多模态组件不会被自动导出，需从原始 checkpoint 中完整复制，保障语音输入功能正常。 ```bash ORIGIN_DIR="./Hy-SimulMT-30B-A3B-Interleave" QUANT_DIR="./Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN" # 复制分词器相关文件 cp "$ORIGIN_DIR/tokenizer_config.json" "$QUANT_DIR/" cp "$ORIGIN_DIR/tokenizer.json" "$QUANT_DIR/" cp "$ORIGIN_DIR/special_tokens_map.json" "$QUANT_DIR/" # 复制音频编码器权重与预处理配置 cp "$ORIGIN_DIR/model-audio-tower.safetensors" "$QUANT_DIR/" cp "$ORIGIN_DIR/preprocessor_config.json" "$QUANT_DIR/" ``` ## 3. 补充量化配置字段 原始模型 `config.json` 不包含量化元数据，需手动添加 `quantization_config` 字段，使 vLLM 可正确识别 compressed-tensors 格式的 4bit 权重。 在 `config.json` 根节点中新增如下配置： ```json { "quantization_config": { "config_groups": { "group_0": { "format": "pack-quantized", "input_activations": null, "output_activations": null, "targets": [ "Linear" ], "weights": { "dynamic": false, "group_size": 128, "num_bits": 4, "observer": "memoryless_minmax", "strategy": "group", "symmetric": true, "type": "int" } } }, "format": "pack-quantized", "global_compression_ratio": null, "ignore": [ "model.layers.1.mlp.router.gate", "model.layers.2.mlp.router.gate", "model.layers.3.mlp.router.gate", "model.layers.4.mlp.router.gate", "model.layers.5.mlp.router.gate", "model.layers.6.mlp.router.gate", "model.layers.7.mlp.router.gate", "model.layers.8.mlp.router.gate", "model.layers.9.mlp.router.gate", "model.layers.10.mlp.router.gate", "model.layers.11.mlp.router.gate", "model.layers.12.mlp.router.gate", "model.layers.13.mlp.router.gate", "model.layers.14.mlp.router.gate", "model.layers.15.mlp.router.gate", "model.layers.16.mlp.router.gate", "model.layers.17.mlp.router.gate", "model.layers.18.mlp.router.gate", "model.layers.19.mlp.router.gate", "model.layers.20.mlp.router.gate", "model.layers.21.mlp.router.gate", "model.layers.22.mlp.router.gate", "model.layers.23.mlp.router.gate", "model.layers.24.mlp.router.gate", "model.layers.25.mlp.router.gate", "model.layers.26.mlp.router.gate", "model.layers.27.mlp.router.gate", "model.layers.28.mlp.router.gate", "model.layers.29.mlp.router.gate", "model.layers.30.mlp.router.gate", "model.layers.31.mlp.router.gate", "model.layers.32.mlp.router.gate", "model.layers.33.mlp.router.gate", "model.layers.34.mlp.router.gate", "model.layers.35.mlp.router.gate", "model.layers.36.mlp.router.gate", "model.layers.37.mlp.router.gate", "model.layers.38.mlp.router.gate", "model.layers.39.mlp.router.gate", "model.layers.40.mlp.router.gate", "model.layers.41.mlp.router.gate", "model.layers.42.mlp.router.gate", "model.layers.43.mlp.router.gate", "model.layers.44.mlp.router.gate", "model.layers.45.mlp.router.gate", "model.layers.46.mlp.router.gate", "model.layers.47.mlp.router.gate", "lm_head" ], "kv_cache_scheme": null, "moe_sparsity_config": null, "quant_method": "compressed-tensors", "quantization_status": "compressed", "sparsity_config": {}, "transform_config": {}, "version": "0.20.1" } } ``` 完成以上三步后，即可参照双卡部署的启动方式，在单卡 RTX 3090 上通过 vLLM 加载量化模型，配合 `--max-model-len` 控制上下文长度即可正常启动服务。 # 量化前后效果对比 本次评测基于 RealSI 中英双向语音翻译测试集，测试环境与基准 FP8 版本完全一致，覆盖 SSE、POLL、WebSocket 三种请求模式，从**语音识别与翻译质量**、**延迟与推理性能**两个维度进行对比。 测试集详细说明可参考：[【S2TT测试集】中英双向语音到文本翻译测试集 RealSI](https://blog.csdn.net/yd778473278/article/details/170193047) ## 1. 语音识别与翻译质量对比 | 模型版本 | 请求模式 | 中文CER | 英文WER | 英中BLEU | 英中COMET | 中英BLEU | 中英COMET | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | Hy-SimulMT-30B-A3B-Interleave-FP8 | SSE | 6.79% | 14.03% | 19.67 | 0.6457 | 29.88 | 0.6075 | | Hy-SimulMT-30B-A3B-Interleave-FP8 | POLL | 6.75% | 14.55% | 19.81 | 0.6324 | 32.21 | 0.6082 | | Hy-SimulMT-30B-A3B-Interleave-FP8 | WebSocket | 7.01% | 13.97% | 20.35 | 0.6525 | 31.58 | 0.6050 | | Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN | SSE | 7.43% | 14.32% | 19.09 | 0.6128 | 29.96 | 0.5921 | | Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN | POLL | 6.66% | 13.99% | 18.50 | 0.6250 | 27.88 | 0.5761 | | Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN | WebSocket | 7.03% | 13.72% | 18.66 | 0.6226 | 29.82 | 0.5798 | ## 2. 延迟与推理性能对比 | 模型版本 | 请求模式 | 端到端延迟(s) | 平均首字延迟(s) | 平均总处理时长(s) | | :--- | :--- | :--- | :--- | :--- | | Hy-SimulMT-30B-A3B-Interleave-FP8 | SSE | 5.05 | 23.66 | 82.61 | | Hy-SimulMT-30B-A3B-Interleave-FP8 | POLL | 5.15 | 24.86 | 78.10 | | Hy-SimulMT-30B-A3B-Interleave-FP8 | WebSocket | 2.96 | 1.69 | 288.09 | | Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN | SSE | 5.35 | 24.63 | 80.84 | | Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN | POLL | 5.27 | 25.12 | 79.95 | | Hy-SimulMT-30B-A3B-Interleave-W4A16-RTN | WebSocket | 3.20 | 1.60 | 288.06 | ## 结果总结 1. **精度损失可控**：W4A16 RTN 量化后，ASR 识别准确率基本与 FP8 基准持平，CER/WER 波动在 0.5% 以内；翻译质量 BLEU 下降约 1-2 分，COMET 下降约 0.02-0.03，属于可接受范围，满足日常同传场景使用需求。 2. **延迟影响微小**：三种请求模式下，量化后端到端延迟仅增加 0.2~0.3s，首字延迟与总处理时长无显著差异，模型实时性得到完整保留。 3. **显存收益显著**：通过 4bit 量化，模型权重显存占用降低约 50%，配合上下文长度限制，可在单张 RTX 3090（24GB）上完整部署运行，大幅降低了该同传模型的硬件门槛。