--- title: MiniMax-H3-Songyu-LoRA canonical_url: "https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Songyu-LoRA" md_url: "https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Songyu-LoRA.md" repository: mibei0804/MiniMax-H3-Songyu-LoRA chinese_name: "MiniMax-H3-宋玉-人物LoRA" last_updated: 2026-09-08 license: "Apache License 2.0" base_model: - MiniMax/MiniMax-H3 base_model_relation: adapter parameters: 37.3M tensor_type: - BF16 library_name: - pytorch - lora - safetensors downloads: 146 stars: 7 tags: - LoRA - MiniMax-H3 - video - audio - character --- # MiniMax-H3-Songyu-LoRA > MiniMax-H3-Songyu-LoRA - mibei0804 在 ModelScope 开源的模型。MiniMax-H3 Songyu LoRA mibei0804/MiniMax-H3-Songyu-LoRA 是 ModelScope 魔搭社区上的 37.3M 参数机器学习模型，采用 Apache License 2.0 许可，基于 MiniMax/MiniMax-H3 构建。 - **Repository**: mibei0804/MiniMax-H3-Songyu-LoRA - **License**: Apache License 2.0 - **Parameters**: 37.3M - **Base model**: MiniMax/MiniMax-H3 - **Tags**: LoRA, MiniMax-H3, video, audio, character - **Downloads**: 146 - **Stars**: 7 - **Last updated**: 2026-09-08 Source: https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Songyu-LoRA --- # MiniMax-H3 Songyu LoRA 给 [MiniMax-H3](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) 注入「宋玉」（《凡人修仙传》）角色身份的人物 LoRA，共训练 2000 step。加载后，在 prompt 里写上触发词即可生成该角色的视频 + 音频。 本模型同时是 [MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter)（DeCFG Training Adapter）方案的一个完整验证案例。 LoRA 权重文件：`model.safetensors` ## 触发词与 Prompt 写法 触发词： ``` 宋玉 凡人修仙传,宋玉 (凡人修仙传)， ``` MiniMax-H3 的 T2VA prompt 采用三字段结构，字段名保持字面量原样书写： - `integrated_multimodal_description`：画面描述，多镜头用 `[Shot 1]` / `[Shot 2]` 标记，可在描述里写台词（`(S1)`、`[Chinese] ...`） - `overall_soundscape`：环境音 / 音效 - `non_diegetic_music`：背景音乐，不需要时写 `N/A` 完整示例： ```text 宋玉 凡人修仙传,宋玉 (凡人修仙传)，integrated_multimodal_description: [Shot 1] 3d国漫风格，高精度cg渲染，虚幻引擎5画质，中景镜头，一位修仙女子置身于宏伟的修仙宗门庭院，背景是云雾缭绕的仙山与古建筑，远处有红色立柱与灯笼。镜头以慢速小幅度推近，她的衣料与发丝随风轻微飘动，柔和的自然日光配合电影级布光，景深效果使背景虚化。 overall_soundscape: 庭院中有轻微的山风穿过，远处传来一声清亮的风铃，衣料在风中发出细微的摩擦声。 non_diegetic_music: 缓慢的古筝旋律，音符稀疏，下方铺一层持续的低音弦乐，结尾渐弱。 ``` ## 快速开始 ### 1. 安装 DiffSynth-Studio ```bash pip install diffsynth ``` 或从源码安装： ```bash git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` ### 2. 推理 ```python import torch from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig from diffsynth.utils.data.audio_video import write_video_audio pipe = MiniMaxH3Pipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/text_encoder/model*.safetensors"), ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/transformer/model*.safetensors"), ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/video_vae/source/model.safetensors"), ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/audio_vae/model.safetensors"), ], processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"), vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2, ) pipe.load_lora(pipe.dit, ModelConfig( model_id="mibei0804/MiniMax-H3-Songyu-LoRA", origin_file_pattern="model.safetensors", )) prompt = ( "宋玉 凡人修仙传,宋玉 (凡人修仙传)，" "integrated_multimodal_description: [Shot 1] 3d国漫风格，高精度cg渲染，虚幻引擎5画质，" "中景镜头，一位修仙女子置身于宏伟的修仙宗门庭院，背景是云雾缭绕的仙山与古建筑，远处有红色立柱与灯笼。" "镜头以慢速小幅度推近，她的衣料与发丝随风轻微飘动，柔和的自然日光配合电影级布光，景深效果使背景虚化。" "\n\noverall_soundscape: 庭院中有轻微的山风穿过，远处传来一声清亮的风铃，衣料在风中发出细微的摩擦声。" "\n\nnon_diegetic_music: 缓慢的古筝旋律，音符稀疏，下方铺一层持续的低音弦乐，结尾渐弱。" ) video, audio = pipe( prompt=prompt, height=768, width=1344, num_frames=124, num_inference_steps=30, seed=0, ) write_video_audio( video=video, audio=audio, output_path="songyu.mp4", fps=24, audio_sample_rate=pipe.audio_vae.sample_rate, ) # To unload the LoRA: # pipe.clear_lora() ``` ## 训练方法 ### 使用 [MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter) 作为 preset lora MiniMax-H3 基座是经过 CFG 蒸馏的，直接在上面做 LoRA 微调容易不稳定，或者破坏蒸馏后的 CFG-free 行为。[MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter) 在训练期间把蒸馏后的 DiT 临时拉回接近蒸馏前的状态，为新 LoRA 提供更好的优化面。 **Training Adapter 只在训练时用，推理时不要加载**——本仓库发布的 `model.safetensors` 是已经脱离 Training Adapter 的独立 LoRA，直接加载到原版 MiniMax-H3 即可。 思路来源：DiffSynth-Studio 的 [Differential LoRA Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Differential_LoRA.html#using-differential-lora-training-in-the-training-framework)，以及 [ai-toolkit](https://github.com/ostris/ai-toolkit) 的 Training Adapter 概念。 ### 完整训练脚本 先下载 Training Adapter： ```bash modelscope download --model DiffSynth-Studio/MiniMax-H3-TrainingAdapter --local_dir ./models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter ``` **Stage 1：数据预处理** ```bash accelerate launch --num_processes 1 examples/minimax_h3/model_training/train.py \ --dataset_base_path "" \ --dataset_metadata_path "/metadata.json" \ --data_file_keys "video,input_audio" \ --extra_inputs "input_audio" \ --max_pixels 399360 \ --num_frames 124 \ --dataset_repeat 1 \ --model_id_with_origin_paths "MiniMax/MiniMax-H3:FL2VA/text_encoder/model*.safetensors,MiniMax/MiniMax-H3:FL2VA/video_vae/source/model.safetensors,MiniMax/MiniMax-H3:FL2VA/audio_vae/model.safetensors" \ --learning_rate 1e-4 \ --num_epochs 1 \ --remove_prefix_in_ckpt "pipe.dit." \ --output_path "./models/train/songyu_cache" \ --lora_base_model "dit" \ --lora_target_modules "blocks.25.attn.qkv_proj,blocks.25.attn.out_proj,blocks.25.mlp.fc1,blocks.25.mlp.fc2,blocks.26.attn.qkv_proj,blocks.26.attn.out_proj,blocks.26.mlp.fc1,blocks.26.mlp.fc2,blocks.27.attn.qkv_proj,blocks.27.attn.out_proj,blocks.27.mlp.fc1,blocks.27.mlp.fc2,blocks.28.attn.qkv_proj,blocks.28.attn.out_proj,blocks.28.mlp.fc1,blocks.28.mlp.fc2,blocks.29.attn.qkv_proj,blocks.29.attn.out_proj,blocks.29.mlp.fc1,blocks.29.mlp.fc2,blocks.30.attn.qkv_proj,blocks.30.attn.out_proj,blocks.30.mlp.fc1,blocks.30.mlp.fc2,blocks.31.attn.qkv_proj,blocks.31.attn.out_proj,blocks.31.mlp.fc1,blocks.31.mlp.fc2,blocks.32.attn.qkv_proj,blocks.32.attn.out_proj,blocks.32.mlp.fc1,blocks.32.mlp.fc2,blocks.33.attn.qkv_proj,blocks.33.attn.out_proj,blocks.33.mlp.fc1,blocks.33.mlp.fc2,blocks.34.attn.qkv_proj,blocks.34.attn.out_proj,blocks.34.mlp.fc1,blocks.34.mlp.fc2,blocks.35.attn.qkv_proj,blocks.35.attn.out_proj,blocks.35.mlp.fc1,blocks.35.mlp.fc2,blocks.36.attn.qkv_proj,blocks.36.attn.out_proj,blocks.36.mlp.fc1,blocks.36.mlp.fc2,blocks.37.attn.qkv_proj,blocks.37.attn.out_proj,blocks.37.mlp.fc1,blocks.37.mlp.fc2,blocks.38.attn.qkv_proj,blocks.38.attn.out_proj,blocks.38.mlp.fc1,blocks.38.mlp.fc2,blocks.39.attn.qkv_proj,blocks.39.attn.out_proj,blocks.39.mlp.fc1,blocks.39.mlp.fc2,blocks.40.attn.qkv_proj,blocks.40.attn.out_proj,blocks.40.mlp.fc1,blocks.40.mlp.fc2,blocks.41.attn.qkv_proj,blocks.41.attn.out_proj,blocks.41.mlp.fc1,blocks.41.mlp.fc2,blocks.42.attn.qkv_proj,blocks.42.attn.out_proj,blocks.42.mlp.fc1,blocks.42.mlp.fc2,blocks.43.attn.qkv_proj,blocks.43.attn.out_proj,blocks.43.mlp.fc1,blocks.43.mlp.fc2,blocks.44.attn.qkv_proj,blocks.44.attn.out_proj,blocks.44.mlp.fc1,blocks.44.mlp.fc2,blocks.45.attn.qkv_proj,blocks.45.attn.out_proj,blocks.45.mlp.fc1,blocks.45.mlp.fc2,blocks.46.attn.qkv_proj,blocks.46.attn.out_proj,blocks.46.mlp.fc1,blocks.46.mlp.fc2,blocks.47.attn.qkv_proj,blocks.47.attn.out_proj,blocks.47.mlp.fc1,blocks.47.mlp.fc2,blocks.48.attn.qkv_proj,blocks.48.attn.out_proj,blocks.48.mlp.fc1,blocks.48.mlp.fc2,blocks.49.attn.qkv_proj,blocks.49.attn.out_proj,blocks.49.mlp.fc1,blocks.49.mlp.fc2" \ --lora_rank 16 \ --use_gradient_checkpointing \ --task "sft:data_process" ``` **Stage 2：训练 LoRA** ```bash accelerate launch --num_processes 1 examples/minimax_h3/model_training/train.py \ --dataset_base_path ./models/train/songyu_cache \ --data_file_keys "video,input_audio" \ --extra_inputs "input_audio" \ --max_pixels 399360 \ --num_frames 124 \ --dataset_repeat 100 \ --model_id_with_origin_paths "MiniMax/MiniMax-H3:FL2VA/transformer/model*.safetensors" \ --learning_rate 1e-4 \ --num_epochs 1000 \ --remove_prefix_in_ckpt "pipe.dit." \ --output_path "./models/train/songyu" \ --lora_base_model "dit" \ --lora_target_modules "blocks.25.attn.qkv_proj,blocks.25.attn.out_proj,blocks.25.mlp.fc1,blocks.25.mlp.fc2,blocks.26.attn.qkv_proj,blocks.26.attn.out_proj,blocks.26.mlp.fc1,blocks.26.mlp.fc2,blocks.27.attn.qkv_proj,blocks.27.attn.out_proj,blocks.27.mlp.fc1,blocks.27.mlp.fc2,blocks.28.attn.qkv_proj,blocks.28.attn.out_proj,blocks.28.mlp.fc1,blocks.28.mlp.fc2,blocks.29.attn.qkv_proj,blocks.29.attn.out_proj,blocks.29.mlp.fc1,blocks.29.mlp.fc2,blocks.30.attn.qkv_proj,blocks.30.attn.out_proj,blocks.30.mlp.fc1,blocks.30.mlp.fc2,blocks.31.attn.qkv_proj,blocks.31.attn.out_proj,blocks.31.mlp.fc1,blocks.31.mlp.fc2,blocks.32.attn.qkv_proj,blocks.32.attn.out_proj,blocks.32.mlp.fc1,blocks.32.mlp.fc2,blocks.33.attn.qkv_proj,blocks.33.attn.out_proj,blocks.33.mlp.fc1,blocks.33.mlp.fc2,blocks.34.attn.qkv_proj,blocks.34.attn.out_proj,blocks.34.mlp.fc1,blocks.34.mlp.fc2,blocks.35.attn.qkv_proj,blocks.35.attn.out_proj,blocks.35.mlp.fc1,blocks.35.mlp.fc2,blocks.36.attn.qkv_proj,blocks.36.attn.out_proj,blocks.36.mlp.fc1,blocks.36.mlp.fc2,blocks.37.attn.qkv_proj,blocks.37.attn.out_proj,blocks.37.mlp.fc1,blocks.37.mlp.fc2,blocks.38.attn.qkv_proj,blocks.38.attn.out_proj,blocks.38.mlp.fc1,blocks.38.mlp.fc2,blocks.39.attn.qkv_proj,blocks.39.attn.out_proj,blocks.39.mlp.fc1,blocks.39.mlp.fc2,blocks.40.attn.qkv_proj,blocks.40.attn.out_proj,blocks.40.mlp.fc1,blocks.40.mlp.fc2,blocks.41.attn.qkv_proj,blocks.41.attn.out_proj,blocks.41.mlp.fc1,blocks.41.mlp.fc2,blocks.42.attn.qkv_proj,blocks.42.attn.out_proj,blocks.42.mlp.fc1,blocks.42.mlp.fc2,blocks.43.attn.qkv_proj,blocks.43.attn.out_proj,blocks.43.mlp.fc1,blocks.43.mlp.fc2,blocks.44.attn.qkv_proj,blocks.44.attn.out_proj,blocks.44.mlp.fc1,blocks.44.mlp.fc2,blocks.45.attn.qkv_proj,blocks.45.attn.out_proj,blocks.45.mlp.fc1,blocks.45.mlp.fc2,blocks.46.attn.qkv_proj,blocks.46.attn.out_proj,blocks.46.mlp.fc1,blocks.46.mlp.fc2,blocks.47.attn.qkv_proj,blocks.47.attn.out_proj,blocks.47.mlp.fc1,blocks.47.mlp.fc2,blocks.48.attn.qkv_proj,blocks.48.attn.out_proj,blocks.48.mlp.fc1,blocks.48.mlp.fc2,blocks.49.attn.qkv_proj,blocks.49.attn.out_proj,blocks.49.mlp.fc1,blocks.49.mlp.fc2" \ --lora_rank 16 \ --use_gradient_checkpointing \ --find_unused_parameters \ --task "sft:train" \ --save_steps 100 \ --audio_loss_weight 0 \ --preset_lora_path "./models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter/model.safetensors" \ --preset_lora_model "dit" ``` ### Note 1. **如何加 Training Adapter**：在 Stage 2 的训练命令里加上 `--preset_lora_path .safetensors --preset_lora_model dit`，训练框架会在训练开始前把 Training Adapter 融进 DiT。训练结束后只保留新训出的 LoRA，Training Adapter 不再需要，推理时也不要加载它。 2. **音频 loss**：如果没有想要学习的音频目标，可以将 `--audio_loss_weight` 设置为 `0`，只优化视频分支，可以加快训练速度。 3. **LoRA 注入位置**：`--lora_target_modules` 只覆盖 `blocks.25` ~ `blocks.49`（DiT 共 50 层）的后半段。这么设计是为了加速训练——未被 LoRA 覆盖的前半段模块不参与反向传播，效果也不错。 ## 效果示例 生成结果，统一参数：768×1344、124 帧 @ 24 fps、`num_inference_steps=30`、`seed=0`。
## 相关链接 - 基座模型：[MiniMax/MiniMax-H3](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) - Training Adapter：[DiffSynth-Studio/MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter) - 训练框架：[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio)