--- title: MiniMax-H3-Ref2VA-FirstFrame-Lineart canonical_url: "https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart" md_url: "https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart.md" repository: mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart chinese_name: MiniMax-H3-Ref2VA-FirstFrame-Lineart last_updated: 2026-09-08 license: "Apache License 2.0" base_model: - MiniMax/MiniMax-H3 base_model_relation: adapter parameters: 37.3M tensor_type: - BF16 library_name: - pytorch - lora - safetensors downloads: 41 stars: 6 tags: - LoRA - MiniMax-H3 - video - audio - lineart - ref2va --- # MiniMax-H3-Ref2VA-FirstFrame-Lineart > MiniMax-H3-Ref2VA-FirstFrame-Lineart - mibei0804 在 ModelScope 开源的模型。MiniMax-H3 Ref2VA First-Frame Lineart LoRA mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart 是 ModelScope 魔搭社区上的 37.3M 参数机器学习模型，采用 Apache License 2.0 许可，基于 MiniMax/MiniMax-H3 构建。 - **Repository**: mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart - **License**: Apache License 2.0 - **Parameters**: 37.3M - **Base model**: MiniMax/MiniMax-H3 - **Tags**: LoRA, MiniMax-H3, video, audio, lineart, ref2va - **Downloads**: 41 - **Stars**: 6 - **Last updated**: 2026-09-08 Source: https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart --- # MiniMax-H3 Ref2VA First-Frame Lineart LoRA 给 [MiniMax-H3](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) 的 Ref2VA 任务（参考 → 视频 + 音频）加上线稿首帧控制能力的 LoRA，共训练 1000 step。给一张首帧线稿和一段 prompt，即可生成以该线稿为开场构图、并把线稿"上色完成"为成片动画的视频 + 音频。 本模型同时是 [MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter)（DeCFG Training Adapter）方案在 Ref2VA 任务上的验证案例：训练时把一个 Ref2VA 版的 Training Adapter 作为 preset lora 融进基座，训练结束后只保留新训出的 LoRA。 LoRA 权重文件：`model.safetensors` ## Prompt 与参考图写法 参考图是一张首帧线稿（RGB），通过 `references` 传入，顺序对应 prompt 里的 ``、``……： ```python references=[{"type": "image", "image": Image.open("lineart.png").convert("RGB")}] ``` prompt 采用 Ref2VA 的多字段结构，字段名保持字面量原样书写。所有字段的内容都用英文书写，不要写中文： - `subject_definitions`、`retention_analysis`：逐字固定的英文模板，直接照抄下方完整示例中的对应段落，不要改写 - `summary`：固定英文前缀加一句英文场景描述，前缀照抄下方完整示例 - `detailed_description`：唯一完全自由编写的字段，英文。画面与分镜描述，多镜头用 `[Shot 1]` / `[Shot 2]` 标记，第一个镜头要写从 `` 开始 - `overall_soundscape` / `non_diegetic_music`：环境音与背景音乐，不需要时写 `N/A` 完整示例： ```text subject_definitions: is the complete visible scene content in , including all depicted characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing. is the line-art first frame of [Shot 1] at 0.00 seconds and provides the opening composition for the target video. summary: [keyframe completion] The target video begins from , preserves the line-defined content and composition of , and develops it as finished 2D animation. A sun-drenched rural road scene featuring lush greenery and warm, nostalgic lighting. retention_analysis: (appears in [Shot 1]): fully_preserved - the characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing established by the line-art first frame are retained. ([Shot 1] first frame): partially_preserved - its composition and structural content are retained at 0.00 seconds while the monochrome line art is rendered with the target colors, lighting, textures, and detail. detailed_description: A sun-drenched rural road scene featuring lush greenery and warm, nostalgic lighting. [Shot 1] The shot begins from , preserving 's outlines, subject placement, poses, spatial relationships, and camera framing while rendering the line art as finished 2D animation. A girl with a dark ponytail, wearing a white shirt and dark skirt, stands on a cracked asphalt road next to a mossy wall. A delivery person on a yellow scooter with a blue crate approaches from behind. [Shot 2] At 00:04.000, A close-up shows the delivery person in a red uniform and helmet looking at the girl. [Shot 3] At 00:05.000, A high-angle view shows the delivery person riding the scooter toward the camera, passing the girl who stands by the guardrail overlooking a river. overall_soundscape: N/A non_diegetic_music: N/A ``` ## 快速开始 ### 1. 安装 DiffSynth-Studio ```bash pip install diffsynth ``` 或从源码安装： ```bash git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` ### 2. 推理 ```python import torch from PIL import Image from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig from diffsynth.utils.data.audio_video import write_video_audio pipe = MiniMaxH3Pipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/text_encoder/model*.safetensors"), ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/transformer/model*.safetensors"), ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/video_vae/source/model.safetensors"), ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/audio_vae/model.safetensors"), ], processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/processor/"), vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 5, ) pipe.load_lora(pipe.dit, ModelConfig( model_id="mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart", origin_file_pattern="model.safetensors", )) prompt = """subject_definitions: is the complete visible scene content in , including all depicted characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing. is the line-art first frame of [Shot 1] at 0.00 seconds and provides the opening composition for the target video. summary: [keyframe completion] The target video begins from , preserves the line-defined content and composition of , and develops it as finished 2D animation. retention_analysis: (appears in [Shot 1]): fully_preserved - the characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing established by the line-art first frame are retained. detailed_description: [Shot 1] The shot begins from , preserving 's outlines, subject placement, poses, spatial relationships, and camera framing while rendering the line art as finished 2D animation. overall_soundscape: N/A non_diegetic_music: N/A""" video, audio = pipe( prompt=prompt, height=768, width=1344, num_frames=124, num_inference_steps=30, seed=0, cfg_scale=1.0, references=[{"type": "image", "image": Image.open("lineart.png").convert("RGB")}], ) write_video_audio( video=video, audio=audio, output_path="lineart_ref2va.mp4", fps=24, audio_sample_rate=pipe.audio_vae.sample_rate, ) # To unload the LoRA: # pipe.clear_lora() ``` 基座权重与 processor 均位于 `Ref2VA/` 子目录。 ## 训练方法 ### 使用 Ref2VA 版 Training Adapter 作为 preset lora MiniMax-H3 基座是经过 CFG 蒸馏的，直接在上面做 LoRA 微调容易不稳定，或者破坏蒸馏后的 CFG-free 行为。Training Adapter 在训练期间把蒸馏后的 DiT 临时拉回接近蒸馏前的状态，为新 LoRA 提供更好的优化面。 Ref2VA 任务使用 [MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter) 仓库下的 Ref2VA 版文件 `model_ref2va.safetensors`。 **Training Adapter 只在训练时用，推理时不要加载**——本仓库发布的 `model.safetensors` 是已经脱离 Training Adapter 的独立 LoRA，直接加载到原版 MiniMax-H3 Ref2VA 即可。 思路来源：DiffSynth-Studio 的 [Differential LoRA Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Differential_LoRA.html#using-differential-lora-training-in-the-training-framework)，以及 [ai-toolkit](https://github.com/ostris/ai-toolkit) 的 Training Adapter 概念。 ### 完整训练脚本 8 卡训练，共 1000 step。 先下载 Training Adapter： ```bash modelscope download --model DiffSynth-Studio/MiniMax-H3-TrainingAdapter --local_dir ./models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter ``` **Stage 1：数据预处理** ```bash accelerate launch --num_processes 8 examples/minimax_h3/model_training/train.py \ --dataset_base_path "" \ --dataset_metadata_path "/metadata_train.jsonl" \ --data_file_keys "video,input_audio,references" \ --extra_inputs "input_audio,references" \ --max_pixels 399360 \ --num_frames 124 \ --dataset_repeat 1 \ --model_id_with_origin_paths "MiniMax/MiniMax-H3:Ref2VA/text_encoder/model*.safetensors,MiniMax/MiniMax-H3:Ref2VA/video_vae/source/model.safetensors,MiniMax/MiniMax-H3:Ref2VA/audio_vae/model.safetensors" \ --processor_path "MiniMax/MiniMax-H3:Ref2VA/processor/" \ --learning_rate 1e-4 \ --num_epochs 1 \ --remove_prefix_in_ckpt "pipe.dit." \ --output_path "./models/train/lineart_ref2va_cache" \ --lora_base_model "dit" \ --lora_target_modules "blocks.25.attn.qkv_proj,blocks.25.attn.out_proj,blocks.25.mlp.fc1,blocks.25.mlp.fc2,blocks.26.attn.qkv_proj,blocks.26.attn.out_proj,blocks.26.mlp.fc1,blocks.26.mlp.fc2,blocks.27.attn.qkv_proj,blocks.27.attn.out_proj,blocks.27.mlp.fc1,blocks.27.mlp.fc2,blocks.28.attn.qkv_proj,blocks.28.attn.out_proj,blocks.28.mlp.fc1,blocks.28.mlp.fc2,blocks.29.attn.qkv_proj,blocks.29.attn.out_proj,blocks.29.mlp.fc1,blocks.29.mlp.fc2,blocks.30.attn.qkv_proj,blocks.30.attn.out_proj,blocks.30.mlp.fc1,blocks.30.mlp.fc2,blocks.31.attn.qkv_proj,blocks.31.attn.out_proj,blocks.31.mlp.fc1,blocks.31.mlp.fc2,blocks.32.attn.qkv_proj,blocks.32.attn.out_proj,blocks.32.mlp.fc1,blocks.32.mlp.fc2,blocks.33.attn.qkv_proj,blocks.33.attn.out_proj,blocks.33.mlp.fc1,blocks.33.mlp.fc2,blocks.34.attn.qkv_proj,blocks.34.attn.out_proj,blocks.34.mlp.fc1,blocks.34.mlp.fc2,blocks.35.attn.qkv_proj,blocks.35.attn.out_proj,blocks.35.mlp.fc1,blocks.35.mlp.fc2,blocks.36.attn.qkv_proj,blocks.36.attn.out_proj,blocks.36.mlp.fc1,blocks.36.mlp.fc2,blocks.37.attn.qkv_proj,blocks.37.attn.out_proj,blocks.37.mlp.fc1,blocks.37.mlp.fc2,blocks.38.attn.qkv_proj,blocks.38.attn.out_proj,blocks.38.mlp.fc1,blocks.38.mlp.fc2,blocks.39.attn.qkv_proj,blocks.39.attn.out_proj,blocks.39.mlp.fc1,blocks.39.mlp.fc2,blocks.40.attn.qkv_proj,blocks.40.attn.out_proj,blocks.40.mlp.fc1,blocks.40.mlp.fc2,blocks.41.attn.qkv_proj,blocks.41.attn.out_proj,blocks.41.mlp.fc1,blocks.41.mlp.fc2,blocks.42.attn.qkv_proj,blocks.42.attn.out_proj,blocks.42.mlp.fc1,blocks.42.mlp.fc2,blocks.43.attn.qkv_proj,blocks.43.attn.out_proj,blocks.43.mlp.fc1,blocks.43.mlp.fc2,blocks.44.attn.qkv_proj,blocks.44.attn.out_proj,blocks.44.mlp.fc1,blocks.44.mlp.fc2,blocks.45.attn.qkv_proj,blocks.45.attn.out_proj,blocks.45.mlp.fc1,blocks.45.mlp.fc2,blocks.46.attn.qkv_proj,blocks.46.attn.out_proj,blocks.46.mlp.fc1,blocks.46.mlp.fc2,blocks.47.attn.qkv_proj,blocks.47.attn.out_proj,blocks.47.mlp.fc1,blocks.47.mlp.fc2,blocks.48.attn.qkv_proj,blocks.48.attn.out_proj,blocks.48.mlp.fc1,blocks.48.mlp.fc2,blocks.49.attn.qkv_proj,blocks.49.attn.out_proj,blocks.49.mlp.fc1,blocks.49.mlp.fc2" \ --lora_rank 16 \ --use_gradient_checkpointing_offload \ --task "sft:data_process" ``` **Stage 2：训练 LoRA** ```bash accelerate launch --num_processes 8 examples/minimax_h3/model_training/train.py \ --dataset_base_path ./models/train/lineart_ref2va_cache \ --data_file_keys "video,input_audio,references" \ --extra_inputs "input_audio,references" \ --max_pixels 399360 \ --num_frames 124 \ --dataset_repeat 8 \ --model_id_with_origin_paths "MiniMax/MiniMax-H3:Ref2VA/transformer/model*.safetensors" \ --processor_path "MiniMax/MiniMax-H3:Ref2VA/processor/" \ --learning_rate 1e-4 \ --num_epochs 1 \ --remove_prefix_in_ckpt "pipe.dit." \ --output_path "./models/train/lineart_ref2va" \ --lora_base_model "dit" \ --lora_target_modules "blocks.25.attn.qkv_proj,blocks.25.attn.out_proj,blocks.25.mlp.fc1,blocks.25.mlp.fc2,blocks.26.attn.qkv_proj,blocks.26.attn.out_proj,blocks.26.mlp.fc1,blocks.26.mlp.fc2,blocks.27.attn.qkv_proj,blocks.27.attn.out_proj,blocks.27.mlp.fc1,blocks.27.mlp.fc2,blocks.28.attn.qkv_proj,blocks.28.attn.out_proj,blocks.28.mlp.fc1,blocks.28.mlp.fc2,blocks.29.attn.qkv_proj,blocks.29.attn.out_proj,blocks.29.mlp.fc1,blocks.29.mlp.fc2,blocks.30.attn.qkv_proj,blocks.30.attn.out_proj,blocks.30.mlp.fc1,blocks.30.mlp.fc2,blocks.31.attn.qkv_proj,blocks.31.attn.out_proj,blocks.31.mlp.fc1,blocks.31.mlp.fc2,blocks.32.attn.qkv_proj,blocks.32.attn.out_proj,blocks.32.mlp.fc1,blocks.32.mlp.fc2,blocks.33.attn.qkv_proj,blocks.33.attn.out_proj,blocks.33.mlp.fc1,blocks.33.mlp.fc2,blocks.34.attn.qkv_proj,blocks.34.attn.out_proj,blocks.34.mlp.fc1,blocks.34.mlp.fc2,blocks.35.attn.qkv_proj,blocks.35.attn.out_proj,blocks.35.mlp.fc1,blocks.35.mlp.fc2,blocks.36.attn.qkv_proj,blocks.36.attn.out_proj,blocks.36.mlp.fc1,blocks.36.mlp.fc2,blocks.37.attn.qkv_proj,blocks.37.attn.out_proj,blocks.37.mlp.fc1,blocks.37.mlp.fc2,blocks.38.attn.qkv_proj,blocks.38.attn.out_proj,blocks.38.mlp.fc1,blocks.38.mlp.fc2,blocks.39.attn.qkv_proj,blocks.39.attn.out_proj,blocks.39.mlp.fc1,blocks.39.mlp.fc2,blocks.40.attn.qkv_proj,blocks.40.attn.out_proj,blocks.40.mlp.fc1,blocks.40.mlp.fc2,blocks.41.attn.qkv_proj,blocks.41.attn.out_proj,blocks.41.mlp.fc1,blocks.41.mlp.fc2,blocks.42.attn.qkv_proj,blocks.42.attn.out_proj,blocks.42.mlp.fc1,blocks.42.mlp.fc2,blocks.43.attn.qkv_proj,blocks.43.attn.out_proj,blocks.43.mlp.fc1,blocks.43.mlp.fc2,blocks.44.attn.qkv_proj,blocks.44.attn.out_proj,blocks.44.mlp.fc1,blocks.44.mlp.fc2,blocks.45.attn.qkv_proj,blocks.45.attn.out_proj,blocks.45.mlp.fc1,blocks.45.mlp.fc2,blocks.46.attn.qkv_proj,blocks.46.attn.out_proj,blocks.46.mlp.fc1,blocks.46.mlp.fc2,blocks.47.attn.qkv_proj,blocks.47.attn.out_proj,blocks.47.mlp.fc1,blocks.47.mlp.fc2,blocks.48.attn.qkv_proj,blocks.48.attn.out_proj,blocks.48.mlp.fc1,blocks.48.mlp.fc2,blocks.49.attn.qkv_proj,blocks.49.attn.out_proj,blocks.49.mlp.fc1,blocks.49.mlp.fc2" \ --lora_rank 16 \ --use_gradient_checkpointing_offload \ --find_unused_parameters \ --task "sft:train" \ --save_steps 100 \ --audio_loss_weight 0 \ --preset_lora_path "./models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter/model_ref2va.safetensors" \ --preset_lora_model "dit" ``` Stage 2 从缓存目录读数据，所以不再需要 `--dataset_metadata_path`。 ### Note 1. **如何加 Training Adapter**：在 Stage 2 的训练命令里加上 `--preset_lora_path .safetensors --preset_lora_model dit`，训练框架会在训练开始前把 Training Adapter 融进 DiT。训练结束后只保留新训出的 LoRA，Training Adapter 不再需要，推理时也不要加载它。 2. **音频 loss**：如果没有想要学习的音频目标，可以将 `--audio_loss_weight` 设置为 `0`，只优化视频分支，可以加快训练速度。 3. **LoRA 注入位置**：`--lora_target_modules` 只覆盖 `blocks.25` ~ `blocks.49`（DiT 共 50 层）的后半段。这么设计是为了加速训练——未被 LoRA 覆盖的前半段模块不参与反向传播，效果也不错。 4. **训练显存**：如果显存不够，可以尝试开启 DeepSpeed 等方式降低训练显存，参考 [Low-VRAM Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Model_Training.html#low-vram-training)。 ## 效果示例 生成结果，统一参数：768×1344、124 帧 @ 24 fps、`num_inference_steps=30`、`seed=0`、`cfg_scale=1.0`。每行左为输入线稿，右为生成结果。
## 相关链接 - 基座模型：[MiniMax/MiniMax-H3](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) - Training Adapter：[DiffSynth-Studio/MiniMax-H3-TrainingAdapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter) - 训练框架：[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio)