--- title: NitroFusion canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/NitroFusion" md_url: "https://www.modelscope.cn/models/AI-ModelScope/NitroFusion.md" repository: AI-ModelScope/NitroFusion last_updated: 2025-01-07 pipeline_tag: text-to-image-synthesis tasks: - text-to-image-synthesis base_model: - ByteDance/Hyper-SD - stabilityai/stable-diffusion-xl-base-1.0 base_model_relation: finetune parameters: 12.1B tensor_type: - BF16 - F16 library_name: - pytorch - lora - safetensors frameworks: - pytorch downloads: 1266 stars: 0 tags: - text-to-image - stable-diffusion - sdxl - "adversarial diffusion distillation" --- # NitroFusion > NitroFusion - AI-ModelScope 在 ModelScope 开源的模型。NitroFusion NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training, --> NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training Dar-Yen Chen, Hmrishav… AI-ModelScope/NitroFusion 是 ModelScope 魔搭社区上的 12.1B 参数text-to-image-synthesis模型，基于 ByteDance/Hyper-SD、stabilityai/stable-diffusion-xl-base-1.0 构建。 - **Repository**: AI-ModelScope/NitroFusion - **Tasks**: text-to-image-synthesis - **Parameters**: 12.1B - **Base model**: ByteDance/Hyper-SD, stabilityai/stable-diffusion-xl-base-1.0 - **Tags**: text-to-image, stable-diffusion, sdxl, adversarial diffusion distillation - **Downloads**: 1266 - **Stars**: 0 - **Last updated**: 2025-01-07 Source: https://www.modelscope.cn/models/AI-ModelScope/NitroFusion --- # NitroFusion > **NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training** > > Dar-Yen Chen, Hmrishav Bandyopadhyay, Kai Zou, Yi-Zhe Song [[arXiv Paper]](https://arxiv.org/abs/2412.02030) [[Project Page]](https://chendaryen.github.io/NitroFusion.github.io/) ![](./assets/banner.jpg) ## News * 06 Jan 2025: ComfyUI checkpoints `nitrosd-realism_comfyui.safetensors` and `nitrosd-vibrant_comfyui.safetensors`, as well as a [workflow](https://github.com/ChenDarYen/ComfyUI-TimestepShiftModel) are now released. * 04 Dec 2024: [Paper](https://arxiv.org/abs/2412.02030) is released on arXiv, and the [project page](https://chendaryen.github.io/NitroFusion.github.io/) is now public. * 30 Nov 2024: Our single-step text-to-image demo is publicly available on [🤗 Hugging Face Space](https://huggingface.co/spaces/ChenDY/NitroFusion_1step_T2I). * 29 Nov 2024: Released two checkpoints: **NitroSD-Realism** and **NitroSD-Vibrant**. ## Online Demos NitroFusion single-step Text-to-Image demo hosted on [🤗 Hugging Face Space](https://huggingface.co/spaces/ChenDY/NitroFusion_1step_T2I) ## Model Overview - `nitrosd-realism_unet.safetensors`: Produces photorealistic images with fine details. - `nitrosd-vibrant_unet.safetensors`: Offers vibrant, saturated color characteristics. - Both models support 1 to 4 inference steps. ## Usage First, we need to implement the scheduler with timestep shift for multi-step inference: ```python from diffusers import LCMScheduler class TimestepShiftLCMScheduler(LCMScheduler): def __init__(self, *args, shifted_timestep=250, **kwargs): super().__init__(*args, **kwargs) self.register_to_config(shifted_timestep=shifted_timestep) def set_timesteps(self, *args, **kwargs): super().set_timesteps(*args, **kwargs) self.origin_timesteps = self.timesteps.clone() self.shifted_timesteps = (self.timesteps * self.config.shifted_timestep / self.config.num_train_timesteps).long() self.timesteps = self.shifted_timesteps def step(self, model_output, timestep, sample, generator=None, return_dict=True): if self.step_index is None: self._init_step_index(timestep) self.timesteps = self.origin_timesteps output = super().step(model_output, timestep, sample, generator, return_dict) self.timesteps = self.shifted_timesteps return output ``` We can then utilize the diffuser pipeline: ```python import torch from diffusers import DiffusionPipeline, UNet2DConditionModel from huggingface_hub import hf_hub_download from safetensors.torch import load_file # Load model. base_model_id = "stabilityai/stable-diffusion-xl-base-1.0" repo = "ChenDY/NitroFusion" # NitroSD-Realism ckpt = "nitrosd-realism_unet.safetensors" unet = UNet2DConditionModel.from_config(base_model_id, subfolder="unet").to("cuda", torch.float16) unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda")) scheduler = TimestepShiftLCMScheduler.from_pretrained(base_model_id, subfolder="scheduler", shifted_timestep=250) scheduler.config.original_inference_steps = 4 # # NitroSD-Vibrant # ckpt = "nitrosd-vibrant_unet.safetensors" # unet = UNet2DConditionModel.from_config(base_model_id, subfolder="unet").to("cuda", torch.float16) # unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda")) # scheduler = TimestepShiftLCMScheduler.from_pretrained(base_model_id, subfolder="scheduler", shifted_timestep=500) # scheduler.config.original_inference_steps = 4 pipe = DiffusionPipeline.from_pretrained( base_model_id, unet=unet, scheduler=scheduler, torch_dtype=torch.float16, variant="fp16", ).to("cuda") prompt = "a photo of a cat" image = pipe( prompt=prompt, num_inference_steps=1, # NotroSD-Realism and -Vibrant both support 1 - 4 inference steps. guidance_scale=0, ).images[0] ``` ## ComfyUI Usage 1. Download the `nitrosd-realism_comfyui.safetensors` and `nitrosd-vibrant_comfyui.safetensors`, and place them in the `ComfyUI/models/checkpoints`. 2. Clone the [ComfyUI-TimestepShiftModel](https://github.com/ChenDarYen/ComfyUI-TimestepShiftModel) repository into the `ComfyUI/custom_nodes`. 3. Play with the [workflow](https://github.com/ChenDarYen/ComfyUI-TimestepShiftModel/blob/main/ComfyUI_NitroSD_workflow.json)! ## License NitroSD-Realism is released under [cc-by-nc-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en), following its base model *DMD2*. NitroSD-Vibrant is released under [openrail++](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md).