--- title: FLUX.1-dev-ControlNet-Union-Pro-2.0 canonical_url: "https://www.modelscope.cn/models/LiblibAI/FLUX.1-dev-ControlNet-Union-Pro-2.0" md_url: "https://www.modelscope.cn/models/LiblibAI/FLUX.1-dev-ControlNet-Union-Pro-2.0.md" repository: LiblibAI/FLUX.1-dev-ControlNet-Union-Pro-2.0 last_updated: 2025-04-24 license: other pipeline_tag: text-to-image-synthesis tasks: - text-to-image-synthesis base_model: - black-forest-labs/FLUX.1-dev base_model_relation: finetune parameters: 2.1B tensor_type: - BF16 library_name: - pytorch - lora - diffusers - safetensors frameworks: - pytorch language: - en downloads: 8890 stars: 23 tags: - Text-to-Image - ControlNet - Diffusers - Flux.1-dev - image-generation - "Stable Diffusion" --- # FLUX.1-dev-ControlNet-Union-Pro-2.0 > FLUX.1-dev-ControlNet-Union-Pro-2.0 - LiblibAI 在 ModelScope 开源的模型。FLUX.1-dev-ControlNet-Union-Pro-2.0 LiblibAI/FLUX.1-dev-ControlNet-Union-Pro-2.0 是 ModelScope 魔搭社区上的 2.1B 参数text-to-image-synthesis模型，采用 other 许可，基于 black-forest-labs/FLUX.1-dev 构建。 - **Repository**: LiblibAI/FLUX.1-dev-ControlNet-Union-Pro-2.0 - **License**: other - **Tasks**: text-to-image-synthesis - **Parameters**: 2.1B - **Base model**: black-forest-labs/FLUX.1-dev - **Tags**: Text-to-Image, ControlNet, Diffusers, Flux.1-dev, image-generation, Stable Diffusion - **Downloads**: 8890 - **Stars**: 23 - **Last updated**: 2025-04-24 Source: https://www.modelscope.cn/models/LiblibAI/FLUX.1-dev-ControlNet-Union-Pro-2.0 --- # FLUX.1-dev-ControlNet-Union-Pro-2.0 This repository contains an unified ControlNet for FLUX.1-dev model released by [Shakker Labs](https://huggingface.co/Shakker-Labs). We provide an [online demo](https://huggingface.co/spaces/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0). A FP8 quantized version provided by community can be found in [ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8](https://huggingface.co/ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8). # Keynotes In comparison with [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro), - Remove mode embedding, has smaller model size. - Improve on canny and pose, better control and aesthetics. - Add support for soft edge. Remove support for tile. # Model Cards - This ControlNet consists of 6 double blocks and 0 single block. Mode embedding is removed. - We train the model from scratch for 300k steps using a dataset of 20M high-quality general and human images. We train at 512x512 resolution in BFloat16, batch size = 128, learning rate = 2e-5, the guidance is uniformly sampled from [1, 7]. We set the text drop ratio to 0.20. - This model supports multiple control modes, including canny, soft edge, depth, pose, gray. You can use it just as a normal ControlNet. - This model can be jointly used with other ControlNets. # Showcases
canny
softedge
pose
depth
gray
# Inference ```python import torch from diffusers.utils import load_image from diffusers import FluxControlNetPipeline, FluxControlNetModel base_model = 'black-forest-labs/FLUX.1-dev' controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0' controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16) pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16) pipe.to("cuda") # replace with other conds control_image = load_image("./conds/canny.png") width, height = control_image.size prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones." image = pipe( prompt, control_image=control_image, width=width, height=height, controlnet_conditioning_scale=0.7, control_guidance_end=0.8, num_inference_steps=30, guidance_scale=3.5, generator=torch.Generator(device="cuda").manual_seed(42), ).images[0] ``` # Multi-Inference ```python import torch from diffusers.utils import load_image # https://github.com/huggingface/diffusers/pull/11350 # You can directly import from diffusers by install the laster version from source # from diffusers import FluxControlNetPipeline, FluxControlNetModel # use local files for this moment from pipeline_flux_controlnet import FluxControlNetPipeline from controlnet_flux import FluxControlNetModel base_model = 'black-forest-labs/FLUX.1-dev' controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0' controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16) pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=[controlnet], torch_dtype=torch.bfloat16) # use [] to enable multi-CNs pipe.to("cuda") # replace with other conds control_image = load_image("./conds/canny.png") width, height = control_image.size prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones." image = pipe( prompt, control_image=[control_image, control_image], # try with different conds such as canny&depth, pose&depth width=width, height=height, controlnet_conditioning_scale=[0.35, 0.35], control_guidance_end=[0.8, 0.8], num_inference_steps=30, guidance_scale=3.5, generator=torch.Generator(device="cuda").manual_seed(42), ).images[0] ``` # Recommended Parameters You can adjust controlnet_conditioning_scale and control_guidance_end for stronger control and better detail preservation. For better stability, we highly suggest to use detailed prompt, for some cases, multi-conditions help. - Canny: use cv2.Canny, controlnet_conditioning_scale=0.7, control_guidance_end=0.8. - Soft Edge: use [AnylineDetector](https://github.com/huggingface/controlnet_aux), controlnet_conditioning_scale=0.7, control_guidance_end=0.8. - Depth: use [depth-anything](https://github.com/DepthAnything/Depth-Anything-V2), controlnet_conditioning_scale=0.8, control_guidance_end=0.8. - Pose: use [DWPose](https://github.com/IDEA-Research/DWPose/tree/onnx), controlnet_conditioning_scale=0.9, control_guidance_end=0.65. - Gray: use cv2.cvtColor, controlnet_conditioning_scale=0.9, control_guidance_end=0.8. # Resources - [InstantX/FLUX.1-dev-IP-Adapter](https://huggingface.co/InstantX/FLUX.1-dev-IP-Adapter) - [InstantX/FLUX.1-dev-Controlnet-Canny](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny) - [Shakker-Labs/FLUX.1-dev-ControlNet-Depth](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Depth) - [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro) # Acknowledgements This model is developed by [Shakker Labs](https://huggingface.co/Shakker-Labs). The original idea is inspired by [xinsir/controlnet-union-sdxl-1.0](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0). All copyright reserved. # Citation If you find this project useful in your research, please cite us via ``` @misc{flux-cn-union-pro-2, author = {Shakker-Labs}, title = {ControlNet-Union}, year = {2025}, howpublished={\url{https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0}}, } ```