---
title: AnyDoor_models
canonical_url: "https://www.modelscope.cn/models/iic/AnyDoor_models"
md_url: "https://www.modelscope.cn/models/iic/AnyDoor_models.md"
repository: iic/AnyDoor_models
last_updated: 2025-06-13
library_name:
  - pytorch
frameworks:
  - pytorch
downloads: 662
stars: 21
---

# AnyDoor_models

> AnyDoor_models - iic 在 ModelScope 开源的模型。AnyDoor是一种基于扩散模型的图像生成模型，以非常和谐的方式将目标对象传送到用户制定位置的新场景的能力。过程中不需要为每个对象调整参数，只需要训练一次，并在推理阶段适用于各种不同的对象-场景组合。

- **Repository**: iic/AnyDoor_models
- **Downloads**: 662
- **Stars**: 21
- **Last updated**: 2025-06-13

Source: https://www.modelscope.cn/models/iic/AnyDoor_models

---

<p align="center">

  <h2 align="center">AnyDoor: Zero-shot Object-level Image Customization</h2>
  <p align="center">
    <a href="https://xavierchen34.github.io/"><strong>Xi Chen</strong></a>
    ·
    <a href="https://scholar.google.com/citations?user=JYVCn3AAAAAJ&hl=en"><strong>Lianghua Huang</strong></a>
    ·
    <a href="https://scholar.google.com/citations?user=8zksQb4AAAAJ&hl=zh-CN"><strong>Yu Liu</strong></a>
    ·
    <a href="https://shenyujun.github.io/"><strong>Yujun Shen</strong></a>
    ·
    <a href="https://scholar.google.com/citations?user=7LhjCn0AAAAJ&hl=en"><strong>Deli Zhao</strong></a>
    ·
    <a href="https://hszhao.github.io/"><strong>Hengshuang Zhao</strong></a>
    <br>
    <br>
        <a href="https://arxiv.org/abs/2307.09481"><img src='https://img.shields.io/badge/arXiv-AnyDoor-red' alt='Paper PDF'></a>
        <a href='https://damo-vilab.github.io/AnyDoor-Page/'><img src='https://img.shields.io/badge/Project_Page-AnyDoor-green' alt='Project Page'></a>
    <br>
    <b>The University of Hong Kong &nbsp; | &nbsp;  Alibaba Group  | &nbsp;  Ant Group </b>
  </p>
  
  <table align="center">
    <tr>
    <td>
      <img src="assets/Figures/Teaser.png">
    </td>
    </tr>
  </table>

## News
* **[2023.12.17]** Release train & inference & demo code, and pretrained checkpoint.
* **[Soon]** Release the new version paper.
* **[Soon]** Support online demo.
* **[On-going]** Scale-up the training data and release stronger models as the foundaition model for downstream region-to-region generation tasks.
* **[On-going]** Release specific-designed models for downstream tasks like virtual tryon, face swap, text and logo transfer, etc.


## Installation
Install with `conda`: 
```bash
conda env create -f environment.yaml
conda activate anydoor
```
or `pip`:
```bash
pip install -r requirements.txt
```
Additionally, for training, you need to install panopticapi, pycocotools, and lvis-api.
```bash
pip install git+https://github.com/cocodataset/panopticapi.git

pip install pycocotools -i https://pypi.douban.com/simple

pip install lvis
```

or use in modelscope notebook
```python
!git clone https://github.com/modelscope/modelscope.git
%cd modelscope
!pip install .
```
## Download Checkpoints
Download AnyDoor checkpoint: 
* URL: https://huggingface.co/spaces/xichenhku/AnyDoor/tree/main
* We include all the optimizer params for Adam, so the checkpoint is big. You could only keep the "state_dict" to make it much smaller.


Download DINOv2 checkpoint and revise `/configs/anydoor.yaml` for the path (line 83)
* URL: https://github.com/facebookresearch/dinov2?tab=readme-ov-file

Download Stable Diffusion V2.1 if you want to train from scratch.
* URL: https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main


## Inference
We provide inference code in `run_inference.py` (from Line 222 - ) for both inference single image and inference a dataset (VITON-HD Test). You should modify the data path and run the following code. The generated results are provided in `examples/TestDreamBooth/GEN` for single image, and `VITONGEN` for VITON-HD Test.

```python
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

ref_image = 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_anydoor_fg.jpg'
ref_mask = 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_anydoor_fg_mask.png'
bg_image = 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_anydoor_bg.png'
bg_mask = 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_anydoor_bg_mask.png'

anydoor_pipline = pipeline(
    Tasks.image_to_image_generation, model='damo/AnyDoor_models', trust_remote_code=True, model_revision='master')
out = anydoor_pipline((ref_image, ref_mask, bg_image, bg_mask))
image = out['output_img']
image.save('./image_anydoor_gen.png')
```
The inferenced results on VITON-Test would be like [garment, ground truth, generation].

*Noticing that AnyDoor does not contain any specific design/tuning for tryon, we think it would be helpful to add skeleton infos or warped garment, and tune on tryon data to make it better :)*
  <table align="center">
    <tr>
    <td>
      <img src="assets/Figures/tryon.png">
    </td>
    </tr>
  </table>


Our evaluation data for DreamBooth an COCOEE coud be downloaded at Google Drive:
* URL: [to be released]





## Gradio demo 
Currently, we suport local gradio demo. To launch it, you should firstly modify `/configs/demo.yaml` for the path to the pretrained model, and `/configs/anydoor.yaml` for the path to DINOv2(line 83). 

Afterwards, run the script:
```bash
python run_gradio_demo.py
```
The gradio demo would look like the UI shown below:

<table align="center">
  <tr>
  <td>
    <img src="assets/Figures/gradio.png">
  </td>
  </tr>
</table>

## Train

### Prepare datasets
* Download the datasets that present in `/configs/datasets.yaml` and modify the corresponding paths.
* You could prepare you own datasets according to the formates of files in `./datasets`.
* If you use UVO dataset, you need to process the json following `./datasets/Preprocess/uvo_process.py`
* You could refer to `run_dataset_debug.py` to verify you data is correct.


### Prepare initial weight
* If your would like to train from scratch, convert the downloaded SD weights to control copy by running:
```bash
sh ./scripts/convert_weight.sh  
```
### Start training
* Modify the training hyper-parameters in `run_train_anydoor.py` Line 26-34 according to your training resources. We verify that using 2-A100 GPUs with batch accumulation=1 could get satisfactory results after 300,000 iterations.


* Start training by executing: 
```bash
sh ./scripts/train.sh  
```

## Acknowledgements
This project is developped on the codebase of [ControlNet](https://github.com/lllyasviel/ControlNet). We  appreciate this great work! 


## Citation
If you find this codebase useful for your research, please use the following entry.
```BibTeX
@article{chen2023anydoor,
  title={Anydoor: Zero-shot object-level image customization},
  author={Chen, Xi and Huang, Lianghua and Liu, Yu and Shen, Yujun and Zhao, Deli and Zhao, Hengshuang},
  journal={arXiv preprint arXiv:2307.09481},
  year={2023}
}
```