[Community pipeline] SDXL Differential Diffusion Img2Img Pipeline

[asomoza]

What does this PR do?

Adds Differential Diffusion as a community pipeline as discussed here, this is the SDXL img2img version.

More information and original PR here. I'll add the documentation to the README after the initials reviews.

Once this one is merged I plan to do also:

• SDXL Differential Diffusion Img2Img Pipeline
• SDXL Differential Diffusion Img2Img Controlnet Adapter Pipeline
• SDXL Differential Diffusion Inpaint Pipeline
• SDXL Differential Diffusion Inpaint Controlnet Adapter Pipeline

Note: It works with turbo and lighting variations, but IMO the only half good one is the lighting 8 steps, with less steps the model can't properly do the inpaint and even with 8, the inpainting isn't that good.

Fixes #7038

Paper: https://differential-diffusion.github.io/paper.pdf
Project site: https://differential-diffusion.github.io/
Code: https://github.com/exx8/differential-diffusion

Example usage:

import torch
from torchvision import transforms

from diffusers import DPMSolverMultistepScheduler
from diffusers.utils import load_image
from examples.community.pipeline_stable_diffusion_xl_differential_img2img import (
    StableDiffusionXLDifferentialImg2ImgPipeline,
)


pipeline = StableDiffusionXLDifferentialImg2ImgPipeline.from_pretrained(
    "SG161222/RealVisXL_V4.0", torch_dtype=torch.float16, variant="fp16"
).to("cuda")
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, use_karras_sigmas=True)


def preprocess_image(image):
    image = image.convert("RGB")
    image = transforms.CenterCrop((image.size[1] // 64 * 64, image.size[0] // 64 * 64))(image)
    image = transforms.ToTensor()(image)
    image = image * 2 - 1
    image = image.unsqueeze(0).to("cuda")
    return image


def preprocess_map(map):
    map = map.convert("L")
    map = transforms.CenterCrop((map.size[1] // 64 * 64, map.size[0] // 64 * 64))(map)
    map = transforms.ToTensor()(map)
    map = map.to("cuda")
    return map


image = preprocess_image(
    load_image(
        "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png?download=true"
    )
)

mask = preprocess_map(
    load_image(
        "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask.png?download=true"
    )
)

prompt = "a green pear"
negative_prompt = "blurry"

image = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    guidance_scale=7.5,
    num_inference_steps=25,
    original_image=image,
    image=image,
    strength=1.0,
    map=mask,
).images[0]

image.save("result.png")

Examples

Realvis	Juggernaut
Lighting 8 steps	IP Adapters

Who can review?

@yiyixuxu @sayakpaul @exx8 @a-r-r-o-w

[yiyixuxu]

looks good! will merge whenever you're ready!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[xinru-liu]

Hi, I would really like to test the differential img2img pipeline, has this been published yet?

[yiyixuxu]

@xinru-liu
it is merged you can check it out here https://github.com/huggingface/diffusers/tree/main/examples/community#differential-diffusion
we also published a blog about it https://huggingface.co/blog/OzzyGT/outpainting-differential-diffusion

[xinru-liu]

@yiyixuxu
Thank you for getting back, I just tested, it shows
"404 Client Error: Not Found for url: https://raw.githubusercontent.com/huggingface/diffusers/v0.27.2/examples/community/pipeline_stable_diffusion_xl_differential_img2img.py"
I think the name is accurate, not sure why it failed to access the code.

[asomoza]

that's because you're trying to access it in the v0.27.2 release, this was merged after the last release so until the next one you have to install diffusers from the git repository:

pip install git+https://github.com/huggingface/diffusers.git

Also you can just download it and use it directly:

https://github.com/huggingface/diffusers/blob/main/examples/community/pipeline_stable_diffusion_xl_differential_img2img.py

[xinru-liu]

that's because you're trying to access it in the v0.27.2 release, this was merged after the last release so until the next one you have to install diffusers from the git repository:

pip install git+https://github.com/huggingface/diffusers.git

Also you can just download it and use it directly:

https://github.com/huggingface/diffusers/blob/main/examples/community/pipeline_stable_diffusion_xl_differential_img2img.py

Thank you @asomoza ! I've tested the solution, it works really well!