Add Ultimate SD Upscale pipeline for high-quality tiled image upscaling

[BasimBashir]

Is your feature request related to a problem? Please describe.
Currently, diffusers library lacks advanced tiled upscaling capabilities that are available in other Stable Diffusion implementations. While the library supports basic img2img and upscaling, there's no built-in solution for handling large images through intelligent tiling and seam fixing. This makes it difficult to process high-resolution images while maintaining quality and managing memory efficiently.

Describe the solution you'd like
Implement Ultimate SD Upscale functionality (similar to Automatic1111's WebUI extension) as a pipeline in diffusers. Key features should include:

1. Progressive upscaling with intelligent scale factor determination
2. Multiple tiling modes:
   • Linear processing
   • Chess pattern processing
3. Advanced seam fixing options:
   • Band pass mode
   • Half tile offset
   • Half tile with intersections
4. Configurable parameters:
   • Tile sizes
   • Padding
   • Mask blur
   • Denoise strength for seam fixing

The implementation should integrate smoothly with existing diffusers pipelines and maintain the library's user-friendly API style.

Describe alternatives you've considered

• Using basic tiling without seam fixing (leads to visible artifacts)
• Running multiple separate upscale passes (inefficient and lower quality)
• Implementing as a separate package (loses benefits of diffusers' optimization and integration)
• Using other libraries like PIL or cv2 for tiling (lacks SD-specific optimizations)

Additional context

• Reference implementation: https://github.com/Coyote-A/ultimate-upscale-for-automatic1111
• This feature would be particularly valuable for:
  • Professional image upscaling
  • Batch processing of large images
  • Creating high-resolution outputs while managing VRAM
  • Maintaining image quality in tiled processing
• Could potentially be implemented as either a standalone pipeline or an enhancement to existing img2img pipelines
• Would complement existing super-resolution models in the library

[asomoza]

Hi, I've been working on something like this for some time, haven't had the time to finish it and clean the code, but we are going to have an upscaler solution like the Ultimate SD Upscale.

Personally I won't be adding all the options you're mentioning because we always try to make it simple for users to understand it and build on top of it, this doesn't mean people can't add it later.

Also those upscalers are missing some cool new techniques that makes it better and with diffusers we can add them.

Here's some samples but I even haven't updated for it to use controlnet union which is a lot better. I had to convert the upscaled images to jpg because of the size so there's some image loss.

You can open them in other tabs to compare them, they're full size images.

original	upscaled

I don't think they're that good yet but it would help me a lot if you or someone from the community could run the same originals and upscale them 4x so I can compare and see where I'm at.

Original images credits:

Lotus Esprit V8 by Michael Gil

Man wearing blue denim top and black sunglasses by Andrea Dibitonto

[SakshamDhawan]

I can work on adding the features - maybe pick up from @asomoza ’s progress so far?

[sayakpaul]

Cc: @asomoza ^

[nickkolok]

@asomoza It looks like a really good idea if you shared your drafts, even as a gist! The diffusers-based upscaler sounds wonderul!

[asomoza]

Hi, I'll try to clean the code and share the script, but it will probably next week, this won't an official pipeline for the moment but if someone wants to contribute it later, feel free.

I'll do one later but with the new modular diffusers PR.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[John6666cat]

Progress?

[asomoza]

I paused it because we already have a working PR for modular diffusers, this is going to be my first release for that and I'm going to start working on it now.

[okaris]

@asomoza Thank you for looking into this! If you’re able to share your WIP code for non-modular diffusers, I’d gladly take it from there. It could still be valuable for those not switching to modular yet, and I’d be happy to clean it up and get it running rather than starting from scratch.

[Caselles]

I support this initiative! Would be an awesome addition to Diffusers. Let us know if we can help

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[asomoza]

I missed the updates on this.

There's a misunderstanding here, I don't have anything I can share yet, there's just some code that's hard to read and to use, I have to input manually some of the params and I'm not really happy with it (it also uses a old models), so if I don't clean it and update it, it would cause more work for me than help others.

I put it on hold because this would be a lot easier to do with modular diffusers than with a regular pipeline and I discovered some new techniques to make it better, this is not something new also, there's a lot of code examples everywhere (official Ultimate Upscale, comfy upscalers, InvokeAI Upscaler), so you have a lot of references from where to start.

The hard work and more difficult part is to adapt them to a diffusers pipeline which I haven't done yet and the reason I haven't shared any code.

But I suspect that this will be resolved soon, if not by me, from someone from the community.

[asomoza]

#10951

[elismasilva]

Look, I didn't even know this Issue existed. I'm happy if my pipeline met the requirements and if it didn't, it can be improved in the final details.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[asomoza]

Even though we have a custom pipeline made by @elismasilva that should solve this issue, I'll close this issue when I have the other one made with modular diffusers.

[elismasilva]

Even though we have a custom pipeline made by @elismasilva that should solve this issue, I'll close this issue when I have the other one made with modular diffusers.

Soon I will add another pipeline for restoring low quality images that can be used in conjunction with the previous one. So we will have 2 complete stages 1-restoration, 2-upscale.

[nickkolok]

Could you please clarify if we already have a diffusers-based tiled upscaler? To clarify a bit: by saying a "tiled upscaler", I mean an upscaler that can bypass the memory limitations for both UNet and VAE. I mean, if on a machine usual img2img fails e.g. on images larger than 1024x1024px, then tiled upscaler should be able to process images of 1024x1536px by splitting them into tiles - of course, by the cost of slight quality loss around the tile mergelines. A1111 has such a tool, AFAIK. Do diffusers also have it? What version of diffusers does it require?

[asomoza]

@nickkolok at the moment it's a custom pipeline and yes, it uses tiling and you can go a lot higher than you can usually go (limited by model training or VRAM), the PR that merged the pipeline is this one: #10951 and right now it can only be used if you use diffusers from main.

You can also download it though and execute the script directly from the current version of diffusers.

It also has a huggingface space that you can clone too and use it online or locally.

The only caveats it's that it's not a diffusers core upscaler and that it's a little different that the SD Upscaler that this issue mentioned, but you can test it and see it by yourself.

[nickkolok]

Sounds really great, thank you!!