Guidance needed for Full finetuning of SDXL

[Anustup900]

I am looking to full finetune the base SDXL model, not Dreambooth / LoRa approach. I want to finetune using Kohya SS training script
So asking for best practices for those who had did this before for :

• what should be a good Learning rate to keep ?
• choice of scheduler
• choice of optimizers
• and steps needed per image
  Looking forward for some guidelines here

[maybleMyers]

How big is your dataset? Is it realistic or anime?

[Anustup900]

Hey @maybleMyers, the dataset building is in progress but it will have near to 5 to 10 k images, It is realistic

[dill-shower]

https://github.com/kohya-ss/sd-scripts/blob/sdxl/README.md#tips-for-sdxl-training

Read this please. As someone who also trains models I recommend you to avoid sdxl for full finetune. Sdxl is very slow to train. Consider using sd3 instead, dit provides much faster training and high quality results

[blacklig]

wow, that is interesting info. Would you recommend SD 3 / SD 3.5 even for control nets? How come SD3 is easier to train than SDXL? There is lot of good fine tunes of SDXL less for SD3, so how is this possible?

[dill-shower]

Because sd3 was released just "yesterday" and has significant changes in model architecture (so no goos programs for model training yet, waiting for sd-scripts to accept sd3 in main branch). In theory sd3 should be much better in terms of training and generation than sdxl

[co5dt]

The guy responsible for Joytag and Joycaption trained SDXL on 1.4 million images, 30 mio samples. He sheds light on some of the details:
https://www.reddit.com/r/StableDiffusion/s/moGtoCkrlC