Bug introduced in from_pretrained v4.48.3..v4.49.0

[hlky]

Hi 🤗

Diffusers 🧨 noticed some failing tests starting with v4.49.0 in Kolors, one of our models that uses a custom text encoder.

Reproduction

This is working on v4.48.3.

from transformers import AutoModel

model = AutoModel.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b", trust_remote_code=True)

On v4.49.0:

TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=str, device=str), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)

The issue seems to be that the config in the test model and checkpoints like Kwai-Kolors/Kolors-diffusers contain torch_dtype as a string.

On Diffusers end explicitly setting torch_dtype when using ChatGLMModel and setting a default torch_dtype for from_pretrained paths is working huggingface/diffusers#10816 and it's mainly internal effects as torch_dtype wasn't passed for some tests, should be ok for end users as they would generally pass torch_dtype.

[zucchini-nlp]

Hey @hlky ! Thanks for reporting that. Seems to have been broken after the recent addition of dtype for composite configs. I will submit a PR to fix it