Is T5 model supported with HQQ quantization ? (AttributeError: 'HQQLinear' object has no attribute 'weight')

[szwagros]

System Info

• transformers version: 4.48.2
• Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
• Python version: 3.11.11
• Huggingface_hub version: 0.28.1
• Safetensors version: 0.5.2
• Accelerate version: 1.3.0
• Accelerate config: - compute_environment: LOCAL_MACHINE
  • distributed_type: NO
  • mixed_precision: no
  • use_cpu: False
  • debug: False
  • num_processes: 1
  • machine_rank: 0
  • num_machines: 1
  • gpu_ids: all
  • rdzv_backend: static
  • same_network: True
  • main_training_function: main
  • enable_cpu_affinity: False
  • downcast_bf16: no
  • tpu_use_cluster: False
  • tpu_use_sudo: False
  • tpu_env: []
• PyTorch version (GPU?): 2.5.1+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA GeForce RTX 4080 SUPER

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I've created and save quantized version like this:

quant_config = HqqConfig(nbits=4, group_size=64)

model = T5EncoderModel.from_pretrained(
    '/storage/Models/FLUX.1-dev/',
    torch_dtype=torch.bfloat16,
    subfolder = "text_encoder_2",
    device_map="cuda", 
    quantization_config=quant_config
)

model.save_pretrained(
    "./quantized_pipeline/",
    safe_serialization=True  # Use safetensors format
)

During inference I create flux pipeline:

    text_encoder_2 = T5EncoderModel.from_pretrained(
        "./quantized_pipeline/",
        subfolder="text_encoder_2",
        torch_dtype=torch.bfloat16,
        device_map="cuda"
    )


    self.pipeline: FluxPipeline = FluxPipeline.from_pretrained(
        self.model_config.path,
        torch_dtype=torch.bfloat16,
        local_files_only=True,
        text_encoder_2=text_encoder_2
    )    

But when I actually start inference I always get this error:

File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 339, in forward
forwarded_states = self.DenseReluDense(forwarded_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 316, in forward
isinstance(self.wo.weight, torch.Tensor)
^^^^^^^^^^^^^^
File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1931, in getattr
raise AttributeError(
AttributeError: 'HQQLinear' object has no attribute 'weight'

Is it because T5 is not supported or am I doing something wrong ?

Expected behavior

Flux pipeline should work without errors.

[Rocketknight1]

cc @SunMarc @MekkCyber for quantization

[remixer-dec]

Just mentioning that this conversion was done before (not sure how) https://huggingface.co/HighCWu/FLUX.1-dev-4bit/tree/main/text_encoder_2