The output tensor's data type is not torch.long when the input text is empty.

[wangzhen0518]

System Info

• transformers version: 4.48.1
• Platform: Linux-5.15.0-130-generic-x86_64-with-glibc2.35
• Python version: 3.12.8
• Huggingface_hub version: 0.27.1
• Safetensors version: 0.5.2
• Accelerate version: 1.3.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.5.1+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): 0.10.2 (cpu)
• Jax version: 0.5.0
• JaxLib version: 0.5.0
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: No
• GPU type: NVIDIA GeForce RTX 3060 Ti

Who can help?

@ArthurZucker and @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The output tensor's data type is not torch.long when the input text is empty.

t = tokenizer('', return_tensors='pt')
print(t['input_ids'].dtype)
# torch.float32

Expected behavior

t = tokenizer('', return_tensors='pt')
print(t['input_ids'].dtype)
# torch.int64

[Rocketknight1]

Hi @wangzhen0518, does this happen with all tokenizer classes, or just a specific one you tested?

[wangzhen0518]

I have only tested it on the tokenizer of the QWen series models. Here is the complete code.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-1.5B-Instruct')
t = tokenizer('', return_tensors='pt')
print(t['input_ids'].dtype) #torch.float32

[Rocketknight1]

I've investigated further and I believe this is caused by the behaviour of torch.tensor():

torch.tensor([]).dtype # float32

torch.tensor([1, 2, 3]).dtype # int64

The torch behaviour when converting an input list to a tensor is that the output dtype will be int64 when all of the elements in the list are int, otherwise it will be torch.float32. This creates a strange edge case when the input is empty, but this empty input is not a valid model input regardless of dtype, and so I'm not sure if it's worth fixing this bug!