AutoModelForCasualLM.from_pretrained() exits without warning/error

[blazgocompany]

System Info

transformers 4.49.0
Python 3.11.9

Who can help?

@ArthurZucker @gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

This is what I ran:

from transformers import AutoModelForCausalLM, AutoTokenizer
import os
current_dir = os.path.dirname(os.path.abspath(__file__))
tokenizer = AutoTokenizer.from_pretrained(
    current_dir,
    use_safetensors=True,
)
print("tokenizer loaded")
model = AutoModelForCausalLM.from_pretrained(
    current_dir,
    use_safetensors=True,
)
print("model loaded")
input_ids = tokenizer("Hello, whats 23*6/4?", return_tensors="pt").input_ids
response = model.generate(
    input_ids,
    temperature=0.3,
)
print("response generated")
print(tokenizer.decode(response[0]))

I get "tokenizer loaded" but not "model loaded". the python script exits (unerrored) before that.
The current directory of the file was produced by git clone https://huggingface.co/deca-ai/2-mini-beta which does include a model.safetensors.index.json.
Looking at Task Manager, memory never goes higher than 200-300MB, so I'm not sure if it's even getting there.

Expected behavior

The model should run like normal (even if it's slow)

[Rocketknight1]

Hi, we might need a bit more info to help diagnose this! Can you:

1. Try loading the model from the Hub directly instead of a local folder
2. Try loading another model of a similar size (to make sure it's not just a memory issue)
3. Try running this on a non-Windows machine

[blazgocompany]

Ok, I can't do number 3, but for the other two, 1. Where will the model be stored if I load it directly from the Hub? Will it be cached so I can use it again without re-downloading. 2. I think I have enough RAM. By the way, I am not using a GPU. Also, as I mentioned, RAM usage never goes above 200-300MB. Even if it was a memory issue, shouldn't it error?

…

________________________________ From: Matt ***@***.***> Sent: Tuesday, February 18, 2025 6:19 AM To: huggingface/transformers ***@***.***> Cc: blazgocompany ***@***.***>; Author ***@***.***> Subject: Re: [huggingface/transformers] `AutoModelForCasualLM.from_pretrained()` exits without warning/error (Issue #36245) Hi, we might need a bit more info to help diagnose this! Can you: 1. Try loading the model from the Hub directly instead of a local folder 2. Try loading another model of a similar size (to make sure it's not just a memory issue) 3. Try running this on a non-Windows machine — Reply to this email directly, view it on GitHub<#36245 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXA4YJJF2DXSOBRRUAKTVGL2QM6PNAVCNFSM6AAAAABXKL2YGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRVHA2TEOBUGY>. You are receiving this because you authored the thread.Message ID: ***@***.***> [Rocketknight1]Rocketknight1 left a comment (huggingface/transformers#36245)<#36245 (comment)> Hi, we might need a bit more info to help diagnose this! Can you: 1. Try loading the model from the Hub directly instead of a local folder 2. Try loading another model of a similar size (to make sure it's not just a memory issue) 3. Try running this on a non-Windows machine — Reply to this email directly, view it on GitHub<#36245 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXA4YJJF2DXSOBRRUAKTVGL2QM6PNAVCNFSM6AAAAABXKL2YGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRVHA2TEOBUGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[Rocketknight1]

The model is stored in the huggingface cache, when you load from the Hub, so you won't need to redownload it after the first time. On Linux/Mac the cache is in ~/.cache/huggingface, on Windows I'm not sure but I suspect it lives in %appdata%.