ImportError: cannot import name 'driver' from 'triton.runtime' / RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):

[gg22mm]

System Info

Python 3.10.13
torch 2.0.1
torchaudio 2.0.2
torchmetrics 1.3.1
torchvision 0.15.2
tqdm 4.66.5
transformers 4.49.0

accelerate                 0.34.0
aiohttp                    3.9.3
aiosignal                  1.3.1
annotated-types            0.6.0
anyio                      4.0.0
appdirs                    1.4.4
astor                      0.8.1
async-timeout              4.0.3
attrs                      23.2.0
autoawq                    0.2.8
backoff                    2.2.1
beautifulsoup4             4.12.3
bitsandbytes               0.45.2
blinker                    1.7.0
bottle                     0.12.25
certifi                    2024.2.2
cffi                       1.16.0
chardet                    5.2.0
charset-normalizer         3.3.2
click                      8.1.7
cmake                      3.28.3
contourpy                  1.2.0
cpm-kernels                1.0.11
cryptography               42.0.5
cycler                     0.12.1
dataclasses-json           0.6.4
dataclasses-json-speakeasy 0.5.11
datasets                   2.21.0
dbus-python                1.2.16
decorator                  5.1.1
deepspeed                  0.14.0
dill                       0.3.8
distro                     1.9.0
distro-info                0.23+ubuntu1.1
docker-pycreds             0.4.0
emoji                      2.10.1
exceptiongroup             1.1.3
faiss-cpu                  1.8.0
fastapi                    0.110.0
filelock                   3.13.1
filetype                   1.2.0
FlagEmbedding              1.2.11
Flask                      3.0.2
fonttools                  4.50.0
frozenlist                 1.4.1
fsspec                     2024.2.0
gitdb                      4.0.11
GitPython                  3.1.42
greenlet                   3.0.3
h11                        0.14.0
hjson                      3.1.0
httpcore                   1.0.2
httpx                      0.25.1
huggingface-hub            0.28.1
idna                       3.6
InstructorEmbedding        1.0.1
itsdangerous               2.1.2
jieba                      0.42.1
Jinja2                     3.1.3
joblib                     1.3.2
jsonpatch                  1.33
jsonpath-python            1.0.6
jsonpointer                2.4
kiwisolver                 1.4.5
langchain                  0.1.12
langchain-community        0.0.28
langchain-core             0.1.32
langchain-text-splitters   0.0.1
langdetect                 1.0.9
langsmith                  0.1.27
lightning-utilities        0.10.1
lit                        18.1.1
loguru                     0.7.2
lxml                       5.1.0
MarkupSafe                 2.1.5
marshmallow                3.20.2
matplotlib                 3.8.3
mpmath                     1.3.0
multidict                  6.0.5
multiprocess               0.70.16
mypy-extensions            1.0.0
networkx                   3.2.1
ninja                      1.11.1.1
nltk                       3.8.1
numpy                      1.26.4
nvidia-cublas-cu11         11.10.3.66
nvidia-cuda-cupti-cu11     11.7.101
nvidia-cuda-nvrtc-cu11     11.7.99
nvidia-cuda-runtime-cu11   11.7.99
nvidia-cudnn-cu11          8.5.0.96
nvidia-cufft-cu11          10.9.0.58
nvidia-curand-cu11         10.2.10.91
nvidia-cusolver-cu11       11.4.0.1
nvidia-cusparse-cu11       11.7.4.91
nvidia-nccl-cu11           2.14.3
nvidia-nvtx-cu11           11.7.91
openai                     1.14.1
opencv-python              4.9.0.80
opt-einsum                 3.3.0
orjson                     3.9.15
packaging                  23.2
paddlepaddle-gpu           2.6.0.post117
pandas                     2.2.2
pdfkit                     0.6.1
pdfminer.six               20231228
pdfplumber                 0.11.0
Pillow                     10.1.0
pip                        24.0
protobuf                   4.25.0
psutil                     5.9.8
py-cpuinfo                 9.0.0
pyarrow                    17.0.0
pycparser                  2.21
pydantic                   2.6.4
pydantic_core              2.16.3
PyGObject                  3.36.0
PyMySQL                    1.1.0
pynvml                     11.5.0
pyparsing                  3.1.2
pypdfium2                  4.28.0
python-apt                 2.0.1+ubuntu0.20.4.1
python-dateutil            2.8.2
python-iso639              2024.2.7
python-magic               0.4.27
python-multipart           0.0.9
pytorch-lightning          1.9.5
pytz                       2024.1
PyYAML                     6.0.1
rapidfuzz                  3.6.1
redis                      5.0.8
regex                      2023.12.25
requests                   2.32.3
requests-unixsocket        0.2.0
rwkv                       0.8.25
safetensors                0.4.4
scikit-learn               1.4.1.post1
scipy                      1.12.0
sentence-transformers      3.0.1
sentencepiece              0.2.0
sentry-sdk                 1.42.0
setproctitle               1.3.3
setuptools                 68.2.2
six                        1.16.0
smmap                      5.0.1
sniffio                    1.3.0
soupsieve                  2.5
SQLAlchemy                 2.0.28
SQLAlchemy-Utils           0.41.1
starlette                  0.36.3
sympy                      1.12
tabulate                   0.9.0
tenacity                   8.2.3
threadpoolctl              3.3.0
tiktoken                   0.6.0
tokenizers                 0.21.0
torch                      2.0.1
torchaudio                 2.0.2
torchmetrics               1.3.1
torchvision                0.15.2
tqdm                       4.66.5
transformers               4.49.0
triton                     2.0.0
typing_extensions          4.9.0
typing-inspect             0.9.0
tzdata                     2024.1
unattended-upgrades        0.1
unstructured               0.12.6
unstructured-client        0.18.0
urllib3                    1.26.18
uvicorn                    0.28.0
wandb                      0.16.4
websockets                 12.0
Werkzeug                   3.0.1
wheel                      0.43.0
wrapt                      1.16.0
xxhash                     3.5.0
yarl                       1.9.4
zstandard                  0.23.0

Who can help?

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1863, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/transformers/integrations/bitsandbytes.py", line 21, in <module>
    import bitsandbytes as bnb
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/__init__.py", line 15, in <module>
    from .nn import modules
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/nn/__init__.py", line 21, in <module>
    from .triton_based_modules import (
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/nn/triton_based_modules.py", line 7, in <module>
    from bitsandbytes.triton.int8_matmul_mixed_dequantize import (
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/triton/int8_matmul_mixed_dequantize.py", line 13, in <module>
    from .matmul_perf_model import early_config_prune, estimate_matmul_time
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/triton/matmul_perf_model.py", line 10, in <module>
    from triton.runtime import driver
ImportError: cannot import name 'driver' from 'triton.runtime' (/usr/local/lib/python3.10/dist-packages/triton/runtime/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/paddle/www/txsb/api/ask/deepseek.py", line 22, in <module>
    model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 262, in _wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3698, in from_pretrained
    hf_quantizer.validate_environment(
  File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_8bit.py", line 77, in validate_environment
    from ..integrations import validate_bnb_backend_availability
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1851, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1865, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):
cannot import name 'driver' from 'triton.runtime' (/usr/local/lib/python3.10/dist-packages/triton/runtime/__init__.py)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import init_empty_weights, load_checkpoint_and_dispatch
import torch

device = "cuda"


model_name = "models/Qwen/Qwen1.5-0.5B"


from transformers import BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(       
    #load_in_8bit=True,  # 启用 8-bit 量化
     load_in_4bit=True,
)


with init_empty_weights():
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        quantization_config=quantization_config,  # 启用 8-bit 量化
        # device_map="auto",  # 自动分配设备
    ) 

Expected behavior

想在 配置 8-bit 量化，最后放放：load_checkpoint_and_dispatch

[Rocketknight1]

I suspect the cause here is an outdated version of triton!

[gg22mm]

I suspect the cause here is an outdated version of triton!

triton 2.0.0

[hawk-liu]

same problem, downgrade bitsandbytes to 0.44.0 resolve it.

[gg22mm]

same problem, downgrade bitsandbytes to 0.44.0 resolve it.

thank you very much! But another mistake was reported / 非常感谢你！但是又报了另一个错 ：

  File "/paddle/www/txsb/api/ask/deepseek_test.py", line 175, in <module>
    outputs = model.generate(**inputs, max_length=50)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3257, in _sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0