"A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation."

[KhyamAllami]

Hi all,
I've been training on Colab Pro+ for the last few days and all has been fine. This morning upon reconnecting I got this error.
I disconnected and reconnected a few times and all is back working again, but just thought I'd report in case it was useful.
Thanks,
Khyam

/content/drive/MyDrive/RAVE_COLLAB
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/content/miniconda/lib/python3.9/site-packages/torch/cuda/__init__.py:145: UserWarning: 
A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the A100-SXM4-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Restoring states from the checkpoint path at /content/drive/MyDrive/RAVE_COLLAB/runs/arab_short_vowels/rave/version_9/checkpoints/last.ckpt
/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:342: UserWarning: The dirpath has changed from 'runs/arab_short_vowels/rave/version_9/checkpoints' to 'runs/arab_short_vowels/rave/version_15/checkpoints', therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.
  warnings.warn(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name          | Type                | Params
------------------------------------------------------
0 | pqmf          | CachedPQMF          | 8.5 K 
1 | loudness      | Loudness            | 0     
2 | encoder       | Encoder             | 4.8 M 
3 | decoder       | Generator           | 12.8 M
4 | discriminator | StackDiscriminators | 16.9 M
------------------------------------------------------
34.5 M    Trainable params
0         Non-trainable params
34.5 M    Total params
138.169   Total estimated model params size (MB)
Restored all states from the checkpoint file at /content/drive/MyDrive/RAVE_COLLAB/runs/arab_short_vowels/rave/version_9/checkpoints/last.ckpt
Traceback (most recent call last):
  File "/content/drive/MyDrive/RAVE_COLLAB/train_rave.py", line 175, in <module>
    trainer.fit(model, train, val, ckpt_path=run)
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
    self._call_and_handle_interrupt(
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
    results = self._run_stage()
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1323, in _run_stage
    return self._run_train()
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1353, in _run_train
    self.fit_loop.run()
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 199, in run
    self.on_run_start(*args, **kwargs)
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 207, in on_run_start
    self.trainer.reset_train_dataloader(self.trainer.lightning_module)
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1891, in reset_train_dataloader
    if has_len_all_ranks(self.train_dataloader, self.strategy, module)
  File "/content/miniconda/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py", line 126, in has_len_all_ranks
    if total_length == 0:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[lang216]

May I know how many samples you are using? And what is the duration of each one? Thank you!

[KhyamAllami]

Original dataset was 6229 files, 1 sec duration each, total 1 hour 41 min 9 sec
Unfortunately I deleted runs so I don't know how many were used or their durations after resampling.
The results weren't so good, trying now on something else.

[AffeZwei]

I just re-opened an issue with this exact error, because it is happening more and more often - and even with Colab Pro+ I am getting disconnected run-times. Really annoying. My suspicion is that every time you connect you get a different engine, and there is one that no longer is compatible with this version of Pytorch. If you try later you will be able to reconnect, but the problem keeps recurring. I wish I were better at troubleshooting this, perhaps someone else can help?

[KhyamAllami]

I'm afraid I'm no good with the trouble shooting but if there's anything else I can do please let me know.

[raniczny]

Hey @KhyamAllami @cloudchamber43 , you'll need to reinstall torch with a newer CUDA version. The one that gets installed is coming with 10.2, you'll need >11.

For Colab looks like you can workaround it by adding the line below to the end of 2nd code cell:

!/content/miniconda/bin/pip install torch==1.11.0+cu113 --extra-index-url https://download.pytorch.org/whl/cu113

But I'm not sure if that will break compatibility with other (older) GPUs (not using Colab anymore so can't check that).

[KhyamAllami]

Thanks, will give it a go and report back.

[AffeZwei]

Yes, thanks! I will also give it a try.

[schmolmo]

Worked nicely for me, thanks!

[wnmurphy]

If pip install doesn't work with appending the CUDA version like +cu113, then you can try grabbing from pytorch's website with:

pip install torch==1.13.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

[mamunctg]

@KhyamAllami
I solved a similar problem by upgrading Pytorch and the CUDA version.

$ conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch

[Cassieyy]

Perhaps you can find the rigjt match for the CUDA version and Torch version on https://pytorch.org/get-started/previous-versions/.

[Cassieyy]

您的邮件已收到哦我会及时处理哒~