多卡训练遇到的bug ng=1单卡可以训练，多卡时遇到以下问题

[Leeon-K]

2023-01-30 06:57:42,675-rk0-launch.py#86:Rank 0 initialization finished.
2023-01-30 06:57:42,678-rk0-launch.py#86:Rank 1 initialization finished.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/changkang.li/EOD/up/main.py", line 28, in
main()
File "/home/changkang.li/EOD/up/main.py", line 22, in main
args.run(args)
File "/home/changkang.li/EOD/up/commands/train.py", line 163, in _main
launch(main, args.num_gpus_per_machine, args.num_machines, args=args, start_method=args.fork_method)
File "/home/changkang.li/EOD/up/utils/env/launch.py", line 52, in launch
mp.start_processes(
File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/EOD/up/utils/env/launch.py", line 113, in _distributed_worker
dist_helper.barrier()
File "/home//EOD/up/utils/env/dist_helper.py", line 139, in barrier
dist_barrier(*args, **kwargs)
File "/home/EOD/up/utils/env/dist_helper.py", line 124, in dist_barrier
dist.barrier(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 2709, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[Nina-yang]

Meet the same issue, looking forward to some kind suggestion.

[yqyao]

Maybe you need to provide your environment like pytorch version, cuda version, etc. @Nina-yang @Lick0920

[Nina-yang]

Maybe you need to provide your environment like pytorch version, cuda version, etc. @Nina-yang @Lick0920

torch verison: 1.8.1+cu111, cuda version 11.1, I have a machine with 8 Tesla V100-SXM2 32G, however because of the error, I'm using only 1 GPU now. It takes too long to finish training.

[yqyao]

Maybe you can try to use python3.6.9 (our version), and we will try to reproduce your issue on our machines. @Nina-yang

[Nina-yang]

Maybe you can try to use python3.6.9 (our version), and we will try to reproduce your issue on our machines. @Nina-yang
Thanks. Besides, my python version is 3.7.3

[Leeon-K]

Maybe you can try to use python3.6.9 (our version), and we will try to reproduce your issue on our machines. @Nina-yang
My environment set is as follow: python3.8.10 torch 1.10.1+rocm4.1 torchvision 0.11.2+rocm4.1
I have tried to use python3.6.9, but a new issue is happened:
`from future import annotations
^
SyntaxError: future feature annotations is not defined
I searched for a solution to this problem. Let me upgrade python to >3.7....

[yqyao]

where is the code in our repo, I think you need to re-install pytorch. @Lick0920

[Leeon-K]

Thank you, I will try to re-install pytorch.
Traceback (most recent call last):
File "/home/anaconda3/envs/up_py3.6/lib/python3.6/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/anaconda3/envs/up_py3.6/lib/python3.6/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/home/anaconda3/envs/up_py3.6/lib/python3.6/runpy.py", line 109, in _get_module_details
import(pkg_name)
File "/home/EOD/up/init.py", line 21, in
from .commands import *
File "/home/EOD/up/commands/init.py", line 5, in
from .flops import Flops # noqa
File "/home/EOD/up/commands/flops.py", line 5, in
from prettytable import PrettyTable
File "/home/.local/lib/python3.6/site-packages/prettytable-3.6.0-py3.6.egg/prettytable/init.py", line 1
from future import annotations

[hxy0307]

Maybe you can try to use python3.6.9 (our version), and we will try to reproduce your issue on our machines. @Nina-yang
My environment set is as follow: python3.8.10 torch 1.10.1+rocm4.1 torchvision 0.11.2+rocm4.1
I have tried to use python3.6.9, but a new issue is happened:
`from future import annotations
^
SyntaxError: future feature annotations is not defined
I searched for a solution to this problem. Let me upgrade python to >3.7....
您好，我现在使用的也是adm的显卡，遇到了numba无法在adm的显卡上运行，运行代码时会遇到CUDA_ERROR_NOT_INITIALIZED的错误，请问您遇到相同的错误了吗？