Problem while distributing machine learning

[Johnny-dai-git]

I install Mxnet Version 1.5.0 on Ubuntu 18.04 and I also use 1.50 version soucre code to run the example.

I install using command: pip3 install mxnet=1.5.0

All machine/server can be ssh to each other.

It will be fine if I try to run on the single machine.However, when I try to run distributed machine learning, there is a problem.

bash: 128.105.145.72: command not found
bash: line 0: export: 128.105.145.72': not a valid identifier bash: line 0: export: 128.105.145.69': not a valid identifier
bash: line 0: export: 128.105.145.60': not a valid identifier bash: 128.105.145.72: command not found bash: 128.105.145.72: command not found bash: line 0: export: 128.105.145.72': not a valid identifier
bash: line 0: export: 128.105.145.69': not a valid identifier bash: line 0: export: 128.105.145.60': not a valid identifier
bash: line 0: export: 128.105.145.72': not a valid identifier bash: line 0: export: 128.105.145.69': not a valid identifier
bash: line 0: export: 128.105.145.60': not a valid identifier bash: 128.105.145.72: command not found bash: line 0: export: 128.105.145.72': not a valid identifier
bash: line 0: export: 128.105.145.69': not a valid identifier bash: line 0: export: 128.105.145.60': not a valid identifier

What cause this problem and how can I solve it ?

Best Regards,
Johnny

[szha]

what example did you run?

[Johnny-dai-git]

../../tools/launch.py -n 3 -H hosts --launcher ssh --sync-dst-dir /tmp/mxnet_job/ python image_classification.py --dataset cifar10 --model vgg11 --epochs 1 --kvstore dist_sync

I run this command from the tutorial.

It seems that problem happens in the dmlc_tracker ssh module.

It happens while the ssh.submit(args) running. It should be in launch.py.

[Johnny-dai-git]

Also it is very wired that... I have four server actually. However, only one of them can be sync however, the other three can not be synced. And pompt up the above errors.

[szha]

what's the content of your hosts file? For ssh launcher, the expected content is just IP addresses. If you need to configure for different user, you need to do so ssh config.

[Johnny-dai-git]

HI, the content of my host filee is only IP address: 126.128.19.1 126.128.10.1 126.187.10.43. Kind of these format. Best Regrads, Johnny

…

On Wed, Oct 21, 2020 at 7:13 PM Sheng Zha ***@***.***> wrote: what's the content of your hosts file? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19380 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDGLB6AMUNZMZ4HPZIV35DSL5TIBANCNFSM4SWYMHXA> .

-- Yuanjun Dai (he/him) P.hd Department of Computer and Data Sciences Case Western Reserve University Phone: (216)-235-8330 Office: Glennan 505

[szha]

are you putting them one IP address per line?

[Johnny-dai-git]

No.... Based on the tutorial.... I think they should be in the same line? Oh... Each IP need to be in a new line ? Best Regards, Johnny

…

On Fri, Oct 23, 2020 at 5:30 PM Sheng Zha ***@***.***> wrote: are you putting them one IP address per line? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19380 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDGLB2JZ5RKS6O26FVEQHDSMHYZFANCNFSM4SWYMHXA> .

-- Yuanjun Dai (he/him) P.hd Department of Computer and Data Sciences Case Western Reserve University Phone: (216)-235-8330 Office: Glennan 505