[release-test] GPU memory numbers are always 0 on A100 runner

[xuzhao9]

The command nvidia-smi pmon -s m -c 1 -o T does not give GPU memory numbers correctly.

https://github.com/pytorch/benchmark/blob/main/userbenchmark/release-test/monitor_proc.sh#L24

Error workflow:
https://github.com/pytorch/benchmark/actions/runs/12878326305/job/35904096937

[xuzhao9]

Running on the GCP H100 machine or devGPU, the GPU memory data is generated successfully:

$ bash ~/benchmark/userbenchmark/release-test/monitor_proc.sh python main.py --epochs 4
Running  python main.py --epochs 4 > "/tmp/tmp.fVD4eQHgYE_monitor_proc" 2>&1 &
Watching PID: 2582
Searching for spawned processes with
Tail log file with: ps --forest -o pid --ppid 2582 --pid 2582 | awk 'NR>1 {print}'
    tail -f /tmp/tmp.fVD4eQHgYE_monitor_proc

Max GPU Mem.   Max RSS Mem.   Max PSS Mem.
904            1149.47        1438.21
1438           1209.12        1438.21
1438           1209.15        1438.21
1438           1209.59        1845.58

[xuzhao9]

Debug reproduce: on the CI machine, the output is: https://github.com/pytorch/benchmark/actions/runs/12982872402/job/36203285897

#Time         gpu         pid   type     fb   ccpm    command 
#HH:MM:SS     Idx           #    C/G     MB     MB    name 
 05:53:45       0          -     -      -      -    -              

#Time         gpu         pid   type     fb   ccpm    command 
#HH:MM:SS     Idx           #    C/G     MB     MB    name 
 05:53:47       0        698     C      0      0    python  

cc @atalman @jeanschmidt

[jeanschmidt]

I can't pinpoint yet the reason you get those specific responses. Having said that, the command:

$ nvidia-smi dmon -s m -c 1 -o T -i 0
#Time         gpu     fb   bar1   ccpm
#HH:MM:SS     Idx     MB     MB     MB
 15:54:14       0    429      4      0

works, so maybe you can rely on it?