GPU availability inside a nested docker inside dind container

[zxiiro]

While we've been able to get GPUs assigned inside the runner container, this isn't enough to allow GPU build/test to run as PyTorch's build / test code actually runs docker run to run the build / test inside a nested container inside of the runner.

Since we are deploying dind for ARC this means the dind container is actually the container that requires the GPU and then pass that dind container into docker run --gpus=all to before the build / test jobs can actually utilize the GPU. This further complicates the setup necessary to be able to support PyTorch's CI.

[zxiiro]

I've been looking into this and trying to get the GPUs loaded into the container created by the CI's docker run command has been challenging. While I am able to pass the GPU to the dind container via kubernetes and using nvidia-device-plugin. I have been unsuccessful at getting the GPUs loaded further into the nested container created by docker run that runs inside of the dind container.

To get this far we need to use a nvidia-dind supported container. I searched Docker Hub and used a container image that seemed recent. While this allowed us to load the Nvidia GPUs inside of the dind container, I got stuck at calling docker run --gpus=all.

root@linux:/# docker run -it --rm --gpus=1 --privileged ubuntu /bin/bash
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: stat failed: /proc/driver/nvidia/gpus/0000:00:1b.0: no such file or directory: unknown.

If I list the path:

root@linux:/# ls /proc/driver/nvidia/gpus/
0000:00:1e.0

We can see that nvidia-container-cli seems to pick up the wrong path when trying to load the GPUs from within a container. I wonder if nvidia-container-cli supports being run from within a container to spawn a nested container.

I've found some issues that sound similar to what we ran into but no clear resolution.

• Containers running inside the dind don't have access to the GPU ehfd/nvidia-dind#3
• nvidia-container-cli: mount error by running any nvidia-docker image; "proc/driver/nvidia/gpus/cfdd:00:00.0" bus id is diffferent in mount path and the available bus id is (03f8cfdd:00:00.0). NVIDIA/nvidia-docker#1316

[zxiiro]

I'm feel like us trying to make ARC work with PyTorch's existing CI process is causing a lot of pain to us here. My gut feeling is ARC likely wasn't designed to work with this level of multiple nested layers of containers inside of other containers.

If we could make the build/test run from inside of the runner container without nesting more containers than this we'd be able to run against the GPUs as expected.

[zxiiro]

Okay I made a bit of a breakthrough this morning regarding passing the GPU further down into the container created by docker run --gpus ....

To get past the issue I was having when using --gpus=all where it always tried to load the wrong device not matching with we see in /proc/driver/nvidia/gpus. I discovered that the command also accepts a device parameter via --gpus device=<device-id> so knowing that we can code the run command like this to successfully get the GPU loaded in:

root@linux:/# docker run -it --rm --gpus "device=$(ls /proc/driver/nvidia/gpus)" ubuntu /bin/bash
root@ad992569fc96:/# nvidia-smi
Thu May 16 14:40:20 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-SXM2-16GB           On  | 00000000:00:1C.0 Off |                    0 |
| N/A   42C    P0              36W / 300W |      0MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

[zxiiro]

With all that said I think the resolution here is the following:

1. We need to update our dind container to support GPUs somehow. Need more investigation into what makes the nvidia-dind images found on Docker Hub and applying those settings to our dind image. I think this should help https://github.com/ehfd/nvidia-dind
2. Update our ARC runner config to apply the NVIDIA GPU request to the dind container rather than the runner container.
3. Utilize the --gpus "device=$(ls /proc/driver/nvidia/gpus)" workaround to get by the docker parameter's default parsing.

[zxiiro]

One complication that I just realized is from the runner container we cannot see the /proc/drive/nvidia/gpus folder of the dind container. Another way is to do device=0,1,2,3 etc... a hacky workaround could be to try each number until we find one that works. Unless there's a way we can query the dind container from the runner container.

Another interesting thing is if we only request GPUs for the dind container but NOT the runner container, then the runner container can see all GPUs available to the host when checking /proc/driver/nvidia/gpus.