Required packages when using 'local CUDA' and building on top of nvidia/cuda runtime-flavoured images

[benz0li]

Building on top of the devel-flavoured nvidia/cuda images, everything works out of the box.

Building on top of the runtime-flavoured nvidia/cuda images, I currently install the following packages¹ to make it work:

1. cuda-command-line-tools-12-4
   • 'depends on' and installs² cuda-cuobjdump-12-4 cuda-cupti-12-4 cuda-cupti-dev-12-4 cuda-gdb-12-4 cuda-nvdisasm-12-4 cuda-nvprof-12-4 cuda-sanitizer-12-4
2. cuda-libraries-dev-12-4
   • 'depends on' and additionally installs² cuda-cccl-12-4 cuda-cudart-dev-12-4 cuda-driver-dev-12-4 cuda-nvrtc-dev-12-4 cuda-opencl-dev-12-4 cuda-profiler-api-12-4 libcufft-dev-12-4 libcufile-dev-12-4 libcurand-dev-12-4 libcusolver-dev-12-4 libcusparse-dev-12-4 libnpp-dev-12-4 libnvfatbin-dev-12-4 libnvjitlink-dev-12-4 libnvjpeg-dev-12-4
3. cuda-nvcc-12-4
   • 'depends on' and additionally installs² cuda-crt-12-4 cuda-nvvm-12-4

@maleadt cuda-command-line-tools-12-4 and cuda-nvcc-12-4 are required, right? All of the cuda-libraries-dev-12-4, too? Or would a subset of these packages be sufficient?

Footnotes

1. https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ ↩

2. --no-install-recommends ↩ ↩² ↩³

[maleadt]

It's complicated; nvcc isn't strictly required, but we need the libcudadevrt.a device library which IIRC is packaged as part of NVCC. -dev versions of libraries shouldn't be required. We require the command line tools for runtime compilation and disassembly (i.e., ptxas and nvdisasm). You can see the exact list here: https://github.com/JuliaGPU/CUDA_Runtime_Discovery.jl/blob/39645f97ec6ee363532569e23696c2b73c2e0901/src/CUDA_Runtime_Discovery.jl#L513-L535 (and/or run with JULIA_DEBUG=CUDA_Runtime_Discovery to see a trace of what's being picked up by the local toolkit mode).

[benz0li]

The binaries are available:

root@4876b300ae7e:/# ls /usr/local/cuda/bin/
bin2c              crt       cuda-gdb        cuobjdump  nvcc                 nvcc.profile  nvlink  ptxas
compute-sanitizer  cudafe++  cuda-gdbserver  fatbinary  __nvcc_device_query  nvdisasm      nvprof

---

Without -dev versions of libraries: julia-gpu_no-dev-versions_log.txt

root@4876b300ae7e:/# echo $LD_LIBRARY_PATH
/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib:/usr/local/cuda/lib64
root@4876b300ae7e:/# ls /usr/local/cuda/lib64/
cmake                     libcudart.so.12          libcupti.so                libnppial.so.12.2.5.2   libnppitc.so.12           libnvperf_target.so
libaccinj64.so            libcudart.so.12.4.99     libcupti.so.12             libnppicc.so.12         libnppitc.so.12.2.5.2     libnvptxcompiler_static.a
libaccinj64.so.12.4       libcudart_static.a       libcupti.so.2024.1.1       libnppicc.so.12.2.5.2   libnpps.so.12             libnvrtc-builtins.so.12.4
libaccinj64.so.12.4.127   libcufft.so.11           libcupti_static.a          libnppidei.so.12        libnpps.so.12.2.5.2       libnvrtc-builtins.so.12.4.127
libcheckpoint.so          libcufft.so.11.2.1.3     libcurand.so.10            libnppidei.so.12.2.5.2  libnvblas.so              libnvrtc.so.12
libcublasLt.so            libcufftw.so.11          libcurand.so.10.3.5.147    libnppif.so.12          libnvblas.so.12           libnvrtc.so.12.4.127
libcublasLt.so.12         libcufftw.so.11.2.1.3    libcusolverMg.so.11        libnppif.so.12.2.5.2    libnvblas.so.12.4.2.65    libnvToolsExt.so
libcublasLt.so.12.4.2.65  libcufile_rdma.so.1      libcusolverMg.so.11.6.1.9  libnppig.so.12          libnvfatbin.so.12         libnvToolsExt.so.1
libcublasLt_static.a      libcufile_rdma.so.1.9.1  libcusolver.so.11          libnppig.so.12.2.5.2    libnvfatbin.so.12.4.127   libnvToolsExt.so.1.0.0
libcublas.so              libcufile.so.0           libcusolver.so.11.6.1.9    libnppim.so.12          libnvJitLink.so.12        libOpenCL.so.1
libcublas.so.12           libcufile.so.1.9.1       libcusparse.so.12          libnppim.so.12.2.5.2    libnvJitLink.so.12.4.127  libOpenCL.so.1.0
libcublas.so.12.4.2.65    libcuinj64.so            libcusparse.so.12.3.0.142  libnppist.so.12         libnvjpeg.so.12           libOpenCL.so.1.0.0
libcublas_static.a        libcuinj64.so.12.4       libnppc.so.12              libnppist.so.12.2.5.2   libnvjpeg.so.12.3.1.117   libpcsamplingutil.so
libcudadevrt.a            libcuinj64.so.12.4.127   libnppc.so.12.2.5.2        libnppisu.so.12         libnvperf_host.so         stubs
libcudart.so              libculibos.a             libnppial.so.12            libnppisu.so.12.2.5.2   libnvperf_host_static.a

[benz0li]

Et voilà: It seems only the -dev libraries install the link from e.g. libcufft.so to libcufft.so.11.

[maleadt]

It seems only the -dev libraries install the link from e.g. libcufft.so to libcufft.so.11

Ah yes, that's something I've encountered in the past. We should be looking for unversioned libraries though: https://github.com/JuliaGPU/CUDA_Runtime_Discovery.jl/blob/39645f97ec6ee363532569e23696c2b73c2e0901/src/CUDA_Runtime_Discovery.jl#L43-L57

The log seems to corroborate that:

┌ Debug: Looking for library cufft, no specific version, in /usr/local/cuda or /usr/local/cuda/extras/compute-sanitizer or /usr/local/cuda/compute-sanitizer
│   all_names =
│    1-element Vector{String}:
│     "libcufft.so"
│   all_locations =
│    21-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/nvvm/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
│     "/usr/local/cuda/nvvm/lib64"
│     ⋮
│     "/usr/local/cuda/compute-sanitizer/nvvm/lib"
│     "/usr/local/cuda/compute-sanitizer/lib64"
│     "/usr/local/cuda/compute-sanitizer/libx64"
│     "/usr/local/cuda/compute-sanitizer/nvvm/lib64"
│     "/usr/local/cuda/compute-sanitizer/targets/x86_64-linux/lib"
└ @ CUDA_Runtime_Discovery /usr/local/julia/local/share/julia/packages/CUDA_Runtime_Discovery/ONUpK/src/CUDA_Runtime_Discovery.jl:138
┌ Debug: Did not find cufft

Maybe we're not looking in the correct locations, although Libdl.find_library should consider all relevant directories. You can debug it more easily by using Libdl directly:

julia> using Libdl

julia> Libdl.find_library("libcufft")
"libcufft"

[benz0li]

The libraries are properly installed. IMHO the 'CUDA Runtime Discovery' does not properly detect the libraries.

@maleadt Do you agree? If so, should I open an issue or are you going to do it?

[benz0li]

Without -dev versions of libraries:

julia> using Libdl

julia> Libdl.find_library("libcufft")
""

But that's not unexpected due to JuliaLang/julia#22828, right?

[benz0li]

Is this a bug in https://github.com/JuliaGPU/CUDA_Runtime_Discovery.jl or not?

[maleadt]

Oh, of course, this is the inverse problem, there is no unversioned link. The package then does a brute-force search for versioned libraries, and we've forgotten to update those: https://github.com/JuliaGPU/CUDA_Runtime_Discovery.jl/blob/39645f97ec6ee363532569e23696c2b73c2e0901/src/CUDA_Runtime_Discovery.jl#L187-L189

Can you try if things work when you add the CUDA 12 releases to that list?

[benz0li]

Will do tomorrow and try again.

[benz0li]

I have rebuilt on top of nvidia/cuda:11.8.0 images and encounter the same problem: julia-gpu_no-dev-versions_log.txt

root@feb283fecabc:/# echo $LD_LIBRARY_PATH
/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib:/usr/local/cuda/lib64
root@feb283fecabc:/# ls /usr/local/cuda/lib64/
cmake                     libcudart.so.11.0        libcupti.so                 libnppial.so.11.8.0.86   libnppitc.so.11               libnvrtc.so.11.2
libaccinj64.so            libcudart.so.11.8.89     libcupti.so.11.8            libnppicc.so.11          libnppitc.so.11.8.0.86        libnvrtc.so.11.8.89
libaccinj64.so.11.8       libcudart_static.a       libcupti.so.2022.3.0        libnppicc.so.11.8.0.86   libnpps.so.11                 libnvToolsExt.so
libaccinj64.so.11.8.87    libcufft.so.10           libcupti_static.a           libnppidei.so.11         libnpps.so.11.8.0.86          libnvToolsExt.so.1
libcheckpoint.so          libcufft.so.10.9.0.58    libcurand.so.10             libnppidei.so.11.8.0.86  libnvblas.so                  libnvToolsExt.so.1.0.0
libcublasLt.so            libcufftw.so.10          libcurand.so.10.3.0.86      libnppif.so.11           libnvblas.so.11               libOpenCL.so
libcublasLt.so.11         libcufftw.so.10.9.0.58   libcusolverMg.so.11         libnppif.so.11.8.0.86    libnvblas.so.11.11.3.6        libOpenCL.so.1
libcublasLt.so.11.11.3.6  libcufile_rdma.so.1      libcusolverMg.so.11.4.1.48  libnppig.so.11           libnvjpeg.so.11               libOpenCL.so.1.0
libcublasLt_static.a      libcufile_rdma.so.1.4.0  libcusolver.so.11           libnppig.so.11.8.0.86    libnvjpeg.so.11.9.0.86        libOpenCL.so.1.0.0
libcublas.so              libcufile.so.0           libcusolver.so.11.4.1.48    libnppim.so.11           libnvperf_host.so             libpcsamplingutil.so
libcublas.so.11           libcufile.so.1.4.0       libcusparse.so.11           libnppim.so.11.8.0.86    libnvperf_host_static.a       stubs
libcublas.so.11.11.3.6    libcuinj64.so            libcusparse.so.11.7.5.86    libnppist.so.11          libnvperf_target.so
libcublas_static.a        libcuinj64.so.11.8       libnppc.so.11               libnppist.so.11.8.0.86   libnvptxcompiler_static.a
libcudadevrt.a            libcuinj64.so.11.8.87    libnppc.so.11.8.0.86        libnppisu.so.11          libnvrtc-builtins.so.11.8
libcudart.so              libculibos.a             libnppial.so.11             libnppisu.so.11.8.0.86   libnvrtc-builtins.so.11.8.89

---

The package then does a brute-force search for versioned libraries

CUDA v11.8.0 is in that list. searching for versioned libraries does not seem to happen.

---

I have whitelisted your GitHub account at https://edu.cuda.jupyter.b-data.ch where you can start image Julia (jupyterlab/cuda-11.8.0/julia/base:1.10.2) to reproduce the issue.
ℹ️ To allow sudo: Set environment variable GRANT_SUDO=yes (Server Options).
👉 This is a containerised environment. Only the home directory is persisted.

[benz0li]

Issue: JuliaGPU/CUDA_Runtime_Discovery.jl#10