Issues with openmpi

[rbeucher]

Hi @dsroberts,

I noticed that the containerized environment on hh5 is using an OpenMPI build located at /g/data/hh5/public/apps/openmpi/4.1.6. However, upon checking the build, it appears to actually be version 4.1.7.

Since the default OpenMPI on Gadi is now also version 4.1.7, there shouldn’t be any inherent difference. However, I’ve encountered an issue: there are no Conda packages available for OpenMPI 4.1.7, and most existing packages were built with links to version 4.1.6.

Here’s what I’ve tried so far:

Loading the Gadi OpenMPI module before the Singularity module in the .common_v3 script.
Linking to a copy of the OpenMPI 4.1.6 build in /g/data/xp65/public/apps/openmpi/4.1.6. (This is a copy for now, and I may need to perform a proper build here.)
Unfortunately, we’re still encountering intermittent issues, including:

Problems with OpenFabrics and Infiniband.
Significant slowdowns or errors due to missing library links.
I’m curious about the rationale behind your current implementation. I understand you’re busy with your new role, but any insights or advice would be greatly appreciated.

Thanks for your help!

[dsroberts]

Hi @rbeucher
Apologies for the delayed response, I was on leave for the second half of January. The rationale for building OpenMPI 4.1.6 was that pinning to 4.1.5 was holding back quite a few package updates. The reason I pinned to 4.1.5 was that it was the latest version of OpenMPI available in /apps at the time. OpenMPI installations are not portable, they can't be moved away from whatever path was given in the --prefix argument at configure time. They can, however, be duplicated as long as the original remains in place and accessible. What I did as part of the installation process was to replace every file from openmpi/4.1.5 in the conda environment with a symlink to the equivalent file from /apps/openmpi/4.1.5. Its kind of a combination of the 'hybrid' and 'bind' approaches detailed here: https://docs.sylabs.io/guides/4.2/user-guide/mpi.html.

I didn't particularly want to build a new version of OpenMPI, however, at the time I felt it was the least worst thing to do, as it was blocking some fairly crucial package updates. In building OpenMPI 4.1.6, I tried to get it as close as possible to NCI's OpenMPI 4.1.5 build. I used the same set of configure flags, (recovered from ompi_info -a) apart from cuda and UCX versions, which were chosen to match those already in the environment, and the same runtime configuration. I could not replicate the patches applied by NCI though, which is why there are some additional configuration environment variables that silence warnings that I believe a patched out of NCI's installations. The lack of module for the OpenMPI 4.1.6 installation is intentional, it only exists to replace the conda-installed OpenMPI in the analysis environments. It is not for building or running anything else. I can assure you it is OpenMPI 4.1.6, I did not modify the RPATH of any of the shared objects in the installation, so when another OpenMPI module is loaded, $LD_LIBRARY_PATH is set, and the orterun/ompi_info binaries will now load their libraries from the /apps OpenMPI module, hence the difference in reported version. I'm not surprised loading the Gadi OpenMPI module didn't work. mpirun is responsible for launching singularity when using the containerised environment. Without it doing that, each process would then launch singularity itself, meaning you'd now have singularity being launched up to 104 times simultaneously on each node, which causes issues around resource contention.

I hope that's helpful, the documentation for the installations is still online here: https://coecms.github.io/cms-wiki/resources/resources-conda-setup.html, though admittedly there isn't much detail about how MPI works within the environment.

Dale

[rbeucher]

Hi @dsroberts, thank you. I really appreciate you taking the time to provide such a detailed response. It gives us valuable context and will be very useful moving forward. I'll go through it carefully and let you know if I have any questions.