Issues using hdf5 in parallel through xarray

[anton-seaice]

When using conda/analysis3-25.01, I get lots of errors opening netcdf files through xarray.

See

https://github.com/anton-seaice/sandbox/blob/178e5b331872931d2789c28ae78991273882b80c/hovmoellers_temp_salt.ipynb

First there is all these warnings about not finding backends:

/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'cfradial1' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'datamet' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'furuno' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'gamic' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'hpl' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'iris' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'metek' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'nexradlevel2' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'odim' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'radolan' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/backends/plugins.py:110: RuntimeWarning: Engine 'rainbow' loading failed:
cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/g/data/xp65/public/apps/med_conda/envs/analysis3-25.01/lib/python3.11/site-packages/xarray/core/utils.py)
  external_backend_entrypoints = backends_dict_from_pkg(entrypoints_unique)

But it does open the files

However, when I try and access the data through dask, it fails with RuntimeError: NetCDF: HDF error

This code works in conda/analysis3-24.04

[anton-seaice]

I remembered about this - we saw this last year.

Its could be an issue with hdf 1.14.3:

COSIMA/access-om3#153 (comment)

[rbeucher]

Looking at this now

[rbeucher]

OK. We have hdf 1.14.3. I'm going to try with 1.14.4

[rbeucher]

I don't know if it is related to hdf 1.14.3.
I am struggling to get 1.14.4 in:

No cmor version with support for hdf5.1.14.4
No libgdal-netcdf with support for hdf5 1.14.4.

[anton-seaice]

What about just 1.14.2 ? Or can you do !=1.14.3 and see what happens ?

[rbeucher]

I can try. Is the issue specifically for 1.14.3?

[anton-seaice]

Looks like it:

https://github.com/spack/spack/blob/253ba0573237fa3a21596d38de95442f8e4514a5/var/spack/repos/builtin/packages/hdf5/package.py#L213-L220