Hackathon: 2024-10-15

[ateucher]

Reporting out on earthaccess hack days.

Please create a new comment (at the very bottom, use the text box with a button that says "comment", not "reply") to write about what you did today!

Please include links to GitHub issues or pull requests relevant to your work, and use those issues or pull requests as the "source of truth" for documenting the work; this makes future readers jobs easier :) It's OK to duplicate some information while writing a summary here!

Use the "reply" feature to have a discussion under any given comment.

[betolink]

Thanks for creating this discussion @ateucher!
Some of us did an in-person brainstorming session last Thursday (more than a hackathon). We talked about a series of topics that I'll try to summarize here:

• Variable dictionary per dataset: The idea is to reduce the stops a researcher needs in order to know what is in a given dataset (granule) by having a dictionary that will contain at least name, path, type, description. This variable dictionary will have to be updated every time there is a new collection or a collection gets updated. Similar to what Qiusheng did here: https://github.com/opengeos/NASA-Earth-Data
  It won't be a simple task because data is very heterogeneous, we can start with data that can be easily loaded with xarray data-tree (netcdf, hdf5) and then try more corner cases (geotiff, multi-file granules etc). This dictionary will enable this workflow:

var_df = earthaccess.list_variables(short_name="MOD13Q1v061") # we see what is in this dataset
ds = earthaccess.load_dataset(var_df["var1"."var2", "var3"]) # we load only the variables we need (using xarray) 

and we could even do a more complete search across collections (CMR already supports var seach for some selected datasets)

datasets = earthaccess.search_vars("sea surface temperature", use_dict=True) # we can use CMR + our own dict 

Siri Jodha K. (NSIDC Scientist) mentioned that there is a working group in ESDIS working on improving this in CMR called "Variables First" initiative. I remember Christine Smit from GESDISC was involved on it.

• Plugin mechanism to extend the library: We talked about how some of the development we would like to do is not necessarily part of the library itself and will require the development of a plugin mechanism, something that @nikki-t already started and @jhkennedy has outlined. The idea is to follow the xarray model and have plugins and/or accessors that get registered into an earthaccess entry point. I think that NASA Harmony would be an ideal candidate to be the first plugin.
• Contrib repository similar to xarray-contrib: we created an org to park our projects, a little like conda-forge or xarray-contrib the goal will be to host plugin and related projects, like the variables dictionary! https://github.com/earthaccess-contrib/

I'm probably forgetting something, tagging @JessicaS11 @tsnow03 @andypbarrett @tsutterley @mikala-nsidc