HPC-Cloud-Subsetting

Project tracking repository for the NOAA HPC Subsetting Model Data in the Cloud project

Targets for End of Project

• xarray_subset_grid repository is fully functional for ROMS and FVCOM datasets, contains examples and tests that prove the subsettings method is accurate.
• xreds subset API is capable of delivering subsets 500 MB and below as netCDF, and any size as zarr.

Roadmap

Ingest

• Ingest NFOGS2
  • Add mesh_topology to the zarr datasets so they can be loaded consistently with ugrid and cf
• Ingest WCOFS

Software Development

• Create example subsetting notebooks for FVCOM and ROMS using NODD data with Xarray
  • FVCOM
  • ROMS
• Create python package for subsetting model data with their native grids using Xarray
  • https://github.com/asascience-open/xarray-subset-grid
• Create Xpublish plugin API to subset datasets on demand in the cloud
• Create custom Xpublish based service including all required API plugins (including the subsetting plugin)
• Create simple GUI for creating spatiotemporal subsets using the service
• Deploy demo functionality with existing XREDS service
  • https://nextgen-dev.ioos.us/xreds/subset_export
• Create example workflow scripts and notebooks utilizing the service
• Create API for reusing slicing logic
• Create test and validation suites for python package
• Benchmark subetting performance using test and validation cases
• Create full package documentation using python best practices
• Add CI-CD
• Vertical Subset API
  • Subset Level and Levels
  • Top Vertical Accessor
  • Bottom Vertical Accessor
  • Surface Vertical Accessor
• Add python package deployment integration with github with deployment to pypi
• Optimize performance in subsetting package
  • Dataset subsetting methodology and algorithm performance
• Expand to other grid types
  • ADCIRC
  • SELFE (Untested, but should work, CREOFS)
  • SCHISM
  • HYCOM (RTOFS)
  • Regular Grid (Test with GFS NOMADS OpenDAP)
  • 2D Grid (Test with NAM Conus OpenDAP)