Scope and Plan for GSoC 2024 Contribution

[SorooshMani-NOAA]

@ChrisBarker-NOAA @AtiehAlipour-NOAA let's discuss the project plan here. I think given the discussion in IOOS DMAC and before it, we can start with these generic points:

• Standardization of output datasets
• Using for data on cloud, or any other type of online service, as well as data on local machine
• Can run locally as API or as online service (e.g. on cloud where it's close to the data)
• Types of output formats? API, or files, etc.
• On the fly calculation
• Support for different OFS outputs

[AtiehAlipour-NOAA]

May 23, 2024 Meeting Notes:

• We need to prioritize the list of features for xarray-subset-grid in the README file.

• Start implementing xarray-subset-grid on STOFS-2D-Global; it has a triangular mesh.

• More information on STOFS outputs.
  STOFS-2D-Global location on cloud: https://noaa-gestofs-pds.s3.amazonaws.com/index.html
  Example file: estofs.t00z.fields.cwl.nc
  More info: https://noaa-gestofs-pds.s3.amazonaws.com/README.html

• Mat is the owner of the xarray-subset-grid repo, so reach out to him if you need permission.

• There is a separate projects on chucking data. If we find the subseting process on STOFS very slow, we might use kerchunk data as well.

• Chris has started looking into the project, learning some cool development tools during IOOS Code Sprint. Will share those in the README file.

• We should keep an eye on the JRC method on subsetting. It seems to be a preprocessing step. JRC team are the developers of Thalassa, Seravey, etc. packages.

• General project communication through Slack and technical questions on the repo.

[AtiehAlipour-NOAA]

June 06, 2024 Meeting Notes:

• Omkar provided an update on his progress:
  - Mostly done with "Adding the benchmark example", will submit the progress on that one soon.
  - Working on "Add ADCIRC Compatibility", comparing subset-grid package with Thalassa for subsetting. Currently getting different
  results. For small files, the subsetting time is similar, around 20 to 30 seconds. When working on larger files, Thalassa took the same time, 30 to 40 seconds, but for xarray_subset-grid, it took a few minutes. Here is the notebook Omkar is working on.
  - Matt will take a look at Omkar's code and help Omkar to see what the issue is.
  - Omkar is reading data directly from the cloud.
• Matt: Let's keep all communication regarding the coding on GitHub so later everyone can use it.
• Atieh will email the code that the CERA website uses for chunking STOFS data before visualizing.
• Some of the STOFS output has missing node information; we need to get that information from other STOFS files. There is an issue that we are adding those capabilities to xarray-subset-grid: Create API for reusing subset indexing on other datasets #20
• Chris solved some bugs in xarray-subset-grid.

[AtiehAlipour-NOAA]

June 20, 2024 Meeting Notes:

• The problem for using xarray-subset-grid is fixed (Stofs 2D Support #31), and the example code for subsetting STOFS-2d-global is available here: https://github.com/asascience-open/xarray-subset-grid/blob/main/examples/stofs_2d.ipynb.
• Currently working on STOFS-3D-Atl, having errors using Thalassa and ugrid.
• Ugrid and Thalassa both assume the index starts at 1, but we do not want to hard-code the starting index; for example, Delf model might start from zero, and the CF standard specifies that indexing should start from zero. We need to ensure CF compatibility and check attributes. For example, we need to change indexing in STOFS-2d from 1 to zero.
• xarray-subset-gridcore code expecting data to be fully CF compliant. If it is not fully compliant, we need to first make it compliant before passing it to the core code. This process applies to all models.
• In some STOFS-3d-Atl cases, the element file or node connectivity might be incomplete or nonexistent. This information can be found in the shout-adcirc.nc file (more details available here: https://github.com/ec-jrc/Thalassa/issues/88, https://github.com/AtiehAlipour-NOAA/Subsetting/blob/main/notebooks/KeyBridge_Visualization.ipynb).
• out2d files contain full data in 2d variables, while temperature and salinity files are separate and lack node connectivity information. More information can be found in the readme file: https://noaa-nos-stofs3d-pds.s3.amazonaws.com/README.html.
• One solution is to first read files with complete information, and then read files with missing information like temperature for subsetting. We can write a wrapper for this.
• Let’s start by working on the STOFS-3d-Atl file: noaa-nos-stofs3d-pds/STOFS-3D-Atl/stofs_3d_atl.20240619/stofs_3d_atl.t12z.fields.out2d_f037_048.nc.
• Using fsspec speeds up the code rather than s3fs.
• Saving files takes more time than subsetting; reindexing the nodes and chunking data appropriately can help with that process.
• It would be great if Omkar could write a blog post, and for more details, see https://gist.github.com/callumrollo/ee28f7e1863d30162bdb041156314543 and https://callumrollo.github.io/gliderpy.html.

[AtiehAlipour-NOAA]

July 05, 2024 Meeting Notes:

• Omkar individually met with Matt, got clarity, and is working on STOFS-3D-Atl.
• Omkar is also working on xarray-subset-grid documentation.
• Atieh tested the example code for STOFS-2D. It is working when used a different file and 'anam=true' is added. Compared it with Thalassa for different files and found comparable run times.
  

[AtiehAlipour-NOAA]

July 18, 2024 Meeting Notes:

• Omkar is working on adding examples for subsetting STOFS-3D-Atl (SCHISM). He's having issues with face connectivity, and the subsetting code cannot run properly. Omkar tried to manually collect x and y coordinates and elements from the shout_adcirc file, but the results differ from Thalassa's. Atieh: Do not rely only on comparing the subsetting values from Thalassa's results. Sometimes the Thalassa package gives incorrect values when it fails to subset data properly. Visualization is one way to validate. You can follow this example for STOFS-3D-Atl subsetting and visualization: https://github.com/AtiehAlipour-NOAA/Subsetting/blob/main/notebooks/KeyBridge_Visualization.ipynb

• Saeed: We should subset directly from the native SCHISM output. We can coordinate with SCHISM developers and get some help from them. SCHISM native output includes both quadratic and triangular elements.

• Link to information on STOFS-3D-Atl: https://noaa-nos-stofs3d-pds.s3.amazonaws.com/index.html#STOFS-3D-Atl/

• Link to STOFS-3D-Atl outputs: https://noaa-nos-stofs3d-pds.s3.amazonaws.com/index.html#STOFS-3D-Atl/

• Regarding Omkar's question about the face connectivity issue he raised with STOFS-3D-Atl, the problem lies in the node connectivity indices being floats instead of integers, possibly due to NAN values (because some elements are triangular, not quads). We can assume that node connectivity should be integers; if they are not, we can convert NANs to -1 as fill values (or whatever Ugrid expects). We should discuss this issue with SCHISM developers.

• Saeed, we can include SCHISM developers in the next meeting. In general, feel free to ask them any questions.

• Chris: We may need to write a STOFS-3D-to-Ugrid converter if necessary.

• In the documentation website, we should include all descriptions and details. Chris: Using autodoc might be an option.

• Saeed: In the Thalassa backend code, there are scripts that can convert quadratic meshes to triangular if needed. If subsetting quads is difficult, that might be an option.

• Chris and Matt: There are example codes to visualize the subsetted output:
  https://github.com/asascience-open/xarray-subset-grid/tree/main/xarray_subset_grid/visualization
  https://github.com/asascience-open/xarray-subset-grid/blob/main/examples/fvcom.ipynb

• Atieh: It would be great if Omkar could add visualization code to STOFS-2D and 3D examples.

• If possible, Matt, please label all pull requests and issues Omkar is working on with the GSOC-24 label, as Micah suggested on Slack.

[AtiehAlipour-NOAA]

Aug 02, 2024 Meeting Notes:

• Everything works well when we use the package for shadow forecast outputs.
• The temperature and salinity files have the dimensions information in the shadow model.
• The next step is to create a selector so we do not need to recalculate subsets for salinity and temperature. We will only calculate them once and use them to subset other files.
• Another step is to add plots to the STOFS example files to help validate the subsetting results.
• Omkar has added documentation to the repository as well.
• Another option for Omkar is to define some predefined regions and save them as pickle files. This would allow us to save subset data for those regions, avoiding the need to recalculate the subset and saving time, making it a powerful tool.
• Since the shadow model has some differences compared to the STOFS-3D-Atl operational model (e.g., they have different meshes), we need to test if the subsetting codes work with operational data as well. We should use the out2d files available in the STOFS-3D-Atl operational directories.
• We should use STOFS-3D-Atl operational data and create a selector based on that.
• The package supports both vertical and horizontal subsetting. We can specify which variable from which layer and region we want to subset.
• For vertical subsetting, we currently use layers such as surface velocity, bottom velocity, or entire layers. We do not yet have functionality to subset from z1 to z2. This could be another task for Omkar.
• The pyschism package can be used for visualizing SCHISM outputs (pyschism GitHub). The only issue is plotting both quadratic and triangular meshes.
• The thalassa package generates interactive maps.
• The idea behind the selector is that if someone requests a subset from the data, it computes and selects the indices, saving them in a pickle file. Then, if someone requests the same region again, it uses the saved file instead of recalculating, making the tool very efficient.
• We can add different examples for STOFS. We should come up with a list of use cases that require examples (Atieh will coordinate with Saeed about that) and then add those examples to the script.
• Omkar: How do I add documentation to the package? Chris: Post it as an issue for Matt.
• Google Colab has issues with creating interactive plots.
• Atieh tested the example code for STOFS-2D. It works when using a different file and adding 'anam=true'. Compared to Thalassa for different files, the run times are comparable. Chris: Add this to an issue.
  

[AtiehAlipour-NOAA]

@ChrisBarker-NOAA @mpiannucci @omkar-334

Based on my discussion with the STOFS team, here is a list of features that would be great to include in the STOFS example notebook. Thank you very much for your help and time.

For STOFS-2D-Global:

An example that includes the code and explanations for:

• Reading different STOFS-2D-Global forecast water level data from the cloud ('fields.cwl', 'fields.htp', 'fields.swl')
• Subsetting data for a given bounding box
• Visualizing the subset data
• Saving data in .nc format
• Reading and subsetting data for a predefined region (using pickle files).

For STOFS-3D-Atl:

An example that includes the code and explanations for:

• Reading different STOFS-3D-Atl forecast water level, temperature, salinity, velocity data from the cloud
• Subsetting data for a given bounding box
• Subsetting data for a specific variable
• Visualizing the subset data
• Saving data in .nc format
• Reading and subsetting data for a predefined region (using pickle files).

[AtiehAlipour-NOAA]

Aug 15, 2024 Meeting Notes:

• The end of the project is close, and Omkar is expected to submit either a repository or documentation this week.
• It would be great if Omkar could present the example code and walk the STOFS team through the Jupyter notebook.
• The example code for STOFS-3D-Atl is ready, including some visualizations. Currently waiting for testing and merging.
• The visualization in the FVCOM code did not work for STOFS-3D. Matt suggested that converting the quads to triangles could be a solution. The method used in Thalassa might be helpful: https://gist.github.com/pmav99/5ded91f18ef096b080b2ed45598c7d1c.
• The selector for STOFS-3D is complete and has been tested for both operational and shadow models. Matt suggested adding an ID to the selector and optionally a name, which would be very useful to use the selector.
• The selector is for spatial subsetting.
• Omkar is also creating a test to check the selector.
• Atieh encountered an issue when trying to subset SWL data for STOFS-2D-Global: Add examples for subsetting STOFS outputs #16. Matt mentioned that, similar to STOFS-3D, the selector should be used to get node connectivity from another file. Including an example for subsetting the STOFS-2D-Global SWL file would be very helpful.
• Saving the nc file is taking a lot of time because it downloads all the data. If the original data were chunked better, it would be faster. Suggested adding a note at the top of the relevant code line indicating that it will take time to save the data so that users skip this line if they don't need to save the subset data.
• Matt is changing his position but will continue working on the GSoC project.
• Use git to install and reinstall the package.
• Omkar has added the documentation.
• To speed up the process, consider using Dask.
• Example web for subsetting and visualizing subset output on a web server can be found at: https://nextgen-dev.ioos.us/xreds/subset_export . This tool allows you to subset data on the cloud without downloading it, using the following APIs:
  • https://github.com/asascience-open/xreds/blob/main/xreds/plugins/subset_plugin.py
  • https://github.com/asascience-open/xreds/blob/main/xreds/plugins/export.py
• If Omkar has time, Matt is willing to walk him through the code.
• Next Steps:
  • Add ID/region name to the selector.
  • Complete the examples.

[AtiehAlipour-NOAA]

Hi all, in case you haven't received the news: STOFS 2d Global output in Zarr is available here: https://noaa-nodd-kerchunk-pds.s3.amazonaws.com/index.html#stofs/