Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expanding forcings tooling in datastream #109

Open
JordanLaserGit opened this issue Aug 21, 2024 · 1 comment
Open

Expanding forcings tooling in datastream #109

JordanLaserGit opened this issue Aug 21, 2024 · 1 comment

Comments

@JordanLaserGit
Copy link
Collaborator

JordanLaserGit commented Aug 21, 2024

There are (at least) two primary ways in which forcings tooling can improve in ngen-datastream:

  1. Expanding forcings sources. In the current implementation of forcingprocessor, nwmurl is called to generate URLs of national water model forcings files. So far it has been the goal to support any urlbaseinput option available in nwmurl. Reference this document to see progress on this. Ideally, forcingprocessor is capable of processing over additional grids like Analysis of Record for Calibration (AORC). Which leads to the next improvement: weight generation.

  2. Implementing weight generation: forcingprocessor extracts polygon averaged forcings data using weights provided by hfsubset. forcingprocessor is not currently capable of generating weights solely based on the polygons and forcings grid. Motivation for this choice is ngen-datastream will store the hfsubset/hydrofabric version in the metadata so the weights are effectively versioned, whereas it is not clear how to "version" user supplied weights without saving the weights themselves to the metadata. In addition, it seemed best for the community to abstract this step from users to avoid both miscalculation and needing to update our code with each hydrofabric release. This tooling primarily exists in hfsubset, though this tool is still under development and sometimes does not produce weights for some catchments.

If forcingprocessor is to support additional grids like AORC, then we will need:

  1. Expanding urlbaseinput options in nwmurl to generate URLs for AORC (and other sources?) @jameshalgren @sepehrkrz
  2. Allowing the user to supply weights manually at the CLI or a datastream supported weight generation algorithm. Should this repository support weight generation tools? I would prefer not to for reasons mentioned previously, but it might be a useful capability given current hinderances in the community. (@hellkite500 could your tools support this?)
@hellkite500
Copy link

I think there are some reasonable options for providing weight generation and using a cache mechanism to reuse them.

https://github.com/jmframe/CIROH_DL_NextGen/blob/6c3491b8bc73a9a0d60ff7cc4498cb03fcfff9ca/forcing_prep/weights.py#L20

And

https://github.com/jmframe/CIROH_DL_NextGen/blob/6c3491b8bc73a9a0d60ff7cc4498cb03fcfff9ca/forcing_prep/geo_proc.py#L59

Do basically that. In that same code base is some url wrangling for aorc that might be relevant to this issue as well

https://github.com/jmframe/CIROH_DL_NextGen/blob/6c3491b8bc73a9a0d60ff7cc4498cb03fcfff9ca/forcing_prep/generate.py#L139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants