USGS National Hydrologic Model (NHM), Docker Images

Code to build National Hydrologic Model, Docker images.

Base Docker Image

The base Docker image is currently Anaconda.

Source Code Directory

Source code is installed under /usr/local/src. Currently, this is gridmetetl, onhm-runners, onhm-verify-eval, and PRMS.

Running

To run the app, run compose-test.sh.

Running on Windows

We have not had success running the app on the Windows version of Docker. Whether this is due to the global network security configuration of Windows within .usgs.gov, or specific to Docker on Windows itself (or both), we are unsure.

If running the app on Windows within .usgs.gov, we recommend running Docker Compose on Oracle VM VirtualBox (see below).

Running on Oracle VM VirtualBox

1. install Oracle VM VirtualBox;
2. create a Linux virtual machine managed by Oracle VM VirtualBox using the installation image of distro. of your choice (we use CentOS 7, but this is not required);
3. install Git, Docker Compose and wget on the Linux virtual machine;
4. clone the docker-images repo. on the virtual machine;
5. run the compose-test.sh script as described under Running above.

Running on HPC Architecture

To build the Docker images in preparation for upload to Docker Hub, run the script build.sh.

To upload the Docker images to Docker Hub, run the script push.sh.

This might take a while, especially if you are running it via a home broadband connection.

Login to your HPC machine, and either clone the docker-images repo.:

git clone -b shifter https://github.com/nhm-usgs/docker-images.git

or run git pull to update your existing repo.:

cd docker-images
git pull

Now run the pull.sh script, to download and convert the Docker images to Shifter images.

Be aware that at this time, Shifter pull errors are quite common, so don't be surprised by FAIL messages from the pull.sh script. If this happens, run the script again, and Shifter should pick up where it left off. A new revision of Shifter promises to solve this problem.

When the required Shifter images are present, run the pipeline with the command run-shifter.

The script will submit each container run to Slurm, and stdout from the Slurm jobs will be directed to files in the current directory with a .out suffix. Logs and error messages from each container run should be saved in these files.

The nhm.env file (in this directory) contains environment variable defaults that may need to be changed for testing purposes. Comments in this file explain the function of each environment variable.