Address issue #540

[DwarKapex]

Address issues #540

[yhtang]

Please also configure the submitted job script so that the job would fail if an individual command in it fails. Otherwise, the job will be successful from the eyes of SLURM regardless of whether the runner did its work.

[DwarKapex]

Please also configure the submitted job script so that the job would fail if an individual command in it fails. Otherwise, the job will be successful from the eyes of SLURM regardless of whether the runner did its work.

IIUC, slurm job state shows what happens with the job. The state covers all scenarios including failure of individual commands, allocation failures, timeout and etc. IMO, we are good here.

[yhtang]

Please also configure the submitted job script so that the job would fail if an individual command in it fails. Otherwise, the job will be successful from the eyes of SLURM regardless of whether the runner did its work.

IIUC, slurm job state shows what happens with the job. The state covers all scenarios including failure of individual commands, allocation failures, timeout and etc. IMO, we are good here.

That doc concerns the overall status of the job script, which can be different from the exit status of individual commands in the script. For example:

sbatch --parsable <<EOF
#!/bin/bash
#SBATCH -p <partition>
cat non-existing.txt
EOF

leads to

$ sacct -XP -j <job_id> -o State
State
FAILED

while

sbatch --parsable <<EOF
#!/bin/bash
#SBATCH -p <partition>
cat non-existing.txt
echo Hello
EOF

leads to

$ sacct -XP -j <job_id> -o State
State
COMPLETED

The difference is that in the second example, the exit status of the subsequent echo command replaces the exit status of the failing cat command.

A simple fix for this situation is to add set -e -o pipefail to the batch script.

What is more important is that the docker run command, which starts a GitHub Actions runner instance, may gracefully handle a registration/runtime error, report it in the log, and then exits without an explicit non-zero status code. For this, I'm not 100% what the specific behavior is so please check the documentation of self-hosted runners and figure out a solution accordingly.