nextest: produce output in result even when the command fails

[phip1611]

I use cargo-nextest to produce a JUnit-style XML report. If all tests succeed inside Nix with crane via cargo-nextest, result/target/nextest/ci/rust-junit.xml contains the result file. The problem is: This file is also needed, if cargo-nextest fails. I tried nix-build --keep-failed --keep-going release.nix -A test but there is still no result folder.

A working workaround is adding || true:

  cargoNextest = crane.cargoNextest (commonArgs // {
    inherit cargoArtifacts;
    # working workaround
    cargoNextestExtraArgs = "--profile ci || true";
  });

but it is inconvenient, as I still need the test command to fail.

I'd like to:

• ask for advice and
• an update of the documentation. (I can submit an PR, once the solution is clarified) :)

[phip1611]

I've experimented with the following code:

  nextestReportWrapped = stdenv.mkDerivation {
    name = "nextest-report-wrapped";
    dontCheck = true;
    dontInstall = true;
    broken = true;
    inherit src;
    outputs = [ "out" ];
    buildPhase = ''
      mkdir -p $out/target/nextest/ci
      cp ${cargoNextest}/target/nextest/ci/rust-junit.xml $out/target/nextest/ci
      if grep -q 'failures="[1-9][0-9]*"' $out/target/nextest/ci/rust-junit.xml;
      then
        echo "FAILURE!"
        exit 1
      else
        echo "NO FAILURE!"
      fi
    '';
  };

But, no matter what I've tried, the result folder was never created.

[phip1611]

Okay, so far the best approach I could find is the following:

• two Nix attributes that execute cargo nextest, but one with || true
• in CI, execute the following:

  # never fails but produces the junit.xml file
  nix-build nix/release.nix -A particle.test-ci
  # always fails if a test fails, but has no output
  nix-build --no-out-link nix/release.nix -A particle.test >/dev/null 2>/dev/null

[ipetkov]

It's worth noting that nix build --keep-going will continue trying to build unrelated derivations; if a derivation's build script fails, Nix won't install anything (and any other derivations which depend on it will not be built either).

You may want to try adding an additional postInstall hook which installs the nextest files (outside of the compressed tarball of target) in addition to the || true short circuit

  craneLib.cargoNextest (commonArgs // {
    inherit cargoArtifacts;
    # working workaround
    cargoNextestExtraArgs = "--profile ci || true";
    postInstall = ''
      cp ./target/nextest/ci/rust-junit.xml $out/rust-junit.xml
    '';
  });

---

I suppose it may be worth exposing an allowFailure flag on cargoNextest which automatically does the short circuit when set...

[szlend]

Another thing we could do is make the package runnable by including a bin/$pname script in the test derivation that returns with the exit code of cargo test/cargo nextest. Maybe print out the path to test artifacts or copy them into cwd as well. This way you could simply nix run .#tests in CI and easily access test reports regardless of whether the tests failed or not.

[phip1611]

That would be nice! That's a good idea.