Improved handling of non-writable rlib/rmeta files

[j-baker]

Problem

I use Crane to build my projects in a Nix environment. This pretty much ends up being vanilla cargo, but with one difference. Crane builds dependencies in one Nix build, storing the resulting target dir, and then moves it into place for the subsequent build. The advantage is that when subsequent builds run, they do not have to rebuild dependencies. This is convenient in a CI environment, as it provides for transparent caching.

Crane uses two mechanisms to handle this linking procedure. In the old one, it copies everything. This is fast compared to recompiling, but slow compared to a normal cargo workflow, as the data is voluminous. It additionally bloats the output size.
The other mechanism is symlinking, where it adds symlinks for each rlib/rmeta file into the Cargo target directory. These symlinks are read-only.

This mostly works, but fails when Cargo decides that a given library needs to rebuilt. Here, something like:

error: output file /private/tmp/nix-build-retina-clippy-0.0.0.drv-0/source/target/release/deps/libopencv-6a1a900b840d12b6.rmeta is not writeable -- check its permissions

might be printed by rustc. I believe this is because Cargo is telling rustc to emit to the symlink file, which is then resolved to the read-only path. Rebuilds are typically not required, but can happen due to a buggy build.rs (e.g. a build.rs which emits a non-existent file as a reason to rebuild).

Proposed Solution

I could see three potential solutions here.

1. This naively seems like a failure in content-addressability, in that Cargo is deciding it needs to rewrite files into files with the same hash. Not sure what guarantees Cargo provides here, but given how well Rust tooling is thought through in general, this seems somewhat unlikely.
2. If Cargo decides that a library needs to be rebuilt and the relevant rlib/rmeta files are not writeable/are a symlink, delete the symlink before continuing. The user's intentions are clear and a non-writeable symlink target is not the same as a non-writeable output.
3. Cargo could allow reading from alternative sources of compiled data (as some kind of fallback). Essentially, allow the user to specify an extra target dir, and Cargo looks in here for compiled data if not otherwise present. This likely suffers from concurrency issues around build-dir locking.

There are additional crane-side solutions that could be applied (one simple one might be creating the cargo target dir as an overlayfs, for example). I'd note that I don't believe this problem is exclusively crane related. I've filed a similar tracking issue on Crane itself as ipetkov/crane#385.

Notes

It's very possible that I've misunderstood the separation of responsibilities between rustc and cargo - if so, my apologies. It's also very possible that I'm considering this issue too myopically - that there is some larger improvement that would solve my workflow - I tried to be minimalistic here because piecewise minimalist solutions to problems are usually faster to resolve if project maintainers are amenable to them.

Lastly, it's possible that this is considered a reasonable wontfix - I'm surfacing here because I quite like the Crane behaviour and would love it to be less error-prone.