Lock markers and groups #9427

radoering · 2024-05-20T13:23:39Z

Pull Request Check List

Requires: #9345
Requires: python-poetry/poetry-core#733

Added tests for changed code.
Updated documentation for changed code.

First commit

Write information about the resulting groups that require a locked package and the resulting marker to the lock file. (Information is only written, not used yet.)

Second commit

Introduce a new config option installer.re-resolve (default true); when set to false the marker information from the lock file is used, there is no re-resolving anymore when installing from the lock file.

Motivation

What triggered me was the discussion at https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists and especially https://github.com/brettcannon/peps/blob/lock-file/peps/pep-9999.rst, which might become a draft for a general lock file. To support this lock file format, we have to lock the resulting markers for packages so that re-resolving is not required for installation.

Further

We can simplifiy the dependency walker in poetry-plugin-export and get rid of dependency walk failed issues.
make use of locked markers in lock file version 2.1 and thereby avoid dependency walk failed poetry-plugin-export#286
We can do other stuff like creating an extra with pinned depenencies (not sure if that is something we want or if it is more something for a plugin).
PoC: automatically create a pinned extra with dependencies pinned from the lock file #9428

Performance

We have to distinguish between locking and installing.

Locking

There is an additional step to calculate the resulting markers so this change can only make things slower. However, I could not measure a significant increase in time required for locking for several projects. I suppose that's because what we are doing in the additional step are mostly marker operations, which have probably been cached during dependency resolution before.

Installing

Evaluating markers instead of re-resolving could be faster or slower. However, I also could not measure a significant effect here. I suppose determining the exact packages to install is just not a bottleneck compared to the actual installation. Even when all packages were already installed (and installation is a noop), I could not measure a difference. However, there is a side effect:

Side effect on installation process

With re-resolving we are installing batches of packages in parallel. The batches are determined by the depth in the dependency graph (same depth -> same batch). Since we do not lock the depth, without re-resolving all packages are in the same batch.

On the one hand, this can have a positive effect on performance because we do not have to wait for the last package of a batch.
(I could observe the complete installation of the dependencies of a largish project with all packages in the cache going down from 70s to 60s.)

On the other hand, this could have a negative effect on robustness (due to race conditions). I suppose, creating batches of packages with the same depth results in only installing packages in parallel that are (more or less) independent of each other.
(If a package depends on another package, they will normally have a different depth.) Parallel installation might be an issue if two packages write to the same directory (name space packages) and try to create it at the same time.

Conclusion

Locking the resulting groups and markers opens up some new possibilities, especially for poetry-plugin-export. Installation without re-resolving is not a game changer on its own. Only the side effect will probably decrease the time for installation in some cases.

For now, I decided to make installation without re-resolving opt-in. We can also make it opt-out if we think it is safe enough and want to get more feedback.

…etry#9427)