`NotGitRepository` error when installing multiple packages from one git repository #6958

gnuletik · 2022-11-03T19:11:00Z

Poetry version: 1.2.2
Python version: 3.10.8
OS version and name: macOS 13.0
pyproject.toml: https://gist.github.com/gnuletik/8d876426a36b9bfefee4327823c1459b

I am on the latest stable Poetry version, installed using a recommended method.
I have searched the issues of this repo and believe that this is not a duplicate.
I have consulted the FAQ and blog for any relevant entries or release notes.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

It seems that a race condition occurs when installing two packages:

from the same git repository
with a different subdirectory
on a non-default git branch

Repro:

cd /tmp
git clone https://github.com/gnuletik/poetry-lib-monorepo-issue
cd poetry-lib-monorepo-issue
poetry install

It fails with

Package operations: 2 installs, 0 updates, 0 removals

  • Installing package1 (0.1.0 c6f487b): Failed

  NotGitRepository

  No git repository was found at /private/tmp/test-poetry/.venv/src/poetry-multipackages-example

  at /opt/homebrew/Cellar/poetry/1.2.2/libexec/lib/python3.10/site-packages/dulwich/repo.py:1090 in __init__
      1086│             elif (os.path.isdir(os.path.join(root, OBJECTDIR))
      1087│                     and os.path.isdir(os.path.join(root, REFSDIR))):
      1088│                 bare = True
      1089│             else:
    → 1090│                 raise NotGitRepository(
      1091│                     "No git repository was found at %(path)s" % dict(path=root)
      1092│                 )
      1093│
      1094│         self.bare = bare

The following error occurred when trying to handle this error:

NB: output of poetry install -vvv can be found here: https://gist.github.com/gnuletik/ddcb05ff3467f022f9d3540f379763df

Please note that subsequent calls may succeed but a fresh install (after a poetry env remove --all) always fails.

The text was updated successfully, but these errors were encountered:

24rr · 2022-12-02T13:59:51Z

Based on the error message you provided, it looks like the package you are trying to install requires a git repository, but the installation process is unable to find one at the specified location: /private/tmp/test-poetry/.venv/src/poetry-multipackages-example.

To fix this error, you will need to first determine the root cause of the problem. This may involve examining the package's code, as well as the installation process, to identify any issues. It may also be helpful to consult the documentation for the package, or seek help from the package's maintainers or the community.

Once you have determined the cause of the error, you can then take the appropriate steps to fix it. This may involve modifying the package's code, changing the way it is installed, or taking some other action.

neersighted · 2022-12-02T15:33:49Z

@pneb In this case, the fault lies with Poetry; the diagnosis in the original issue appears correct to me. Related: #7113.

danieldanciu · 2023-05-12T10:18:22Z

We are also seeing this issue with a docker build that depends on multiple packages from the same git repository.

I suspect that as more and more people adopt the monorepo strategy that is now quite well supported by poetry.

None of the workarounds presented here worked for us, we had to manually serialize the installation of the packages to avoid the race condition.

gnuletik · 2023-06-08T10:59:39Z

@danieldanciu can you describe the following ?

we had to manually serialize the installation of the packages to avoid the race condition

Did you run a pip install (in your venv) before running poetry install?

pdarulewski · 2023-06-12T15:07:55Z

Are there any workarounds for this? I have multiple misc modules in a utilities repo and I'd really like to use a few of them in other projects.
The issue is pretty annoying because it's hard to pinpoint the exact problem. Especially when the installation seems to work locally but then it randomly fails in CI or in a Docker container, and after retrying, it works again.
I have the same issue for Poetry 1.3.2, 1.4.2, and 1.5.1.

gnuletik · 2023-06-12T16:18:33Z

@pdarulewski I think that the root issue is in the way poetry clone multiple dependencies in parallel.

The fix could be something that disable parallel install for dependencies that comes from the same repository.

poetry/src/poetry/installation/executor.py

Lines 71 to 77 in 6e94298

    
           if parallel is None: 
        
               parallel = config.get("installer.parallel", True) 
        
           if parallel: 
        
               self._max_workers = self._get_max_workers( 
        
                   desired_max_workers=config.get("installer.max-workers") 
        
               )

You could try to totally disable parallel installer with:

poetry config installer.parallel false

as stated here #7949 (comment)

pdarulewski · 2023-06-13T06:21:45Z

@gnuletik yes, I think so too, I guess I've had other errors related to the .git directory of the monorepo inside the project's virtualenv directory.
Setting the parallel to false seems to work, although as expected, the installation time is much slower. It's fine for now, thanks for the hint

Oblynx · 2023-09-25T07:45:51Z

This would be a great fix! We also use monorepos to handle private python packages and end up with this issue. Turning parallelism off can increase the build time x10 for a large project...

ogreyesp · 2024-01-31T17:03:29Z

@gnuletik

Setting the parallel to false didn't work in my case.

JonathanRayner · 2024-07-16T03:06:44Z

Please note that subsequent calls may succeed but a fresh install (after a poetry env remove --all) always fails.

Does anyone have any ideas on how to better consistently reproduce this? I can reproduce it sometimes locally, but not always, which is making fixing it a pain. @gnuletik I was able to reproduce it a few times with your repos, but not every time (even after deleting the environment).

*edit: I seem to be able to reproduce it more consistently running poetry install with this repo https://github.com/JonathanRayner/some_other_repo

JonathanRayner · 2024-07-16T15:56:27Z

I see a few possible ways forward, but can I ask: what is the expected behavior?

Suppose the following monorepo structure:

monorepo/pkg_1/pyproject.toml
monorepo/pkg_2/pyproject.toml

and another repo that wants to use pkg_1 and pkg_2 as git dependencies:

some_repo/pyproject.toml

which is

[tool.poetry]
name = "some_repo"
version = "0.1.0"
description = ""
authors = ["my name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.10 <3.13"

pkg_1 = {git = "[email protected]:MyOrg/monorepo.git", subdirectory = "pkg_1"}
pkg_2 = {git = "[email protected]:MyOrg/monorepo.git", subdirectory = "pkg_2"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

When the user installs some_repo, there are some possibilities of what should happen

The repo monorepo is cloned once and reused to install pkg_1 and pkg_2. This advantageous for large repos. We would need to either throw an error if pkg_1 and pkg_2 point to different branches/revs or allow for reverting to two separate clones if this is the case.
The repo monorepo is cloned twice, completely independently for pkg_1 and pkg_2.

Jozefiel · 2024-07-16T16:13:14Z

The repo monorepo is cloned once and reused to install pkg_1 and pkg_2. This advantageous for large repos. We would need to either throw an error if pkg_1 and pkg_2 point to different branches/revs or allow for reverting to two separate clones if this is the case.

The 1. option with throwing error is probably breaking change for us. We are using monorepo approach for storing microservices APIs. Then in other projects, we combine package releases (tags) based on deployment. With throwing error, monorepo approach will not be suitable anymore.

JonathanRayner · 2024-07-17T15:59:36Z

The repo monorepo is cloned once and reused to install pkg_1 and pkg_2. This advantageous for large repos. We would need to either throw an error if pkg_1 and pkg_2 point to different branches/revs or allow for reverting to two separate clones if this is the case.

The 1. option with throwing error is probably breaking change for us. We are using monorepo approach for storing microservices APIs. Then in other projects, we combine package releases (tags) based on deployment. With throwing error, monorepo approach will not be suitable anymore.

Fair! It sounds like a separate clone per parallel install is a sensible default then? ie. each package is completely separate. Perhaps people with very large monorepos use other tooling to handle reducing redundancy with multiple clones anyway?

Jozefiel · 2024-07-17T16:02:43Z

The repo monorepo is cloned once and reused to install pkg_1 and pkg_2. This advantageous for large repos. We would need to either throw an error if pkg_1 and pkg_2 point to different branches/revs or allow for reverting to two separate clones if this is the case.

The 1. option with throwing error is probably breaking change for us. We are using monorepo approach for storing microservices APIs. Then in other projects, we combine package releases (tags) based on deployment. With throwing error, monorepo approach will not be suitable anymore.

Fair! It sounds like a separate clone per parallel install is a sensible default then? ie. each package is completely separate. Perhaps people with very large monorepos use other tooling to handle reducing redundancy with multiple clones anyway?

Maybe git worktree can solve both problems?

Multiple installs from the same git repository causes arace condition when the repository is cloned. This commit only allows one operation per repository to be executed by the parallel workers. Any extra operation will be performed serially. Since the git repository is cached after the first operation, this is blazingly fast. Resolves: python-poetry#6958

github-actions · 2024-10-29T00:29:47Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

gnuletik added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Nov 3, 2022

gnuletik changed the title ~~NotGitRepository error when installing multiple packages from one git repository on a non-default branch~~ NotGitRepository error when installing multiple packages from one git repository Nov 4, 2022

neersighted mentioned this issue Dec 2, 2022

Git repository and package name mismatch #7113

Open

4 tasks

ralbertazzi mentioned this issue May 19, 2023

Poetry fails to install multiple dependencies form the same git repo in parallel #7949

Closed

4 tasks

andyhasit mentioned this issue Jul 6, 2023

No such file or directory due to race condition #8159

Open

4 tasks

dimbleby mentioned this issue Feb 1, 2024

Unable to install multiple branches of the same repository #8932

Closed

4 tasks

dimbleby mentioned this issue Apr 3, 2024

Poetry fails to install multiple packages from the same repo #9267

Closed

JonathanRayner mentioned this issue Jul 15, 2024

Bug: poetry install fails with transitive path dependencies via git #8765

Open

4 tasks

gustavgransbo mentioned this issue Aug 30, 2024

Avoid race conditions when installing multiple dependencies from the same git repository. #9658

Merged

2 tasks

radoering closed this as completed in #9658 Sep 27, 2024

github-actions bot locked as resolved and limited conversation to collaborators Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`NotGitRepository` error when installing multiple packages from one git repository #6958

`NotGitRepository` error when installing multiple packages from one git repository #6958

gnuletik commented Nov 3, 2022 •

edited

Loading

24rr commented Dec 2, 2022

neersighted commented Dec 2, 2022

danieldanciu commented May 12, 2023

gnuletik commented Jun 8, 2023

pdarulewski commented Jun 12, 2023 •

edited

Loading

gnuletik commented Jun 12, 2023

pdarulewski commented Jun 13, 2023

Oblynx commented Sep 25, 2023

ogreyesp commented Jan 31, 2024

JonathanRayner commented Jul 16, 2024 •

edited

Loading

JonathanRayner commented Jul 16, 2024

Jozefiel commented Jul 16, 2024

JonathanRayner commented Jul 17, 2024

Jozefiel commented Jul 17, 2024

github-actions bot commented Oct 29, 2024

NotGitRepository error when installing multiple packages from one git repository #6958

NotGitRepository error when installing multiple packages from one git repository #6958

Comments

gnuletik commented Nov 3, 2022 • edited Loading

Issue

24rr commented Dec 2, 2022

neersighted commented Dec 2, 2022

danieldanciu commented May 12, 2023

gnuletik commented Jun 8, 2023

pdarulewski commented Jun 12, 2023 • edited Loading

gnuletik commented Jun 12, 2023

pdarulewski commented Jun 13, 2023

Oblynx commented Sep 25, 2023

ogreyesp commented Jan 31, 2024

JonathanRayner commented Jul 16, 2024 • edited Loading

JonathanRayner commented Jul 16, 2024

Jozefiel commented Jul 16, 2024

JonathanRayner commented Jul 17, 2024

Jozefiel commented Jul 17, 2024

github-actions bot commented Oct 29, 2024

`NotGitRepository` error when installing multiple packages from one git repository #6958

`NotGitRepository` error when installing multiple packages from one git repository #6958

gnuletik commented Nov 3, 2022 •

edited

Loading

pdarulewski commented Jun 12, 2023 •

edited

Loading

JonathanRayner commented Jul 16, 2024 •

edited

Loading