CI good practice

[ego-thales]

Hello,

In my CI workflow, amongst others, I wish to test the following:

• building
• installing built wheel
• running pytest

First question is:
Is it considered a better practice to run pytest against the installed wheel vs against the local source code?

Second question is:
If the answer to the first question is yes, how to do that? Because I'm sure manipulating the sys.path to remove . is a bad idea.

Thanks!

[webknjaz]

Yes, it's the best practice to test as installed: https://blog.ganssle.io/articles/2019/08/test-as-installed.html.

Different people do this differently, though. I've seen some just removing the locally importable thing. Others do python -I to make sure -m wouldn't mess with sys.path.

Personally, I use a combination of these and even have an action for running tests from sdists instead of Git checkouts. Here's one of my recent examples: https://github.com/ansible/awx-plugins/tree/0d569b5/.github/workflows. Note that it takes a number of things to extremes and uses tox as a workflow tool (which is what I recommend for ensuring local runs are close to the CI).

[graingert]

One option is to use an src layout to hide the source files from the import system

[ego-thales]

Thanks for both your answers and the reference!

I do use separate "project" and "test" folders, with the pytest configuration

[tool.pytest.ini_options]
testpaths = "test"

I didn't realize that this meant that pytest was ran "from test/", which automatically makes source code unavailable. I think this it what you refer to as "src layout" @graingert?

Thanks!

[graingert]

No this isn't what I meant, something more like this: https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/

[ego-thales]

Hmm I see, thanks.

I use uv forenv management. In spite of using a "flat" layout, when I run uv run pytest with the above configuration, pytest does use the installed version above source code (I tested it by installing without extra packages to notice a difference in tests). Do you know why? It is normal pytest behaviour or some uv magic?

[webknjaz]

@ego-thales that depends on a few factors:

• when you run python -m pytest, Python itself adds the current working directory to PYTHONPATH, while running pytest does not do that (which is what you do, it seems — with uv run pytest and not uv run python -m pytest; it's not uv magic).

  I like running python -Im pytest to counteract that import path influence, while still being in control of what Python interpreter is being used.

• depending on whether you have an __init__.py in your tests directory, I think pytest auto-injects things into sys.path too

With that in mind, if you know what you're doing, you can trick it into one behavior or the other, and do it however you like, really.

OTOH, when you're not sure, or you just don't care, or don't want somebody to make a mistake in the future by accident due to lack of understanding, using src-layout is your best bet.