DOC: add documentation about using shared libraries

[rgommers]

Opening this PR now to get feedback on topics and structure, not ready for detailed review yet.

We get a lot of questions about shared libraries, and this is a tricky thing to get right. So try to document how to use internal libraries as well as link to external shared libraries as well as possible. Subproject-related questions also come up more and more, and there are some extra gotchas here, so treat those as a third "source" of shared (or static) libraries.

• Closes Document how shared libraries distributed with the wheel are handled #539
• xref Building Windows wheels with libraries fails #525 and ENH: support shared libraries on Windows when explicitly enabled #551 for internal shared libraries on Windows
• xref How can I bundle a shared library in a wheel with mesonpy? #410 and Best practice for optionally bundling a library dependency #556 for optionally bundled a shared library
• xref Simple copying of windows DLLs in a wheel #629 for shared library in subproject use cases

[rgommers]

Okay, changed to os.add_dll_directory with some compat notes added. The new test case passes on all Windows configs now except for Cygwin. I don't know what's going on there - I don't think we do anything special in SciPy for Cygwin with the same kind of shared-library-inside-python-package setup. The build completes but at runtime the shared library goes missing:

[1/5] Compiling C object mypkg/cygexamplelib.dll.p/examplelib.c.o
[2/5] Compiling C object mypkg/_example.cpython-39-x86_64-cygwin.dll.p/_examplemod.c.o
[3/5] Linking target mypkg/cygexamplelib.dll
[4/5] Generating symbol file mypkg/cygexamplelib.dll.p/cygexamplelib.dll.symbols
[5/5] Linking target mypkg/_example.cpython-39-x86_64-cygwin.dll
[1/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/cygexamplelib.dll
[2/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/libexamplelib.dll.a
[3/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll
[4/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll.a
[5/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/mypkg/__init__.py
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pytest-of-runneradmin/pytest-0/mesonpy-test-venv4/lib/python3.9/site-packages/mypkg/__init__.py", line 32, in <module>
    from ._example import example_sum
ImportError: No such file or directory

@DWesl could you perhaps have a look at this? I'm not even sure it's supposed to be working, or it requires support within Cygwin somehow.

[dnicolodi]

[1/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/cygexamplelib.dl

Why does the shared library have this funny name?

[rgommers]

Why does the shared library have this funny name?

Seems like the fun of building on Windows - with MinGW we get libexamplelib.dll, with MSVC we get examplelib.dll and with Cygwin we get cygexamplelib.dll.

[dnicolodi]

Would it be worth to run pyupgrade --py38-plus as part of dropping support for Python 3.7? I've just tried and it upgrades almost all typing annotations (and introduces some bugs about encoding...) thus the noise may be not worth the simplifications.

[rgommers]

Would it be worth to run pyupgrade --py38-plus as part of dropping support for Python 3.7? I've just tried and it upgrades almost all typing annotations (and introduces some bugs about encoding...) thus the noise may be not worth the simplifications.

I'll have a look, but if that's useful enough then I'd prefer to do it in a separate PR I think.

[DWesl]

Okay, changed to os.add_dll_directory with some compat notes added. The new test case passes on all Windows configs now except for Cygwin. I don't know what's going on there - I don't think we do anything special in SciPy for Cygwin with the same kind of shared-library-inside-python-package setup. The build completes but at runtime the shared library goes missing:

[1/5] Compiling C object mypkg/cygexamplelib.dll.p/examplelib.c.o
[2/5] Compiling C object mypkg/_example.cpython-39-x86_64-cygwin.dll.p/_examplemod.c.o
[3/5] Linking target mypkg/cygexamplelib.dll
[4/5] Generating symbol file mypkg/cygexamplelib.dll.p/cygexamplelib.dll.symbols
[5/5] Linking target mypkg/_example.cpython-39-x86_64-cygwin.dll
[1/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/cygexamplelib.dll
[2/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/libexamplelib.dll.a
[3/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll
[4/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll.a
[5/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/mypkg/__init__.py
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pytest-of-runneradmin/pytest-0/mesonpy-test-venv4/lib/python3.9/site-packages/mypkg/__init__.py", line 32, in <module>
    from ._example import example_sum
ImportError: No such file or directory

@DWesl could you perhaps have a look at this? I'm not even sure it's supposed to be working, or it requires support within Cygwin somehow.

Most of the packages I've worked with have only depended on system DLLs/shared libraries, but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy. The easy things to check are permissions (both shared libraries need execute permissions to work properly) and whether the PATH is set properly to find dependent shared libraries actually linked into the extension module (ldd is the tool you're probably familiar with, though I tend to use cygcheck).

From what I remember of the last time I compiled SciPy on Cygwin a few years back, SciPy doesn't link to itself; dependencies are Python-level or header-only, so this problem wouldn't come up. I haven't run into os.add_dll_directory (in the CPython repo it looks Windows-only and would not be available on Cygwin), I've only ever directly modified PATH outside Python. I wonder whether something like:

if sys.platform == "cygwin":
    def add_dll_directory(path: str):
        os.environ["PATH"] = f"{os.environ['PATH']:s}:{path:s}"
    os.add_dll_directory = add_dll_directory

might help.

[rgommers]

Thanks for the input @DWesl!

From what I remember of the last time I compiled SciPy on Cygwin a few years back, SciPy doesn't link to itself; dependencies are Python-level or header-only, so this problem wouldn't come up.

That changed recently, the SciPy 1.14.0 release has a shared library:
https://github.com/scipy/scipy/blob/ea916c6f7f487bd53e98de082649d542cc6106ed/scipy/special/meson.build#L37

I've only ever directly modified PATH outside Python. I wonder whether something like: [...] might help

I'll try but I doubt it, since modifying os.environ shouldn't affect the existing process IIRC.

but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy.

Is there a patch/repo somewhere for how the numpy package in Cygwin is built?

[DWesl]

Thanks for the input @DWesl!

but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy.

Is there a patch/repo somewhere for how the numpy package in Cygwin is built?

For NumPy:
https://github.com/numpy/numpy/blob/main/.github/workflows/cygwin.yml

For SciPy, most recent I have:
DWesl/scipy#7

That changed recently, the SciPy 1.14.0 release has a shared library

It might still work, depending on what Windows thinks is the current directory for a dlopen call and which extensions load it. I can't test at the moment, since Cygwin doesn't have Python>=3.10 (numpy/numpy#26247)

[rgommers]

The Cygwin tests pass now, thanks for the pointers @DWesl. Amending os.environ['PATH'] worked ('LD_LIBRARY_PATH' did not), and is necessary when the shared library is in another directory and also when it is right next to a Python extension module that needs it. For regular Windows, curdir is searched by default so os.add_dll_directory is only necessary when the shared library is elsewhere.

[DWesl]

The Cygwin tests pass now, thanks for the pointers @DWesl. Amending os.environ['PATH'] worked ('LD_LIBRARY_PATH' did not),

I think LD_LIBRARY_PATH gets used by dlopen on Cygwin, but Python already has its own logic for that (sys.path), so that's not relevant here.

and is necessary when the shared library is in another directory and also when it is right next to a Python extension module that needs it. For regular Windows, curdir is searched by default so os.add_dll_directory is only necessary when the shared library is elsewhere.

It sounds like Windows does some helpful chdir things/PATH manipulation around its version of dlopen then (LoadDynamicLibraryEx maybe?), that Cygwin (and the lower-level Windows function it passes things off to) does not.
Interesting to know about in case it comes up again.

[HaoZeke]

This seems to be shaping up very nicely, and is super useful (drive by spelling nits).

[rgommers]

@dnicolodi the reviews comments were addressed a while back. WDYT about hitting the green button on this?

[dnicolodi]

Sorry for being slow on this @rgommers. There are still a couple of todo items assigned to myself in the comments and I wanted to address them before merging, mostly because otherwise I'll forget to address them. I'll try to work on this today or over the weekend. If I'll not come to it I'll merge this as it is and we will iterate on it later.

[dnicolodi]

I've added a few suggestions in the places that I thought needed some little work. @rgommers please have a look.

[dnicolodi]

I've rebased to kick cirrus CI into behaving