DOC: add documentation about using shared libraries #700
Conversation
rgommers
force-pushed
the
doc-sharedlibs
branch
from
2dd3221 to
d254472
Compare
rgommers
force-pushed
the
doc-sharedlibs
branch
2 times, most recently
from
44f372e to
89dbcc7
Compare
|
Okay, changed to @DWesl could you perhaps have a look at this? I'm not even sure it's supposed to be working, or it requires support within Cygwin somehow. |
Why does the shared library have this funny name? |
Seems like the fun of building on Windows - with MinGW we get |
rgommers
force-pushed
the
doc-sharedlibs
branch
2 times, most recently
from
ddaea0f to
f95d160
Compare
|
Would it be worth to run |
I'll have a look, but if that's useful enough then I'd prefer to do it in a separate PR I think. |
Most of the packages I've worked with have only depended on system DLLs/shared libraries, but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy. The easy things to check are permissions (both shared libraries need execute permissions to work properly) and whether the From what I remember of the last time I compiled SciPy on Cygwin a few years back, SciPy doesn't link to itself; dependencies are Python-level or header-only, so this problem wouldn't come up. I haven't run into if sys.platform == "cygwin":
def add_dll_directory(path: str):
os.environ["PATH"] = f"{os.environ['PATH']:s}:{path:s}"
os.add_dll_directory = add_dll_directorymight help. |
|
Thanks for the input @DWesl!
That changed recently, the SciPy 1.14.0 release has a shared library:
I'll try but I doubt it, since modifying
Is there a patch/repo somewhere for how the |
For NumPy: For SciPy, most recent I have:
It might still work, depending on what Windows thinks is the current directory for a |
rgommers
force-pushed
the
doc-sharedlibs
branch
from
f95d160 to
83e194b
Compare
|
The Cygwin tests pass now, thanks for the pointers @DWesl. Amending |
I think
It sounds like Windows does some helpful |
tests/packages/link-library-in-subproject/subprojects/bar/examplelib.c
Outdated
Show resolved
Hide resolved
|
@rgommers, just wanted to thank you for writing these docs! It has been key in getting the Windows builds in basnijholt/pfapack#21 to work. If only I found this PR 5 hours ago 😅 |
rgommers
force-pushed
the
doc-sharedlibs
branch
2 times, most recently
from
0e43351 to
7f0cc23
Compare
|
Thanks for the review @dnicolodi, and for the feedback @basnijholt! |
There was a problem hiding this comment.
Thanks @rgommers, especially for bearing with me for the tedious work of keeping indentation consistent. I've spotted a couple more things that may warrant some attention. Otherwise I think we can just merge this and iterate on it in follow up patches, if needed.
There is another tedious thing I have noticed: the copyright years in some files are wrong. I don't care and I remember suggesting quite strongly that having the copyright year in the SPDX headers is just a waste of time, but I lost that battle 😃 and if we have the years we should have them accurate.
| standalone library, then the method shown above won't work - the library has to | ||
| be installed to the default ``libdir`` location. In that case, ``meson-python`` | ||
| will detect that the library is going to be installed to ``libdir`` - which is | ||
| not a recommended install location for wheels, and not supported by |
There was a problem hiding this comment.
I think "recommended install location" is a bit weak: there is no way for a Python wheel to install in libdir. I'm pointing it out because I foresee someone reading this and interpreting it as meson-python enforcing a recommendation as a strict rule and asking for this to be supported (we already had cases of this interpretation and of this request in the issue tracker). Maybe a stronger wording is justified here.
There was a problem hiding this comment.
There is a way: use the data directory. Unpack an mkl or pybind11-global wheel to verify - they install into prefix/lib and prefix/include. It is indeed a recommendation that we enforce as of today.
There was a problem hiding this comment.
I think we may be referring to two different things. libdir as defined earlier in this document is a system wide installation location and the location where meson would install a shared library. What these wheels do is to install in the lib directory of the current Python environment, which is not the same thing.
There was a problem hiding this comment.
Ah okay. Yeah not the same, although it does end up being the same if you're not using a virtualenv on top of your system Python.
Let me think about how to rephrase this.
There was a problem hiding this comment.
It is definitely not at all the same thing. The issue isn't that wheels cannot install outside of site-packages. They absolutely can do that by brute force, as long as the location is still in sys.prefix. The issue is that even if you assume the garden path of sys.prefix being /use and wanting to install to a location within /usr...
... on some systems this is /usr/lib, and on other systems it is /usr/lib64, and on yet more systems it is /usr/lib/x86_64-linux-gnu
There are more configurations too, this is just the most common ones for Linux specifically.
If you have system-specific wheels then this all is entirely fine. It's not physically possible to design a wheel that is simultaneously installable on Arch Linux and on Debian and on Gentoo, without adding new features to the wheel standard (described in a PEP that was rejected years ago).
There was a problem hiding this comment.
I rephrased this section and make it a bit shorter as well - please take another look.
Also, I defined what libdir means right above its first usage, in the table.
|
|
||
| If this happens, workarounds include adding the directory that the shared | ||
| library is located in to the search path by amending the ``LD_LIBRARY_PATH`` | ||
| environment variable, or by using a compile flag ``-Wl,-rpath,/path/to/sharedlib/dir``. |
There was a problem hiding this comment.
I don't understand this note. Adding an RPATH is equivalent to setting LD_LIBRARY_PATH and the compiler flag is just a way to add to RPATH. Without studying them in detail, what is going on in the linked issues is that RPATH is absolute/relative but the user would like it to be relative/absolute, respectively.
The only way for the library to go missing at load time is when the RPATH is relative and the library install location is changed or the library is not at the location it was when the artifact linking to it was built.
Am I missing something?
There was a problem hiding this comment.
What is going on is that C/C++/Fortran libraries like zlib, openblas, etc. expect to be installed into a path that's on the loader search path. As a result, the link line of their .pc file doesn't include any RPATH. It will include something like -lmylib or -L/path/to/mylib -lmylib. This then works fine when the library is indeed in /usr/lib/ or another standard location.
However, if you build such a library yourself and put it elsewhere, an RPATH is needed, but not present at all. Meson isn't aware of what directories are or aren't in the loader search path, and hence it's not able to add (or avoid stripping) of RPATHs to such directories at install time. This is particularly confusing because Autotools does know how to do this (xref mesonbuild/meson#13046 (comment)).
It's actually not uncommon for this to bite you in practice if your Python package depends on a shared library, because you'll have it in some custom directory as soon as you want to do any development on it (or put it into its own wheel, like we did with scipy-openblas32).
There was a problem hiding this comment.
I understand the problem you are describing in your latest comment, but I am not able to make the connection from what this PR adds to the documentation and the problem you describe. The documentation added here, almost literally says "An RPATH entry alone is not always enough... if this is the case, add an RPATH" because setting LD_LIBRARY_PATH or the compiler flag just do that.
I now understand what you have in mind, but I think the explanation in the documentation is a bit too terse: I think I have an above average understanding of these issues and still I could not make the connection. I'll try to come up with a concrete suggestion on how to explain this.
There was a problem hiding this comment.
It's always enough to add your own RPATH by adding -Wl,-rpath=... to your personal LDFLAGS with the same value that you'd add to LD_LIBRARY_PATH, I guess.
The problem is not with rpath, it's with assuming that rpath will as of today be configured out of the box for you.
There was a problem hiding this comment.
hence it's not able to add (or avoid stripping) of RPATHs to such directories at install time
Once upon a long time ago meson stripped rpaths unless they were listed in the install_rpath in meson.
These days it fixed that, and now only strips rpaths that are listed exclusively in build_rpath (or point to locations inside the build tree).
So your note is not true for versions of meson that meson-python supports. Your manually added rpaths will NOT be stripped.
There was a problem hiding this comment.
The problem is not with rpath, it's with assuming that rpath will as of today be configured out of the box for you.
Ah, that's a helpful way of phrasing it I think, thanks.
There was a problem hiding this comment.
I rewrote this section - shorter now, and hopefully more correct and easier to understand.
IIRC what I did is for files that I copied from another test package to keep the year unchanged, and for files that I wrote from scratch I added 2024. Happy to do it some other way (2024 for everything?), but I did try to follow a consistent rule at least. No preference from my side - please let me know what you prefer here. |
It makes sense. I don't have a strong opinion. I would just remove the copyright years, but IANAL. Things are good as they are. Thanks @rgommers |
rgommers
force-pushed
the
doc-sharedlibs
branch
from
7f0cc23 to
e6b52a9
Compare
|
Thanks a lot for these very helpful and easy-to-understand instructions! |
Note that for Meson versions older than 1.2.0, CI failed with:
```
mesonpy.BuildError: Could not map installation path to an equivalent wheel directory: '{libdir_static}/libexamplelib.a'
```
because the `--skip-subprojects` install option isn't honored.
Hence the test skip on older versions.
In addition, the `c_shared_libs` usage requires Meson 1.3.0
dnicolodi
force-pushed
the
doc-sharedlibs
branch
from
e6b52a9 to
e577eac
Compare
|
@dnicolodi the reviews comments were addressed a while back. WDYT about hitting the green button on this? |
|
Sorry for being slow on this @rgommers. There are still a couple of todo items assigned to myself in the comments and I wanted to address them before merging, mostly because otherwise I'll forget to address them. I'll try to work on this today or over the weekend. If I'll not come to it I'll merge this as it is and we will iterate on it later. |
| +-----------------------+------------------+---------+-------+-------+ | ||
| | shared library source | install location | Windows | macOS | Linux | | ||
| +=======================+==================+=========+=======+=======+ | ||
| | internal | libdir | no (1) | ✓ | ✓ | |
There was a problem hiding this comment.
I think there is a very serious lack of use of emojis in our documentation 😄. I was thinking to use emojis for the check mark columns of this table ✔️ and 🚧 maybe. The easy way to add these in rest would be to add unicode substitution directives https://docutils.sourceforge.io/docs/ref/rst/directives.html#unicode-character-codes
|
|
||
| If an internal shared library is not only used as part of a Python package, but | ||
| for example also as a regular shared library in a C/C++ project or as a | ||
| standalone library, then the method shown above won't work. The library is` |
There was a problem hiding this comment.
| standalone library, then the method shown above won't work. The library is` | |
| standalone library, then the method shown above won't work. The library is |
| 1: Internal shared libraries on Windows cannot be automatically handled | ||
| correctly, and currently ``meson-python`` therefore raises an error for them. | ||
| `PR meson-python#551 <https://github.com/mesonbuild/meson-python/pull/551>`__ | ||
| may improve that situation in the near future. | ||
|
|
||
| 2: External shared libraries require ``delvewheel`` usage on Windows (or | ||
| some equivalent way, like amending the DLL search path to include the directory | ||
| in which the external shared library is located). Due to the lack of RPATH | ||
| support on Windows, there is no good way around this. |
There was a problem hiding this comment.
Should this be formatted as an enumerated list?
|
|
||
| 2: External shared libraries require ``delvewheel`` usage on Windows (or | ||
| some equivalent way, like amending the DLL search path to include the directory | ||
| in which the external shared library is located). Due to the lack of RPATH |
There was a problem hiding this comment.
Should we assume that readers are familiar with the concept of RPATH? It may be something unknown to users working predominantly on Windows.
| Hence you'll need to add the needed RPATH yourself, for example by adding | ||
| ``-Wl,rpath=/path/to/dir/sharedlib/is/in`` to ``LDFLAGS`` before starting | ||
| the build. In case you run into this problem after a wheel is built and | ||
| installed, adding that same path to ``LD_LIBRARY_PATH`` is a quick way of |
There was a problem hiding this comment.
| installed, adding that same path to ``LD_LIBRARY_PATH`` is a quick way of | |
| installed, adding that same path to the ``LD_LIBRARY_PATH`` environment variable is a quick way of |
|
|
||
| On Windows, the solution is similar - the shared library can either be | ||
| preloaded, or the directory that the library is in added to ``PATH`` or with | ||
| ``os.add_dll_directory``, or vendored into the wheel with ``delvewheel`` in |
There was a problem hiding this comment.
Is this true? I thought that Python on Windows does not load DLLs from %PATH% because of security concerns.
|
|
||
|
|
||
| # start-literalinclude | ||
| def _enable_sharedlib_loading(): |
There was a problem hiding this comment.
I find the function name a bit misleading. What the function does is to add a path to the list of paths where shared libraries are searched. The capability of loading shared library loading is unaffected. I would name this _append_to_sharedlib_load_path() or something like this.
| """ | ||
| Ensure the shared libraries in this dir and the ``sub`` subdir can be | ||
| loaded on Windows. | ||
|
|
||
| One shared library is installed alongside this ``__init__.py`` file. | ||
| Windows can load it because it searches for DLLs in the directory where a | ||
| ``.pyd`` (Python extension module) is located in. Cygwin does not though. | ||
| For a shared library in another directory inside the package, Windows also | ||
| needs a hint. | ||
|
|
||
| This function is Windows-specific due to lack of RPATH support on Windows. | ||
| It cannot find shared libraries installed within wheels unless we either | ||
| amend the DLL search path or pre-load the DLL. | ||
|
|
||
| Note that ``delvewheel`` inserts a similar snippet into the main | ||
| ``__init__.py`` of a package when it vendors external shared libraries. | ||
|
|
||
| .. note:: | ||
|
|
||
| `os.add_dll_directory` is only available for Python >=3.8, and with | ||
| the Conda ``python`` packages only works as advertised for >=3.10. | ||
| If you require support for older versions, pre-loading the DLL | ||
| with `ctypes.WinDLL` may be preferred (the SciPy code base has an | ||
| example of this). | ||
| """ |
There was a problem hiding this comment.
| """ | |
| Ensure the shared libraries in this dir and the ``sub`` subdir can be | |
| loaded on Windows. | |
| One shared library is installed alongside this ``__init__.py`` file. | |
| Windows can load it because it searches for DLLs in the directory where a | |
| ``.pyd`` (Python extension module) is located in. Cygwin does not though. | |
| For a shared library in another directory inside the package, Windows also | |
| needs a hint. | |
| This function is Windows-specific due to lack of RPATH support on Windows. | |
| It cannot find shared libraries installed within wheels unless we either | |
| amend the DLL search path or pre-load the DLL. | |
| Note that ``delvewheel`` inserts a similar snippet into the main | |
| ``__init__.py`` of a package when it vendors external shared libraries. | |
| .. note:: | |
| `os.add_dll_directory` is only available for Python >=3.8, and with | |
| the Conda ``python`` packages only works as advertised for >=3.10. | |
| If you require support for older versions, pre-loading the DLL | |
| with `ctypes.WinDLL` may be preferred (the SciPy code base has an | |
| example of this). | |
| """ | |
| """ | |
| Ensure the shared libraries in this package loaded on Windows. | |
| Windows lacks a concept equivalent to RPATH: Python extension modules cannot | |
| find DLLs installed outside the DLL search path. This function ensured that the | |
| location of the shared libraries distributed with this Python wheel is in the DLL | |
| search path of the process. | |
| The Windows DLL search path includes the object depending on it is located: | |
| the DLL search path needs to be augmented only when the Python extension | |
| modules and the DLLs they require are installed in separate directories. | |
| Cygwin does not have the same default library search path: all locations where | |
| the shared libraries are installed need to be added to the search path. | |
| This function is very similar to the snippet inserted into the main ``__init__.py`` | |
| of a package by ``delvewheel`` when it vendors external shared libraries. | |
| .. note:: | |
| `os.add_dll_directory` is only available for Python 3.8 and later, and in | |
| the Conda ``python`` packages it works as advertised only for version | |
| 3.10 and later. For older Python versions, pre-loading the DLLs with | |
| `ctypes.WinDLL` may be preferred. | |
| """ |
I find this a bit clearer. The line breaks are to be adjusted.
|
I've added a few suggestions in the places that I thought needed some little work. @rgommers please have a look. |
Opening this PR now to get feedback on topics and structure, not ready for detailed review yet.We get a lot of questions about shared libraries, and this is a tricky thing to get right. So try to document how to use internal libraries as well as link to external shared libraries as well as possible. Subproject-related questions also come up more and more, and there are some extra gotchas here, so treat those as a third "source" of shared (or static) libraries.