Package a swi-prolog binary with the binary wheels

[rmanhaeve]

Hi

Thank you for your work on Janus and SWI-Prolog in general.

I was wondering if you had considered packaging a (minimal) binary of SWI-Prolog along with the binary distributions of this package. This would greatly simplify the installation procedure (especially for other projects that have janus_swi as a dependency), and I would imagine this could also boost the popularity of SWI-Prolog as an easy to install package along all platforms.

Kind regards,
Robin Manhaeve

[JanWielemaker]

Thanks for the heads up and suggestion. Considered? Sort of 😄 There are many possible installation targets though and it is hard to me to judge the value and costs of all the alternatives. I've spent some time turning SWI-Prolog into an (ana)conda package. It sort of works with some limitations and getting it all setup and distributed seems a lot of work.

Binary wheels also come with their issues on the various platforms. I have very limited understanding of the Python binary package management. If people want to set this up, I'm happy to help from the Prolog side.

I heard some rumours that the wheel could also download and install the regular binary? Do you have any insight in that?

[rmanhaeve]

I'm not an expert myself, but I have a bit of experience in compiling external dependencies for binary wheels. We can make use of cibuildwheel and the Github-hosted runners to build the binary for basically all common platform.

Is there a way to build SWI Prolog with minimal dependencies?

[JanWielemaker]

Is there a way to build SWI Prolog with minimal dependencies?

Configure using cmake -DSWIPL_PACKAGES=OFF -DUSE_GM=OFF ... Then the only dependency is zlib.

I'm still a bit in doubt on how good an idea this is as it results in a quite crippled SWI-Prolog version. Possibly a better choice is to select all packages that do not pull in new dependencies. That can be done using cmake -DSWIPL_PACKAGE_LIST="clib;plunit;sgml;semweb;chr;clpqr;nlp;yaml;swipy" -DUSE_GM=OFF ... With a bit of tweaking we can probably get in some more (might even have forgotton some).

If you can get this working (even without any packages), I'm happy to jump in and try to add some more functionality.

[rmanhaeve]

I'll try to get a minimal case working in a fork, I'll update here when I've made some progress.

[rmanhaeve]

Which files from the build directory need to be included?

[JanWielemaker]

Which files from the build directory need to be included?

You need to run make install (or ninja install). Set -DCMAKE_INSTALL_PREFIX=dir to control the base location. I'd assume there are more examples around of creating wheels from cmake configured applications.

[JanWielemaker]

This issue has been mentioned on SWI-Prolog. There might be relevant details there:

https://swi-prolog.discourse.group/t/installing-janus-swi-on-windows/8629/10

[JanWielemaker]

Hi @rmanhaeve , Any progress on this? It may help automating the build for janus-swi. Possibly we can do a shared session? With some knowledge on both ends it is probably done quickly 😄 Please contact me on mail or using a private message on Discourse if you like this.

[rmanhaeve]

Hi @JanWielemaker.

I was playing around with it a bit, and got the building working. However, there were some linking issues in the end.
I think I now have a better idea of how to go around it, but I haven't had the time yet to try.

I'll make some time soon, and I'll be in contact when I have an update.

Kind regards,
Robin

[rmanhaeve]

I'm a bit stuck at the moment. The issue I'm running into is that the manylinux build environment includes libraries for building modules (Development.Module) but not for embedding (Development.Embed).
I'm thus able to build the Python package, but it cannot build the Janus SWI-Prolog package.

I'm not sure why this is exactly yet (probably for a good reason), and whether this can be fixed. I'll look deeper into the issue.

[JanWielemaker]

Thanks. If I can access your work I'm happy to have a look. Not that I'm a pip or Python expert ... I learned a little writing Janus though 😄

[rmanhaeve]

I've forked it here https://github.com/rmanhaeve/packages-swipy

I've also asked for help here: pypa/manylinux#1724

[rmanhaeve]

Also, to make sure I understand properly, it is impossible to use the Python package with library(janus) being available in SWI-Prolog?

[JanWielemaker]

The issue I'm running into is that the manylinux build environment includes libraries for building modules (Development.Module) but not for embedding (Development.Embed).
I'm thus able to build the Python package, but it cannot build the Janus SWI-Prolog package.

Ah, now I see what you mean. I don't think that is a problem. Yes, you cannot start the bundled SWI-Prolog as an executable and load Janus into it. You can only start Python and use Prolog through janus_swi. Once you started from Python and call Prolog though, Prolog can use Janus to make callbacks to Python.

So, the idea would be that there is package e.g. janus-swi-bundled that provides you with an embedded SWI-Prolog on any system without trouble. At the same time, there is the SWI-Prolog binary package (or source package) for most systems that allows you to call use Janus to call Python from the Prolog executable. And finally, one can install SWI-Prolog and Python and install janus-swi to provide Prolog embedding into Python with full Prolog. Note that Fedora already bundles janus-swi (experimentally by the maintainer), so you have the whole thing at practically zero effort. I expect Debian/Ubuntu/... to follow that.

What I do miss is Windows and MacOS. Your work should work with minor adoption on MacOS, but doing this on Windows is a different story. The simplest, but possibly bit dubious option would be to install the SWI-Prolog binary for Windows inside the Python package. Note that the installed system is fully relocatable. Building a performant SWI-Prolog on Windows is non-trivial. It compiles using MSVC, but runs at just about half the performance 😢

[JanWielemaker]

Also, to make sure I understand properly, it is impossible to use the Python package with library(janus) being available in SWI-Prolog?

Not sure I understand this. There are two versions of the Janus binary that may be created from the same source: one that embeds Python in Prolog. That is bundled with the binary distributions for SWI-Prolog and if you build SWI-Prolog from source and make sure the Python embedding libraries are installed on your system it will build this Janus version for you when you build SWI-Prolog.

The other is what is on PyPi: janus-swi. That is what you get if you run pip on packages-swipy. It requires SWI-Prolog to be installed. On success, you can embed SWI-Prolog into Python.

These two options only differ in what the "main" program is, i.e., who embeds who. Once loaded, they are symmetrical. E.g., you can call ?- py_shell. to get a Python REPL from Python embedded into Prolog or >>> janus.prolog() to get a Prolog REPL from Python. You can recursively do so until you run out of stack 😄

Does that answer your question?

[rmanhaeve]

I think you've answered my questions. So it should be possible, but I'm still bumping into errors.

When I try to import janus into Python, I get the following errors:

ERROR: source_sink `library(janus)' does not exist
 Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "/tmp/tmp.pYziWM2SFk/venv/lib/python3.6/site-packages/janus_swi/__init__.py", line 24, in <module>
     "--no-signals")
 janus_swi.janus.PrologError: Failed to load library(janus) into Prolog

Could it be the interplay between these lines? We are only adding the current directory to the search path after initialization, but the initialization is already looking for it?

_swipl.initialize("swipl",
                  "-g", "true",
                  "--no-signals")

_swipl.call("(exists_source(library(janus))->true;asserta(user:file_search_path(library, Here)))",
            {"Here":os.path.dirname(__file__)})

[JanWielemaker]

I don't think the initialization should be looking for it. Can you add a print after the _swipl.initialize() call to verify that? You can also add a call to writeln/1 after the asserta/1 call to see whether that passes the correct location.

[rmanhaeve]

I believe that the error is coming from this line:

packages-swipy/janus/mod_swipl.c

Line 696 in cd6377a

{ Py_SetPrologErrorFromChars("Failed to load library(janus) into Prolog");

Here is the output of trying to import janus_swi( I believe some of these are out of order). If you want to see the prints I put in, the code is here https://github.com/rmanhaeve/packages-swipy/blob/master/janus/__init__.py

ERROR: source_sink `library(janus)' does not exist
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/tmp.Y8de7fddep/venv/lib/python3.6/site-packages/janus_swi/__init__.py", line 25, in <module>
    "--no-signals")
janus_swi.janus.PrologError: Failed to load library(janus) into Prolog
Loading janus
from janus_swi.janus import *
import janus_swi._swipl
Initializing swipl

[JanWielemaker]

I see. Yes, that looks fishy. I need to think a little about that.

[JanWielemaker]

Ok. Changed __init__.py to stop loading the library, but instead pass -p library=dir to make loading the library from C work. I first considered loading the Prolog library from Python, but that does not work as the library needs to be loaded before we can call Prolog from Python 😢

Bottom line is that the janus_swi package can now work without Prolog having the janus package installed, as it should.

[rmanhaeve]

Excellent, this indeed fixed the problem I was having. I am now able to build wheels, and import them. I will do some more extensive testing later on, but this is looking promising.

[rmanhaeve]

I guess I can just run the scripts in tests. Although it's not a real test suite, if these run without error, that's fine for now I guess

[JanWielemaker]

If you pull the latest version, you should be able to run the tests using the calls below. The patch is required to be able to access Janus as janus from Prolog when Prolog is embedded into Python. It is not elegant. If you know something better ...

import janus_swi as janus
janus.query_once("use_module(test_janus)")
janus.query_once("test_janus")

That runs the tests from the Prolog perspective, but this also runs the Python calling Prolog tests, initiated through Prolog.

[rmanhaeve]

Good news, I got it working for manylinux and musllinux, tested it on a clean Ubuntu and Alpine Linux. The only additional change I needed was setting the SWI_HOME_DIR to the bundled directory.

Now, I'll look into:

1. Getting it working on MacOS and Windows
2. Getting it working on arm
3. Minimizing the bundled files.

For point 3, I could still use some help: What are the minimal files needed to get the python package working? E.g., I assume we can get rid of the executable if we just set SWI_HOME_DIR?

[JanWielemaker]

E.g., I assume we can get rid of the executable if we just set SWI_HOME_DIR?

Yes, you can dump the executable, but it is really small. Otherwise it mainly depends on packages you enable. Some are pretty big (e.g., CHR). You could consider building using -DUSE_GMP=OFF if libgmp.so is included in the bundle. Switching it off uses the bundled LibBF for large numbers. That is a lot smaller (but also slower, notably on big rational numbers). I guess starting baobab is the easiest way to find where the space is.

On Windows, you may need swipl.exe or you need to replace some of the code that finds the Prolog properties.

[rmanhaeve]

I'm making some more progress towards making the bundled swipl more portable. One issue I'm having is that libswipl has the 3 version of the SOVERSION mechanism. These are symlinked together, but in wheels, symlinks are not allowed, so the eventual wheel contains 3 full copies. To solve this, I'm either thinking of conditionally leaving out the version properties in CMake, or just deleting the more specific versions.

For Windows, I was also thinking of linking statically against zlib and pthreads4w. For zlib, this is fine.
pthreads4w changed to Apache in version 3, so this should now be allowed (given that we include the correct copyright statement). Do you have an opinion on thos?

[JanWielemaker]

Just installing libswipl.so is probably fine. I'm happy with a cmake option, e.g., SWIPL_SO_VERSIONS=OFF. Can you do that and make a PR?

If not too complicated, I'm ok with static linking. The license issue can simply be handled by calling PL_license("asl2", "pthreads4w"), which probably should to be called anyway. Note that this is Apache V2, but on the apache page I find no mention on a version 3.

On the other hand, dynamic linking generally works very easily on Windows ... Oh, and I think some of the dlls from the packages also link against pthreads4w. I would not even be surprised if dynamic linking is required as the dll hooks then allow the library to keep track of created and destroyed threads.

[rmanhaeve]

I should've clarified: pthreads4w version 3 and onwards is under Apache 2.

On my fork, I now manage to build packages for Python 3.8 to Python 3.13, on manylinux and musllinux, currently only for x64. I managed to get the size of each wheel down to 10MB, which I think is decent. There are also no more redundant copies of libswipl.

The only thing that is technically not ok (as far as I know for now) is that there is a dependency on an external libz, which is not allowed according to the manylinux / musllinux specification. Practically, this not an issue, as most Linux distro's have it installed anyway.

There are still a few changes that I want to make, the most important being that if it detects a bundled swipl, it just pulls the relevant information from that directly. Currently, it is still somewhat hardcoded.

[JanWielemaker]

there is a dependency on an external libz

Statically linking libz into libswipl.so should be ok (provided it is compiled with -fpic).