How to build on Mac with Conda Python 3.8

[jtchilders]

It took me a while to figure this out so I just wanted to place this here for documentation:

If you make a miniconda installation on your mac, in my case with Python 3.8.3 and all these modules:

-> conda list
# packages in environment at CONDA_PREFIX:
#
# Name                    Version                   Build  Channel
absl-py                   1.1.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl
bottleneck                1.3.5            py38h67323c0_0
brotli                    1.0.9                hca72f7f_7
brotli-bin                1.0.9                hca72f7f_7
brotlipy                  0.7.0           py38h9ed2024_1003
bzip2                     1.0.8                h1de35cc_0
c-ares                    1.18.1               hca72f7f_0
ca-certificates           2022.07.19           hecd8cb5_0
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.6.15        py38hecd8cb5_0
cffi                      1.15.0           py38hc55c11b_1
charset-normalizer        2.0.4              pyhd3eb1b0_0
cmake                     3.22.1               hbfa4a85_0
conda                     4.13.0           py38hecd8cb5_0
conda-package-handling    1.8.1            py38hca72f7f_0
cryptography              37.0.1           py38hf6deb26_0
cycler                    0.11.0             pyhd3eb1b0_0
eigen                     3.3.7                haf03e11_1
expat                     2.4.4                he9d5cce_0
flatbuffers               1.12                     pypi_0    pypi
fonttools                 4.33.3                   pypi_0    pypi
freetype                  2.11.0               hd8bbffd_0
future                    0.18.2                   py38_1
gast                      0.4.0                    pypi_0    pypi
giflib                    5.2.1                haf1e3a3_0
google-auth               2.8.0                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.46.3                   pypi_0    pypi
h5py                      3.7.0                    pypi_0    pypi
icu                       58.2                 h0a44026_3
idna                      3.3                pyhd3eb1b0_0
importlib-metadata        4.11.4                   pypi_0    pypi
intel-openmp              2021.4.0          hecd8cb5_3538
jpeg                      9e                   hca72f7f_0
keras                     2.9.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.3                    pypi_0    pypi
krb5                      1.19.2               hcd88c3b_0
lcms2                     2.12                 hf1fd2bf_0
libbrotlicommon           1.0.9                hca72f7f_7
libbrotlidec              1.0.9                hca72f7f_7
libbrotlienc              1.0.9                hca72f7f_7
libclang                  14.0.1                   pypi_0    pypi
libcurl                   7.84.0               h6dfd666_0
libcxx                    12.0.0               h2f01273_0
libedit                   3.1.20210910         hca72f7f_0
libev                     4.33                 h9ed2024_1
libffi                    3.3                  hb1e8313_2
libnghttp2                1.46.0               ha29bfda_0
libpng                    1.6.37               ha441bb4_0
libssh2                   1.10.0               h0a4fc7d_0
libtiff                   4.2.0                hdb42f99_1
libuv                     1.40.0               haf1e3a3_0
libwebp                   1.2.2                h56c3ce4_0
libwebp-base              1.2.2                hca72f7f_0
llvm-openmp               8.0.1                h770b8ee_0    conda-forge
lz4-c                     1.9.3                h23ab428_1
markdown                  3.3.7                    pypi_0    pypi
matplotlib                3.5.2                    pypi_0    pypi
matplotlib-base           3.5.1            py38hfb0c5b7_1
mkl                       2021.4.0           hecd8cb5_637
mkl-service               2.4.0            py38h9ed2024_0
mkl_fft                   1.3.1            py38h4ab4a9b_0
mkl_random                1.2.2            py38hb2f4e1b_0
munkres                   1.1.4                      py_0
ncurses                   6.3                  hca72f7f_2
ninja                     1.10.2               hecd8cb5_5
ninja-base                1.10.2               haf03e11_5
numexpr                   2.8.3            py38h2e5f0a9_0
numpy                     1.22.3           py38h2e5f0a9_0
numpy-base                1.22.3           py38h3b1a694_0
oauthlib                  3.2.0                    pypi_0    pypi
openmp                    8.0.1                         0    conda-forge
openssl                   1.1.1q               hca72f7f_0
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0
pandas                    1.4.3            py38he9d5cce_0
pillow                    9.1.1                    pypi_0    pypi
pip                       22.2.2                   pypi_0    pypi
pptk                      0.1.1                    pypi_0    pypi
protobuf                  3.19.4                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycosat                   0.6.3            py38h1de35cc_1
pycparser                 2.21               pyhd3eb1b0_0
pyopenssl                 22.0.0             pyhd3eb1b0_0
pyparsing                 3.0.9                    pypi_0    pypi
pysocks                   1.7.1                    py38_1
python                    3.8.3                h26836e1_1
python-dateutil           2.8.2              pyhd3eb1b0_0
python.app                2                       py38_10
python_abi                3.8                      2_cp38    conda-forge
pytorch                   1.10.2          cpu_py38h903acac_0
pytz                      2022.1           py38hecd8cb5_0
qt                        5.9.7                h468cd18_1
readline                  8.1.2                hca72f7f_1
requests                  2.27.1             pyhd3eb1b0_0
requests-oauthlib         1.3.1                    pypi_0    pypi
rhash                     1.4.1                hbcfaee0_1
rsa                       4.8                      pypi_0    pypi
ruamel_yaml               0.15.100         py38h9ed2024_0
scipy                     1.8.1                    pypi_0    pypi
setuptools                61.2.0           py38hecd8cb5_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.38.3               h707629a_0
tbb                       2020.3               h879752b_0
tbb-devel                 2020.3               h879752b_0
tensorboard               2.9.1                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.9.1                    pypi_0    pypi
tensorflow-estimator      2.9.0                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.26.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tk                        8.6.12               h5d9f67b_0
tornado                   6.1              py38h9ed2024_0
tqdm                      4.64.0           py38hecd8cb5_0
typing-extensions         4.1.1                hd3eb1b0_0
typing_extensions         4.1.1              pyh06a4308_0
urllib3                   1.26.9           py38hecd8cb5_0
werkzeug                  2.1.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.5                hca72f7f_1
yaml                      0.2.5                haf1e3a3_0
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.12               h4dc903c_2
zstd                      1.5.2                hcb37349_0

some of these packages aren't necessary for pptk (tensorflow for example). Some key packages were

conda install -c conda-forge openmp tbb-devel==2020.3 tbb==2020.3 qt eigen

Then you can clone the repo:

git clone https://github.com/heremaps/pptk.git
cd pptk
mkdir build
cd build
export CONDA_INC_PATH=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('INCLUDEDIR'))")
export CONDA_LIB_PATH=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
export PYTHON_SITE=$(python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
cmake .. -DPYTHON_INCLUDE_DIR=$(python -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())")  -DPYTHON_LIBRARY=$CONDA_LIB_PATH -DNumpy_INCLUDE_DIR:PATH=$PYTHON_SITE/numpy/core/include/numpy -DTBB_INCLUDE_DIR:PATH=$CONDA_INC_PATH -DTBB_tbb_LIBRARY:FILEPATH=$CONDA_LIB_PATH/libtbb.dylib -DTBB_tbb_RUNTIME:FILEPATH=$CONDA_LIB_PATH/libtbb.dylib -DTBB_tbbmalloc_LIBRARY:FILEPATH=$CONDA_LIB_PATH/libtbbmalloc.dylib -DTBB_tbbmalloc_RUNTIME:FILEPATH=$CONDA_LIB_PATH/libtbbmalloc.dylib -DEigen_INCLUDE_DIR=$CONDA_INC_PATH/eigen3
make -j

Now you need to build the wheel and things have changed so you need to edit the setup.py in the build directory so that the top of the file looks like this:

from setuptools import setup, find_packages
from distutils.extension import Extension
import subprocess
import os
import os.path
import shutil
import platform

# from pip._internal import wheel
import packaging.tags
# get the first (most detailed) tag describing this system
# analogous to wheel.pep425tags.get_supported()[0]
sys_tag = packaging.tags.sys_tags().__next__()
# tag names are now referenced by name
opts = {'python_tag': sys_tag.interpreter, 'plat_name': sys_tag.platform}

#wheel_tags = wheel.pep425tags.get_supported()[0]
wheel_tags = [ sys_tag.interpreter, '', sys_tag.platform]

system_type = platform.system()

Then you can build and install the wheel:

python setup.py bdist_wheel
pip install dist/*.whl

Done.

[amiltonwong]

@jtchilders,
I got the following error:

Scanning dependencies of target setup.py
[  3%] Copying /data/code13/pptk/setup.py to /data/code13/pptk/build
[  3%] Built target setup.py
Scanning dependencies of target LICENSE
[  6%] Copying /data/code13/pptk/LICENSE to /data/code13/pptk/build
[  6%] Built target LICENSE
Scanning dependencies of target pptk.__init__.py
[ 10%] Copying /data/code13/pptk/pptk/__init__.py to /data/code13/pptk/build/pptk
[ 10%] Built target pptk.__init__.py
Scanning dependencies of target pptk._add_path.py
[ 13%] Copying /data/code13/pptk/pptk/_add_path.py to /data/code13/pptk/build/pptk
[ 13%] Built target pptk._add_path.py
Scanning dependencies of target kdtree
[ 17%] Building CXX object pptk/kdtree/CMakeFiles/kdtree.dir/kdtree_wrapper.cpp.o
In file included from /root/anaconda3/envs/mvtorchenv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940:0,
                 from /root/anaconda3/envs/mvtorchenv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /root/anaconda3/envs/mvtorchenv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,
                 from /data/code13/pptk/pptk/kdtree/../include/python_util.h:7,
                 from /data/code13/pptk/pptk/kdtree/kdtree_wrapper.cpp:4:
/root/anaconda3/envs/mvtorchenv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
In file included from /data/code13/pptk/pptk/kdtree/src/kdtree-impl.h:18:0,
                 from /data/code13/pptk/pptk/kdtree/src/kdtree.h:254,
                 from /data/code13/pptk/pptk/kdtree/kdtree_wrapper.cpp:3:
/root/anaconda3/envs/mvtorchenv/include/tbb/task.h:21:139: note: #pragma message: TBB Warning: tbb/task.h is deprecated. For details, please see Deprecated Features appendix in the TBB reference manual.
 #pragma message("TBB Warning: tbb/task.h is deprecated. For details, please see Deprecated Features appendix in the TBB reference manual.")
                                                                                                                                           ^
In file included from /data/code13/pptk/pptk/kdtree/src/kdtree-impl.h:19:0,
                 from /data/code13/pptk/pptk/kdtree/src/kdtree.h:254,
                 from /data/code13/pptk/pptk/kdtree/kdtree_wrapper.cpp:3:
/root/anaconda3/envs/mvtorchenv/include/tbb/task_scheduler_init.h:21:154: note: #pragma message: TBB Warning: tbb/task_scheduler_init.h is deprecated. For details, please see Deprecated Features appendix in the TBB reference manual.
 essage("TBB Warning: tbb/task_scheduler_init.h is deprecated. For details, please see Deprecated Features appendix in the TBB reference manual.")
                                                                                                                                                 ^
make[2]: *** No rule to make target '/root/anaconda3/envs/mvtorchenv/lib/libtbb.dylib', needed by 'pptk/kdtree/kdtree.so'.  Stop.
CMakeFiles/Makefile2:272: recipe for target 'pptk/kdtree/CMakeFiles/kdtree.dir/all' failed
make[1]: *** [pptk/kdtree/CMakeFiles/kdtree.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Any hints to solve this issue?

Thanks~