diff --git a/.hgignore b/.hgignore index b261c6e4..6ffebef5 100644 --- a/.hgignore +++ b/.hgignore @@ -55,7 +55,7 @@ image_samples/*/Images.bos*/*.h5 fluidimage/works/piv/*.h5 fluidimage/**.dot -doc/generated/* +doc/_generated/* doc/tutorials/*.rst doc/tutorials/*.h5 doc/tutorials/**.png diff --git a/CHANGES.md b/CHANGES.md index 851c5213..2271d1ee 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -3,39 +3,39 @@ See also the [unreleased changes](https://foss.heptapod.net/fluiddyn/fluidimage/-/compare/0.1.5...branch%2Fdefault). -# [0.1.5] (2023-02-15) +## [0.1.5] (2023-02-15) - Requires Python 3.9 - Improves legend, warnings, error log and documentation -# [0.1.4] (2022-12-13) +## [0.1.4] (2022-12-13) - Support Python 3.10 - Avoid a bug with pyfftw 0.13 -# [0.1.3] (2021-09-29) +## [0.1.3] (2021-09-29) - Many bugfixes! - Improve VectorFieldOnGrid and ArrayOfVectorFieldsOnGrid - UVmat compatibility - Fix incompatibility OpenCV and PyQt5 -# [0.1.2] (2019-06-05) +## [0.1.2] (2019-06-05) - Bugfix install Windows -# [0.1.1] (2019-05-23) +## [0.1.1] (2019-05-23) - Optical flow computation - Bugfixes + internal code improvements -# 0.1.0 (2018-10-03) +## 0.1.0 (2018-10-03) - New topologies and executors with Trio! - Much better coverage & many bugfixes! - Better surface tracking -# 0.0.3 (2018-08-29) +## 0.0.3 (2018-08-29) - Requirement Python >= 3.6 - Surface tracking @@ -46,7 +46,7 @@ See also the - fluidimslideshow-pg and fluidimviewer-pg (based on PyQtgraph) - OpenCV backend for preprocessing -# 0.0.2 (2017-04-13) +## 0.0.2 (2017-04-13) - Bug fixes and documentation changes. - Continuous integration (python 2.7 and 3.5) with bitbucket pipelines @@ -54,7 +54,7 @@ See also the - Preprocessing of images. - First simple GUI (`fluidimviewer` and `fluidimlauncher`). -# 0.0.1b (2016-05-31) +## 0.0.1b (2016-05-31) - Topology and waiting queues classes to run work in parallel. - PIV work and topology (multipass, different correlation methods). diff --git a/doc/CONTRIBUTING.md b/doc/CONTRIBUTING.md new file mode 120000 index 00000000..44fcc634 --- /dev/null +++ b/doc/CONTRIBUTING.md @@ -0,0 +1 @@ +../CONTRIBUTING.md \ No newline at end of file diff --git a/doc/Makefile b/doc/Makefile index 5353eda6..9ac2dc8e 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -46,7 +46,7 @@ clean: rm -rf $(BUILDDIR)/* cleanall: clean - rm -rf generated html.zip + rm -rf _generated html.zip cleancache: rm -rf _jupyter_cache diff --git a/doc/autosum.rst b/doc/autosum.rst new file mode 100644 index 00000000..a7c4914e --- /dev/null +++ b/doc/autosum.rst @@ -0,0 +1,5 @@ + +.. autosummary:: + :toctree: _generated/ + + fluidimage diff --git a/doc/build-from-source.md b/doc/build-from-source.md new file mode 100644 index 00000000..a95e4517 --- /dev/null +++ b/doc/build-from-source.md @@ -0,0 +1,303 @@ +# Build from source + +## Requirements + +To build Fluidimage from source, ones needs: + +- A modern Python (>=3.9) with Python headers and `pip` + +- A decent amount of RAM (at least few GB available). + +- A C++ compiler fully compliant with the C++-11 standard (currently not Intel + compilers) + +## Get the source + +Fluidimage development uses the revision control software [Mercurial] with modern +Mercurial extensions like [Evolve] and [Topic]. The main repository is hosted +[here](https://foss.heptapod.net/fluiddyn/fluidimage) in +[Heptapod](https://heptapod.net/). + +There are other ways to get the source but we are going here to assume that you +can install [Mercurial]. It can be useful when working with Fluidimage source to +[fully setup Mercurial with these extensions and learn a bit of Mercurial](http://fluiddyn.readthedocs.org/en/latest/mercurial_heptapod.html). +Then, the Fluidimage repository can be cloned with + +```sh + +hg clone https://foss.heptapod.net/fluiddyn/fluidimage + +``` + +```{admonition} Why Mercurial/Heptapod and not simply Git/Github? + +We consider that modern Mercurial is really great, even better in some aspects +than Git. Moreover, we do not think that it is a good thing that the whole +open-source ecosystem depends on Github, a close-source project owned by +Microsoft. + +Thanks to [Octobus](https://octobus.net/) and [Clever +Cloud](https://www.clever-cloud.com) for providing ! + +``` + +## Installing from the repository + +### Simple installation from source + +We recommend to create a clean virtual environment, for example with: + +```sh +cd fluidimage +python3 -m venv .venv +. .venv/bin/activate +pip install pip -U +``` + +Then, let us use `pip` to install the local project: + +```sh +pip install . -v +``` + +```{note} + +`-v` toggles `pip` verbose mode so that we see the compilation log and +can check that everything goes well. + +``` + +Moreover, the build (which uses [Meson]) can be controlled through environment +variables (for the C++ compilation) and options. The particular build options for +Fluidimage are defined in the file `meson.options` which contains: + +```{literalinclude} ../meson.options +``` + +To choose a value different from the default value, one can use this ugly syntax: + +```sh +pip install . -v --config-settings=setup-args=-Dtransonic-backend=python +# or +pip install . -v -C setup-args=-Dtransonic-backend=python +``` + +```{admonition} Let's decompose this syntax! + +There are 3 levels: + +- `--config-settings` / `-C` is a `pip` option to pass configuration to the PEP + 517 build backend (for Fluidimage [meson-python]). + +- `setup-args` is [understood by + meson-python](https://meson-python.readthedocs.io/en/latest/reference/config-settings.html) + +- `transonic-backend` is a Fluidimage build option. But one needs to add the `-D` + for [Meson]! + +``` + +````{important} + +One can activate a performance oriented and not portable build using: + +```sh +pip install . -v -C setup-args=-Dnative=true +``` + +```` + +````{note} + +Recent versions of `pip` allow one to specify different options like so: + +```sh +pip install . -v \ + -C setup-args=-Dtransonic-backend=python \ + -C setup-args=-Duse-xsimd=false +``` + +```` + +Of course, one can also change values of +[other buildin Meson options](https://meson-python.readthedocs.io/en/latest/reference/config-settings.html). + +````{warning} +(compile-args-j2)= + +[Meson] builds Fluidimage binaries in parallel. This speedups the build process a +lot on most computers. However, it can be a very bad idea on computers with not +enough memory. One can control the number of processes launched in parallel +with: + +```sh +pip install . -v -C compile-args=-j2 +``` + +```` + +````{Admonition} Another example to set the optimization level + +The default optimization level is `-O3`. One can change that with: + +```sh +pip install . -v -C setup-args=-Doptimization=2 +``` + +```` + +```{admonition} Advanced compilation configuration + +The environment variables `CC`, `CXX`, `CFLAGS`, `CXXFLAGS` and `LDFLAGS` +are honored. + +Note that Fluidimage builds are not sensible to the [`~/.pythranrc` file](pythranrc)! + +``` + +```{admonition} FAQ + +- How to know which compilers and compilation flags are used? + How to check if XSIMD was indeed used? + + One can study the file `build/cp39/compile_commands.json`. + +- How to differentiate a native build from a + regular build to produce binaries usable on other computers? + + By default the produced wheels should be portable. There is the `native` + build option to target the exact CPU used for compilation. + +- How to produce a wheel for other architectures (cross-compilation)? + + ??? + +``` + +### Setup a full developer environment with editable installation + +Let us first present the tools used for Fluidimage development. + +- [PDM] is a modern Python package and dependency manager, + +- [Meson] is an open source build system (in particular used by Scipy), + +- [Nox] is a command-line tool that automates testing in multiple Python + environments, + +- [Pytest] is the most popular testing framework for Python, + +- [pip] is the official package installer for Python, + +- [Pythran] is an ahead of time compiler for a subset of the Python language, with + a focus on scientific computing, + +- [Transonic] is a pure Python package to accelerate modern Python-Numpy code with + different accelerators (in particular Pythran). + +Fluidimage is built with [Meson]. We use [PDM] for Fluidimage development. [Pytest] +and [Nox] are used for testing. We use [Pythran] through [Transonic] to accelerate +some numerical kernels written in Python. + +#### Standard Python from Python.org + +We present here how one can build Fluidimage from source like the main developers +and users. + +##### Install PDM + +A first step is to install [PDM] as an external independant application. I (Pierre +Augier) usually use [pipx] for that but +[there are other methods](https://pdm-project.org/latest/#installation). + +```sh +python3 -m pip install pipx +pipx install pdm -U +``` + +Installing in editable mode is a bit particular with Meson, since editable +installations are incompatible with isolated builds, meaning that all build +dependencies have to be installed in the main virtual environment! Fortunatelly, +it's not too difficult with [PDM]. From the root directory of the repository, just +run: + +```sh +pdm install --no-self +``` + +This command creates a virtual environment and installs all build and runtime +dependencies. You can then activate this environment and build/install Fluidimage +with: + +```sh +. .venv/bin/activate +pip install -e . -v --no-build-isolation --no-deps +``` + +### Conda-based Python with conda-forge and Pixi + +One can use [Pixi] to setup a developer environment based on [conda-forge] and +compile from source. From the root directory of Fluidimage repository, just run: + +```sh +pixi run install-editable +``` + +Then, `pip` is available and previous commands should work. + +## Advice for developers + +### Run the tests + +You can run some unit tests by running `make tests` (shortcut for +`fluidsim-test -v`) or `make tests_mpi` (shortcut for +`mpirun -np 2 fluidsim-test -v`). Alternatively, you can also run `pytest` from +the root directory or from any of the source directories. + +(pythranrc)= + +### About using Pythran to compile functions + +When developing with Pythran, it can be useful to have a `~/.pythranrc` file, with +for example something like (see +[the dedicated section in Pythran documentation](https://pythran.readthedocs.io/en/latest/MANUAL.html#customizing-your-pythranrc)): + +```sh + +[pythran] +complex_hook = True + +[compiler] +CXX=clang++ +CC=clang + +``` + +Note however, that Fluidimage build does not take into account this file! Instead +there is a build option `pythran-complex-hook` and one can use environment +variables to change the C++ compilation (performed with [Meson]). + +### Set the MESONPY_EDITABLE_VERBOSE mode + +It can be useful to set this environment variable when using the editable mode. + +```sh +export MESONPY_EDITABLE_VERBOSE=1 +``` + +See +https://meson-python.readthedocs.io/en/latest/how-to-guides/editable-installs.html#verbose-mode + +[conda-forge]: https://conda-forge.org/ +[evolve]: https://www.mercurial-scm.org/doc/evolution/ +[mercurial]: https://www.mercurial-scm.org/ +[meson]: https://mesonbuild.com +[nox]: https://nox.thea.codes +[pdm]: https://pdm-project.org +[pip]: https://pip.pypa.io +[pipx]: https://github.com/pypa/pipx +[pixi]: https://pixi.sh/ +[pytest]: https://docs.pytest.org +[pythran]: https://pythran.readthedocs.io +[topic]: https://www.mercurial-scm.org/doc/evolution/tutorials/topic-tutorial.html +[transonic]: https://transonic.readthedocs.io diff --git a/doc/conf.py b/doc/conf.py index 5959d168..a2d228f8 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -224,7 +224,7 @@ # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ - ("index", "fluidfft.tex", "fluidfft Documentation", "Pierre Augier", "manual") + ("index", "fluidimage.tex", "Fluidimage Documentation", "Pierre Augier", "manual") ] # The name of an image file (relative to this directory) to place at the top of diff --git a/doc/for_dev.md b/doc/for_dev.md index 0e08af7b..ee5e1219 100644 --- a/doc/for_dev.md +++ b/doc/for_dev.md @@ -1,10 +1,13 @@ -# Information for developers +# Contribute to Fluidimage ```{toctree} --- maxdepth: 1 --- +CONTRIBUTING.md Advice for FluidDyn developers -related_codes overview_orga_package +build-from-source +related_codes +to_do ``` diff --git a/doc/index.md b/doc/index.md index 2044c739..4052d277 100644 --- a/doc/index.md +++ b/doc/index.md @@ -1,49 +1,38 @@ % FluidImage documentation master file -# User Guide +# Fluidimage documentation + +FluidImage is a libre Python framework for scientific processing of +large series of images. ```{toctree} --- +caption: User guide maxdepth: 1 --- +overview install tutorial examples +build-from-source ``` -# Modules Reference - -Here is presented the general organization of the package (see also -{doc}`overview_orga_package`) and the documentation of the modules, classes and -functions. - -```{eval-rst} -.. autosummary:: - :toctree: generated/ - - fluidimage.topologies - fluidimage.executors - fluidimage.data_objects - fluidimage.works - fluidimage.calcul - fluidimage.preproc - fluidimage.calibration - fluidimage.reconstruct - fluidimage.postproc - fluidimage.util - fluidimage.gui - +```{toctree} +--- +caption: Python API +maxdepth: 1 +--- +overview_orga_package +autosum.rst ``` -# More - ```{toctree} --- +caption: Fluidimage development maxdepth: 1 --- FluidImage forge on Heptapod -FluidImage in PyPI -to_do +FluidImage on PyPI changes authors for_dev diff --git a/doc/install.md b/doc/install.md index 15e6fb61..3ed33203 100644 --- a/doc/install.md +++ b/doc/install.md @@ -1,4 +1,4 @@ -# Installation and advice +# Installation See our general advice [on using Python](https://fluiddyn.readthedocs.io/en/latest/advice_on_Python.html) and diff --git a/doc/overview.md b/doc/overview.md new file mode 100644 index 00000000..5c7a0797 --- /dev/null +++ b/doc/overview.md @@ -0,0 +1,50 @@ +# Overview + +FluidImage is a libre Python framework for scientific processing of +large series of images. + +**Documentation:** + +Image processing for fluid mechanics is highly dominated by proprietary +tools. Such tools are not ideal when you want to understand and tweak +the processes and/or to use clusters. With the improvement of the +open-source tools for scientific computing and collaborative +development, one can think it is possible to build together a good +library/toolkit specialized in image processing for fluid mechanics. +This is our project with FluidImage. + +This package is young but already good enough to be used "in +production" to + +- display and pre-process images, + +- compute displacement or velocity fields with [Particle Image + Velocimetry](https://en.wikipedia.org/wiki/Particle_image_velocimetry%20(PIV)) + (PIV, i.e. displacements of pattern obtained by correlations of + cropped images) and [optical + flow](https://en.wikipedia.org/wiki/Optical_flow), + +- analyze and display PIV fields. + +We want to make FluidImage easy (useful documentation, easy +installation, usable with scripts and GUI in Qt), reliable (with good +[unittests](https://codecov.io/gh/fluiddyn/fluidimage/)) and very +efficient, in particular when the number of images to process becomes +large. Thus we want FluidImage to be able to run efficiently and easily +on a personal computer and on big clusters. The efficiency is achieved +by using + +- a framework for asynchronous computations (currently, we use + [Trio](https://trio.readthedocs.io) + multiprocessing, and in the + long term we want to be able to plug FluidImage to distributed + computational systems like [Dask](http://dask.pydata.org), + [Spark](https://spark.apache.org/) or + [Storm](http://storm.apache.org/)), + +- the available cores of the central processing units (CPU) and the + available graphics processing units (GPU), + +- good profiling and efficient and specialized algorithms, + +- cutting-edge tools for fast computations with Python (in particular + [Pythran](https://pythonhosted.org/pythran/)). diff --git a/doc/overview_orga_package.md b/doc/overview_orga_package.md index 1cb54f9e..13f74a58 100644 --- a/doc/overview_orga_package.md +++ b/doc/overview_orga_package.md @@ -1,4 +1,4 @@ -# Overview of the organization of the package +# Overview of the package ## Data objects ({mod}`fluidimage.data_objects`) diff --git a/src/fluidimage/__init__.py b/src/fluidimage/__init__.py index d8274fbd..8f4c529f 100644 --- a/src/fluidimage/__init__.py +++ b/src/fluidimage/__init__.py @@ -2,6 +2,21 @@ FluidImage ========== +.. autosummary:: + :toctree: + + fluidimage.topologies + fluidimage.executors + fluidimage.data_objects + fluidimage.works + fluidimage.calcul + fluidimage.preproc + fluidimage.calibration + fluidimage.reconstruct + fluidimage.postproc + fluidimage.util + fluidimage.gui + """ import sys