Combining | and xp.ndarray in a type causes warning in sphinx API docs build

[yousefmoazzam]

As part of the warnings seen in #468, with a bit of trial and error I have narrowed down the cause of the "unsupported operand |" errors being produced by sphinx, despite running with a python 3.12 interpreter (which should recognise the | operator as another way to do Union).

It seems like mixing | and types that come from the xp variable causes the issue. If xp is not used, then the | poses no problem for sphinx.

I found this by being able reproduce the warning sphinx generates, but in a dummy module I created locally in my httomo repo and generating API docs for that module only.

The dummy module (which I lovingly called bloop.py) is the following:

from typing import TypeAlias, Union

import numpy as np

try:
    import cupy as xp

    try:
        xp.cuda.Device(0).compute_capability
        gpu_enabled = True  # CuPy is installed and GPU is available
    except xp.cuda.runtime.CUDARuntimeError:
        import numpy as xp

except ImportError:
    import numpy as xp


Blah: TypeAlias = Union[np.ndarray, xp.ndarray]
Thing: TypeAlias = int | float

where you can see that I have defined xp as it is defined in utils.py:

httomo/httomo/utils.py

Lines 11 to 21 in f619ed1

	try:
	import cupy as xp
	try:
	xp.cuda.Device(0).compute_capability
	gpu_enabled = True # CuPy is installed and GPU is available
	except xp.cuda.runtime.CUDARuntimeError:
	import numpy as xp
	except ImportError:
	import numpy as xp

Having Blah defined as:

Blah: TypeAlias = Union[np.ndarray, xp.ndarray]

causes no issues, but having it defined as:

Blah: TypeAlias = np.ndarray | xp.ndarray

produces the warning:

(/dls/science/users/twi18192/conda-envs/docs-httomo-py310-updated) [twi18192@pc0074 httomo (sphinx-fixes)]$ source docs/sphinx-build.sh
***********************************************************************
                Starting the sphinx script
***********************************************************************
            Creating plugin API files and html pages ..
Running Sphinx v8.1.3
loading translations [en]... done
making output directory... done
Blah is int | float
[autosummary] generating autosummary for: autosummary_api.rst, backends/list.rst, backends/templates.rst, developers/how_to_contribute.rst, developers/memory_calculation.rst, explanation/faq.rst, explanation/process_list.rst, explanation/templates.rst, howto/how_to_run/at_diamond.rst, howto/how_to_run/outside_diamond.rst, ..., introduction/indepth/chunks.rst, introduction/indepth/detailed_about.rst, introduction/indepth/memory_estimators.rst, introduction/indepth/reslice.rst, introduction/indepth/sections.rst, introduction/indepth/wrappers.rst, pipelines/yaml.rst, reference/loaders.rst, reference/yaml.rst, utilities/yaml_checker.rst
WARNING: [autosummary] failed to import httomo.bloop.
Possible hints:
* TypeError: unsupported operand type(s) for |: 'type' and 'ndarray'
* ImportError:
* AttributeError: module 'httomo' has no attribute 'bloop'
Writing evaluated template result to /dls/science/users/twi18192/httomo/docs/build/_static/nbsphinx-code-cells.css
building [mo]: all of 0 po files
writing output...
building [html]: all source files
updating environment: [new config] 36 added, 0 changed, 0 removed
reading sources... [100%] utilities/yaml_checker
/dls/science/users/twi18192/httomo/docs/source/autosummary_api.rst:4: WARNING: autosummary: failed to import httomo.bloop.
Possible hints:
* TypeError: unsupported operand type(s) for |: 'type' and 'ndarray'
* ImportError:
* AttributeError: module 'httomo' has no attribute 'bloop'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets...
copying static files...
Writing evaluated template result to /dls/science/users/twi18192/httomo/docs/build/_static/basic.css
Writing evaluated template result to /dls/science/users/twi18192/httomo/docs/build/_static/documentation_options.js
Writing evaluated template result to /dls/science/users/twi18192/httomo/docs/build/_static/language_data.js
Writing evaluated template result to /dls/science/users/twi18192/httomo/docs/build/_static/copybutton.js
copying static files: done
copying extra files...
copying extra files: done
copying assets: done
writing output... [100%] utilities/yaml_checker
generating indices... genindex done
copying linked files...
copying notebooks ...
highlighting module code...
writing additional pages... search done
copying images... [100%] _static/wrappers/wrappers.png
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 2 warnings.

The HTML pages are in docs/build.
***********************************************************************
                          End of script
***********************************************************************

Furthermore, note that Thing uses | in its definition and that it does not produce the warning in any case.

Therefore, I have concluded for now that "something" about using xp with | isn't quite right, but what the "something" is I don't know.

I don't fancy trying to dig into sphinx internals or python issues regarding | and variables whose values are modules, so I think a reasonable quick solution to this would be to use Union if a union type involving xp is needed, and to use | in any other union type.

[yousefmoazzam]

Similar behaviour involving the | operator in union types seems to be exhibited by the mock objects used when manually mocking imports (see #490):

WARNING: [autosummary] failed to import httomo.data.dataset_store.
Possible hints:
* AttributeError: module 'httomo.data' has no attribute 'dataset_store'
* ImportError:
* TypeError: unsupported operand type(s) for |: 'Mock' and 'CustomMock'
WARNING: [autosummary] failed to import httomo.data.padding.
Possible hints:
* TypeError: unsupported operand type(s) for |: 'Mock' and 'CustomMock'
* AttributeError: module 'httomo.data' has no attribute 'padding'
* ImportError:
WARNING: [autosummary] failed to import httomo.runner.dataset_store_backing.
Possible hints:
* ModuleNotFoundError: No module named 'numpy.typing'; 'numpy' is not a package
* ImportError:
* AttributeError: module 'httomo.runner' has no attribute 'dataset_store_backing'
WARNING: [autosummary] failed to import httomo.runner.task_runner.
Possible hints:
* TypeError: unsupported operand type(s) for |: 'Mock' and 'CustomMock'
* ImportError:
* AttributeError: module 'httomo.runner' has no attribute 'task_runner'

[yousefmoazzam]

Yet another place where similar behaviour is exhibited is anytime | is used with a type from h5py.

Here's a dummy module (called bloop_two.py) which uses h5py.Dataset with:

• |
• Union

from typing import TypeAlias, Union

import h5py

MyType: TypeAlias = h5py.Dataset | int
MyOtherType: TypeAlias = Union[h5py.Dataset, int]

and in the | case sphinx produces a warning:

WARNING: [autosummary] failed to import httomo.bloop_two.
Possible hints:
* TypeError: unsupported operand type(s) for |: 'Dataset' and 'type'
* AttributeError: module 'httomo' has no attribute 'bloop_two'
* ImportError:

but in the Union case sphinx produces no warning.