Restoring scribbles using torchmaxflow

[masadcv]

Signed-off-by: masadcv [email protected]

I have update and released binary for torchmaxflow (https://github.com/masadcv/torchmaxflow) which can now be used for scribbles. It has pre-built windows binaries so will resolve issues reported in #719

This PR restores scribbles functionality using graphcut method from torchmaxflow.
This PR still has issue as reported in #713
It will be resolved with merging of an on-going PR #717 once this is merged.

[SachidanandAlle]

I tried this on Windows.. version is windows 11

[2022-04-11 23:00:09,082] [4780] [MainThread] [ERROR] (uvicorn.error:119) - Traceback (most recent call last):
  File "C:\Users\salle\AppData\Local\Programs\Python\Python39\lib\site-packages\starlette\routing.py", line 621, in lifespan
    async with self.lifespan_context(app):
  File "C:\Users\salle\AppData\Local\Programs\Python\Python39\lib\site-packages\starlette\routing.py", line 518, in __aenter__
    await self._router.startup()
  File "C:\Users\salle\AppData\Local\Programs\Python\Python39\lib\site-packages\starlette\routing.py", line 598, in startup
    await handler()
  File "C:\Projects\MONAILabel\sb\monailabel\app.py", line 101, in startup_event
    instance = app_instance()
  File "C:\Projects\MONAILabel\sb\monailabel\interfaces\utils\app.py", line 44, in app_instance
    c = get_class_of_subclass_from_file("main", main_py, "MONAILabelApp")
  File "C:\Projects\MONAILabel\sb\monailabel\utils\others\class_utils.py", line 54, in get_class_of_subclass_from_file
    return get_class_of_subclass(module_from_file(module_name, file_path), class_c)
  File "C:\Projects\MONAILabel\sb\monailabel\utils\others\class_utils.py", line 26, in module_from_file
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "C:\Projects\MONAILabel\sb\sample-apps\radiology\main.py", line 27, in <module>
    from monailabel.scribbles.infer import HistogramBasedGraphCut
  File "C:\Projects\MONAILabel\sb\monailabel\scribbles\infer.py", line 15, in <module>
    from monailabel.scribbles.transforms import (
  File "C:\Projects\MONAILabel\sb\monailabel\scribbles\transforms.py", line 23, in <module>
    from .utils import make_iseg_unary, make_likelihood_image_histogram, maxflow
  File "C:\Projects\MONAILabel\sb\monailabel\scribbles\utils.py", line 18, in <module>
    import torchmaxflow
ImportError: DLL load failed while importing torchmaxflow: The specified procedure could not be found.

[SachidanandAlle]

Looks like torchmaxflow-0.0.5 is building native dep during the setup

[masadcv]

Thanks for looking into this @SachidanandAlle - let me debug what's causing this. v0.0.4rc1 seems to work fine so must be some recent changes.

[masadcv]

This was caused due to mismatch of pytorch version from the one used to generate the wheel.

torchmaxflow now has requirement pytorch>=1.10 (in torchmaxflow==0.0.6rc1), as previous versions have difference in DLLs on windows and require compiling from source for that version. 0.0.6rc1 works with pytorch versions >=1.10.

It has now been tested and works with pytorch 1.10 to 1.11 - let me know if you still see issues

[jamesobutler]

@masadcv On Windows I'm observing an issue with 0.0.6rc1 still.

PS C:\Users\JamesButler\AppData\Local\Programs\Python\Python39> ./Python.exe -m pip install torchmaxflow==0.0.6rc1
Collecting torchmaxflow==0.0.6rc1
  Downloading torchmaxflow-0.0.6rc1-cp39-cp39-win_amd64.whl (84 kB)
     ---------------------------------------- 84.5/84.5 KB 2.4 MB/s eta 0:00:00
Requirement already satisfied: torch>=1.10.0 in c:\users\jamesbutler\appdata\local\programs\python\python39\lib\site-packages (from torchmaxflow==0.0.6rc1) (1.11.0)
Requirement already satisfied: typing-extensions in c:\users\jamesbutler\appdata\local\programs\python\python39\lib\site-packages (from torch>=1.10.0->torchmaxflow==0.0.6rc1) (4.1.1)
Installing collected packages: torchmaxflow
Successfully installed torchmaxflow-0.0.6rc1
PS C:\Users\JamesButler\AppData\Local\Programs\Python\Python39> ./Python.exe
Python 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchmaxflow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing torchmaxflow: The specified module could not be found.
>>>

[masadcv]

@jamesobutler thanks for trying this! you need to import torch (import torch) before importing torchmaxflow.
This is a standard practice for pytorch extensions as they depend on DLL/so from pytorch (see https://stackoverflow.com/a/65710714/798093)

If you want to try a simple example, maybe give this a try (https://github.com/masadcv/torchmaxflow/blob/main/demo_maxflow.py) - let me know if you see any issues

[jamesobutler]

Ok the following works:

PS C:\Users\JamesButler\AppData\Local\Programs\Python\Python39> ./Python.exe
Python 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torchmaxflow
>>>

I would have expected a c-style extension to not require certain import order relative to another package. Even though scipy has a dependency on numpy it doesn't require importing of numpy before using scipy. I haven't looked further into why torch does it this way. I can confirm they mention the special import order in https://pytorch.org/tutorials/advanced/cpp_extension.html#using-your-extension.

[jamesobutler]

What are the maintenance requirements of such a pytorch extension? What happens when using Python 3.10, or there is a different ABI compatibility for a new PyTorch version? Without an upper limit of the pytorch requirement for torchmaxflow will there likely be a point where it just suddenly breaks due to ABI incompatibility?

[masadcv]

What are the maintenance requirements of such a pytorch extension? What happens when using Python 3.10, or there is a different ABI compatibility for a new PyTorch version? Without an upper limit of the pytorch requirement for torchmaxflow will there likely be a point where it just suddenly breaks due to ABI incompatibility?

If you are using the current binary pre-compiled wheels in future with those (python 3.10 or different ABI) then most likely they will fail.
This kind of pytorch extension is quite flexible and versatile in its application - because you can easily compile it anywhere you can have a pytorch version. I have tested this all the way back with pytorch 1.6 and all works well on linux machines. The reason for current limitation of pytorch 1.10 to 1.11 is mainly due to requirement of having windows pre-compiled wheels (and not requiring user to compile them, as we have seen a mess with SimpleCRF). This is what is currently dominating the targetted pytorch versions for torchmaxflow. If it helps I can put an upper limit of 1.11 but it really is applicable across different versions of pytorch.

I am also looking at better ways of compiling wheel (e.g. https://github.com/charliebudd/torch-extension-builder) - however, this is not ready yet.

The other option could be to interface the maxflow C++ code through numpy (e.g. numpymaxflow) - however, I currently dont have enough bandwidth to attempt this for this PR.

EDIT:
Forgot to mention, this is distributed along with sdist - so where there is a mismatch (e.g. python 3.10) it will automatically switch to compiling from source from pypi

[jamesobutler]

requirement of having windows pre-compiled wheels (and not requiring user to compile them, as we have seen a mess with SimpleCRF)

I asked about maintainability to avoid a situation like SimpleCRF in the future for torchmaxflow. An issue there was that SimpleCRF was primarily developed by 1 individual who was no longer responsive to the project to update it. Does it make sense to transfer torchmaxflow to the Project-MONAI organization if it is primarily going to be used by MonaiLabel and would in theory need to be maintained by any future MonaiLabel maintainers whoever comes and go from the project? I'm assuming the intention for torchmaxflow usage here in MonaiLabel is more than just a short temporary solution.

[masadcv]

I dont see any issues with maintenance as I will be around to maintain it. If there are any PRs for torchmaxflow, I will be happy to review and add those in.

Regarding moving this to MONAI - we already had issues with C++ compilable code in MONAI. To compile these, MONAI needs to be compiled/installed in non-standard way (e.g. using BUILD_MONAI=1 see: https://docs.monai.io/en/stable/installation.html#from-github). I will be happy to move this up there - if the developer agree or have an easier way of compiling C++ code in the library for default installation.

And above all that, you would still end up with situations where users dont have access to build tools (e.g. on Windows) and will need to provide compiled wheel to cover common pytorch/python versions for MONAI.

[jamesobutler]

I dont see any issues with maintenance as I will be around to maintain it.

This may be easy to know for the short term, but I only worry that it ends up like SimpleCRF where the person ultimately moves on to something else. Since it is a repo currently only maintained by 1 person there is a risk of requiring a fork in the future to workaround an issue similar to what you had to do with the creation of the https://pypi.org/project/SimpleCRF-binaries/ project.

Regarding moving this to MONAI - we already had issues with C++ compilable code in MONAI.

My comment about transferring torchmaxflow to the Project-MONAI organization was specifically about transferring the GitHub repository to the Project-MONAI github organization (would be https://github.com/Project-MONAI/torchmaxflow) as there will be multiple people available to perform admin tasks as needed if you for some reason are unavailable or have moved on to another project.

[tvercaut]

I would even say that having this functionality in MONAI proper would be great and certainly would help with maintenance. In the short term, MONAI doesn't curently properly support c++ and cuda extensions though. My suggestion would be for @charliebudd to get a first complete built pipeline from https://github.com/charliebudd/torch-extension-builder and @masadcv to get a first version of torchmaxflow up and running.

@masadcv, @charliebudd: We can also certainly transfer the repos to the Project-MONAI organization to ease collaboration in the very short term.

[SachidanandAlle]

I agree with @tvercaut we need this kind of annotation method in monailabel.. unless license kind of blocker issues, other contribution/collaboration related things can be addressed in near future...

[lassoan]

The other option could be to interface the maxflow C++ code through numpy (e.g. numpymaxflow) - however, I currently dont have enough bandwidth to attempt this for this PR.

The algorithm could be usable in a broad context, so it would make sense to build with minimal dependency. Pytorch on my computer required 20GB disk space during installation and ended up taking up over 6GB, which would scare away people from using this maxflow package if they don't need pytorch for other reasons anyway.

Numpy is a good minimal dependency. It could be distributed as an ITK remote module, too.

[charliebudd]

Pytorch install should be around 1.3GB, and most of this is the cuda runtime which isn't included in the cpu distributions. Binding through pytorch leaves space for a cuda implementation in the future.

Edit: Sorry just checked on my install, uncompressed looks to be around 3.5GB for the cuda 11.3 versions, 2.6GB of this are cuda related binaries. cuda 10 versions will be much smaller and cpu version smaller still.

[masadcv]

This PR now has multi-label scribbles and dynamic scribbles labels from #717

Regarding the torchmaxflow, I agree this should be maintainable – I am happy to work out details of how this can be done.

[lassoan]

cuda 10 versions will be much smaller and cpu version smaller still.

On Windows, pytorch install with CUDA 11.1.74 is 6.2GB. I install pytorch with light-the-torch and the installation fails if I have less than 20GB free disk space. I guess the installer downloads the packages, unpacks them, copies them over to the final location; and does not try to optimize for disk space usage during this process. To make things worse, if installation is incomplete due to running out of disk space then pytorch partially works so detection of pytorch becomes unreliable. Cleaning up the Python environment after this partial installation is very hard (I usually need to create a new environment from scratch).

With only CPU support pytorch is just 800MB, which is good (if a computer has less disk space than that then that leads to other problems anyway).

This PR now has multi-label scribbles and dynamic scribbles labels from

Great!