onnx and ncnn support

[rahmirtrt]

Onnx and ncnn support can be added to upscaler.

[Teriks]

I was looking at this today and it seems that https://pypi.org/project/ncnn-vulkan/ does not currently have a python 3.12 release, though I left a question on the repository about it.

It is maintained by the chainner developers.

This project is on python 3.12

[rahmirtrt]

You can review the resources here.
https://github.com/TNTwise/Universal-NCNN-Upscaler

[Teriks]

The upscaling is possible & easy with the normal ncnn python module on PyPI, I have tested. no ncnn-vulkan required.

The problem is that it depends opencv-python instead of opencv-python-headless

And I would really like to avoid bringing in opencv-python as it is larger, and I am pretty sure it does not need all the GUI functionality in there to function.

So that is a puzzle I have to solve.

[Teriks]

I have an ncnn upscaler implemented in https://github.com/Teriks/dgenerate/tree/upscaler_ncnn, but I am not really able to find many models to test with it aside from these https://github.com/nihui/realsr-ncnn-vulkan/blob/master/models/models-DF2K/

[Teriks]

@rahmirtrt if you could provide me links to ncnn models or converted models that you use with chainner other than the realsr model I pointed out that would be helpful

[rahmirtrt]

I converted most models from pth to onnx on chainner and then from onnx to ncnn.

there are some here.
https://github.com/TNTwise/SPAN-ncnn-vulkan/tree/master/custom_models

[Teriks]

I was able to get these models working with a tweak, thanks for pointing them out.

My implementation does not pre-optimize the models layers as chainner does, but it seems the chainner optimization code for the model just duplicates the functionality of https://github.com/Tencent/ncnn/wiki/use-ncnnoptimize-to-optimize-model in python. So I am not sure if it is necessary for me to implement all that when ncnn distributes a binary to do the same thing. It would be very time consuming.

[Teriks]

these work: https://github.com/upscayl/upscayl/tree/main/models

I think I only need to investigate CPU specific optimization flags for CPU usage and then it is done

I can include ncnn by default in the prebuilt windows environment as its not going to cause any suprises for anyone there that opencv-python is being used instead of opencv-python-headless I think.

for manual installs I will include a setup extra that enables it ie: pip install dgenerate[ncnn] --extra-index-url ...

Main issue with including as a default dependency is that it might cause issues on a headless nix system when opencv tries to look for certain native libraries. So it is pretty much only safe to use on Linux if you have a window manager installed when ncnn is brought in, unless you manually remove opencv-python from your environment and replace it with the headless version.

[Teriks]

A generic NCNN upscaler is now included in the latest windows releases by default. The spandrel project intends to support ONNX directly in the future.

I will probably be doing a bit less feature adding on the project, just bugfixes until diffusers releases its next version with the rest of the new SD3 pipelines that are in the works, or (maybe) if compel finishes SD3 prompt weighting support, and (maybe) if spandrel releases a new version.