Question: GPU support.

[sehHeiden]

I tried the solar panel Segmentation model from the model zoo.

But the execution was ruther slow it took 80+ minutes for a digital Orthophoto tile (20 cm resultion, 5000 points in each direction).

Inference was done on the CPU, question can I do the inference with Deepness on GPU, too?

Or is it feasible to do?

CPU: Ryzen 7 5800X
GPU: NVidea 1660Ti

[przemyslaw-aszkowski]

Hi
the underneath engine in Deepness is ONNX, which supports GPU. But it doesn't work out of the box, you would need to setup your PC - see instructions for onnxruntime with GPU.
Cheers
Przemek

[sehHeiden]

Thanks a lot. That gave me a lot of headaches. As Pycharm did work, but onnx neither here or this plugin, nor for onnxruntime.

I just, updated cudnn it worked for DEEPNESS. Than removed the CPU option for onnxruntime and the gpu was taken. That was easy. ;)

What makes me wonder is the timing I run the model with DEEPNESS and onnxruntime.

1. DEEPNESS:
   1. CPU: 80 min
   2. GPU: 61 s
2. ONNX RUNTIME (with Python 3.12 but without the erosion, class probability and vectorization...)
   1. CPU: 107 s
   2. GPU: 27 s

What I want to say is, theplugin is much slower on CPU, that what I tried in code.

[sehHeiden]

A symbol/message would be nice, whether GPU, etc. was detected (before running).

[przemyslaw-aszkowski]

Hi,
good idea to show the message whether CPU/GPU detected!
Also quite interesting findings with slow CPU. I do not know the reason, an investigation will be needed.

[sehHeiden]

As the plugin used the Gpu lately I couldn't test, why the CPU was so slow.

I read about speedups of about 40 X for using the GPU. That would be the correct order of magnitude. Or even better.
Perhaps I also set the wrong resolution like 5 cm per pixel compared to the original 10 or 20 cm per pixel.

What I noticed that when I select the raster with a 10 cm resolution, than loading the default parameter does not lead to selecting 10 cm as parameter for the net. For 20 cm it did (often?).

[sehHeiden]

While we were discussing about the CPU speed, too.

Yestday, I tried the same mode in Deepness on a Laptop with CPU. Used a Tuxedo Pulse1.
Using the current TuxedoOS, based on the current Ubuntu LTS with a Ryzen4800H.

Which a generation a lower generation than the Ryzen 5800X I tried before and it's a mobile chip. Still it inference is much faster. But the notebook uses All threads, on the Windows Machine which the faster chip, this never maxed out! Therefore an order of magnitude slower. Hopez this new data point helps.

[aloboa]

Hi the underneath engine in Deepness is ONNX, which supports GPU. But it doesn't work out of the box, you would need to setup your PC - see instructions for onnxruntime with GPU. Cheers Przemek

Could you be more specific on which instructions to follow?
I found
https://stackoverflow.com/questions/64452013/how-do-you-run-a-onnx-model-on-a-gpu
which advises the user to uninstall onnxruntime and then install onnxruntime-gpu. Is this what you advise to do?
For a deep learning plugin, some advice for using GPU in the doc would be most appreciated.

[przemyslaw-aszkowski]

Hi,

From what I remember and what I see in the code Deepness installs onnxruntime-gpu in the first place (on Linux) and to have it running with GPU one needs to install cuda driver in a corresponding version to the onnxruntime-gpu version.

On windows we did not manage to run it with GPU, and it is more complicated, so Deepness installs only the onnxruntime instead of onnxruntime-gpu.

I agree it would be worth adding to the documentation on how to run it with GPU, along with the feature previously mentioned in this thread.

Are you running on Linux or Windows?

[aloboa]

I am running on linux Ubuntu 22.04.5, 13th gen Intel Core I9-13900x32, 64 GB RAM and NVDIA RTX A4000.
I have reinstalled and it seems onnxruntime-gpu has been installed. On the QGIS python console:

# Python Console
# Use iface to access QGIS API interface or type '?' for more info
# Security warning: typing commands from an untrusted source can harm your computer
import onnxruntime as ort
print(ort.get_device())
GPU