Where should I submit the PR to compile GPU support by default? Here or llama-cpp-python?

[shouyiwang]

I believe it's inconvenient for GPU users to manually compile the code for llama-cpp-python in Webui every time there is a version bump. I've devised two potential solutions to this issue and written code for both.

The first involves modifying the setup.py file in llama-cpp-python to include default GPU support, assuming the user has a GPU and no envs like CMAKE_ARGS="-DLLAMA_CUBLAS=on" is set.

The second involves altering the text-generation-webui pip install requirements.txt command to python install.py. This python file would also call pip install requirements.txt and would check for GPU availability, subsequently installing the GPU-supported version if one is detected.

There are a couple of potential issues to consider. The first solution might lead to unwanted consequences because I'm uncertain about the implications of making GPU support the default behavior. For the second solution, it represents a significant shift in the installation process for the sake of one module, namely llama-cpp-python.

Given these considerations, I'm seeking advice on the preferable approach. Where should I submit a PR for this proposed solution?

[BarfingLemurs]

make it optional in the one click installers.

For the second solution, it represents a significant shift in the installation process for the sake of one module, namely llama-cpp-python.

the failure to compile results in the one click installer cancelling the entire installation process. llama-cpp-python should be optional

[shouyiwang]

But that doesn't solve the original problem:
"It's inconvenient for GPU users to manually compile the code for llama-cpp-python in Webui every time there is a version bump"

[BarfingLemurs]

maybe they could release wheels for cuda builds?
on windows

[berkut1]

@shouyiwang one click installers also has the updater script. So @BarfingLemurs said right. you need one click installers

[jparmstr]

I have a (Windows only?) workaround for this because it's been inconvenient for me too. You can edit update_windows.bat or create a copy of it and add the lines to automatically reinstall llama-cpp-python with GPU support. The relevant section of the batch script is below, showing what I added and where.

@rem update installer env
call python webui.py --update

@rem reinstall llama.cpp with gpu support
pip uninstall -y llama-cpp-python
set CMAKE_ARGS="-DLLAMA_CUBLAS=on"
set FORCE_CMAKE=1
pip install llama-cpp-python --no-cache-dir

echo.
echo Done!

[berkut1]

Your script update to the latest version, ignore the version that in requirements.txt
But as solution for some people who are tired of manually reinstalling llama-cpp-python, that looks fine :)