"No GPU being used. Careful, inference might be very slow!" #202

dreiscm · 2023-04-28T14:44:44Z

I am running the code right now, and the audio file is in the proccess, but it is REALLY slow, like almost an hour. Along with the proccess, i got this message: "No GPU being used. Careful, inference might be very slow!". The thing is, i do have a NVIDA 1050 as GPU in my laptop. How can i make the programm use my GPU to run it faster?

genetrader · 2023-04-28T16:47:38Z

Same issue - I have a Nvidia GTX 2080 with 8gb of RAM.

gkucsko · 2023-04-28T20:34:26Z

hmm strange so two separate things

first check if torch can see your gpu. basically, try to get torch.cuda.is_available() running first independently of bark following their installs
even on cpu technically it should run faster than that. maybe try loading the smaller version of the models and see if that's faster?

def preload_models(
    text_use_small=True,
    coarse_use_small=True,
    fine_use_small=True,
)

taktakut · 2023-04-29T04:15:00Z

Hi, here is in my case.
I had the same situation, and check Pytorch with print(torch.version) in my current environment. I found it's just for CPU (2.0.0+cpu), not for CUDA ver.

rubinchyk · 2023-04-29T13:34:29Z

RTX3070, same problem...

P.S. This helped #173

SECRET-GUEST · 2023-05-01T06:09:29Z

Same issue, you can check if you have a compatible version of torch with your graphical card while running in a python file .py :

import torch

print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Number of GPUs:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))

For me I'm compatible, so I suppose the problem come from the allocated Vram by default not settable.

dreiscm · 2023-05-01T14:08:51Z

Same issue, you can check if you have a compatible version of torch with your graphical card while running in a python file .py :
import torch

print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Number of GPUs:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))
For me I'm compatible, so I suppose the problem come from the allocated Vram by default not settable.

Thank you for the help. It seems i did not have neither CUDA nor the the Torch Version to use the GPU. I fixed both of those, but now i am getting the following:

untyped_storage = torch.UntypedStorage(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.45 GiB already allocated; 0 bytes free; 3.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

SECRET-GUEST · 2023-05-01T17:07:27Z

Ok so you should probably enable small model, by changing False to True into bark/bark/generation.py

USE_SMALL_MODELS = os.environ.get("SUNO_USE_SMALL_MODELS", False)

Also this interface is much better : https://github.com/C0untFloyd/bark-gui

bdevel · 2023-05-01T21:09:03Z

I received the "No GPU being used" warning because I'm using Apple silicon. Anyone with this error just needs to enable MPS with an environment variable. For example:

SUNO_ENABLE_MPS=True python3 speak.py

Note to maintainers, this should probably just be automatic if torch.backends.mps.is_available() == True

Everyone else, more info on the MacOS MPS backend for PyTorch.

anjy077 · 2023-05-03T03:50:50Z

I also have this problem

HWiese1980 · 2023-05-03T18:33:27Z

Same issue, you can check if you have a compatible version of torch with your graphical card while running in a python file .py :
import torch

print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Number of GPUs:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))
For me I'm compatible, so I suppose the problem come from the allocated Vram by default not settable.
Thank you for the help. It seems i did not have neither CUDA nor the the Torch Version to use the GPU. I fixed both of those, but now i am getting the following:

untyped_storage = torch.UntypedStorage( torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.45 GiB already allocated; 0 bytes free; 3.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Could you please tell us exactly what you did to solve your initial problem?

HWiese1980 · 2023-05-03T19:06:52Z

Okay, I think I solved it too. What I did, in the corresponding virtualenv for bark:

bark$ pip3 install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

This should be enough to get GPU inference going.

johlim · 2023-05-06T17:42:03Z

cuda Download link
https://developer.nvidia.com/cuda-downloads

torch Download Link
https://download.pytorch.org/whl/torch/
pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
tourchaudio tourch audio,vision downloadLink

https://download.pytorch.org/whl/torchaudio/
https://download.pytorch.org/whl/torchvision/

If you're using python 3.10 and torch 2.0, you can use
torchaudio-2.0.0+cu117-cp310-cp310-win_amd64.whl
torchvision-0.15.1+cpu-cp310-cp310-win_amd64.whl

Install Example

 pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu117/torch_stable.html
Collecting torch==2.0.0+cu117
  Using cached https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl (2343.6 MB)
Requirement already satisfied: filelock in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.12.0)
Requirement already satisfied: typing-extensions in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (4.5.0)
Requirement already satisfied: sympy in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (1.12rc1)
Requirement already satisfied: networkx in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1)
Requirement already satisfied: jinja2 in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\python310\lib\site-packages (from jinja2->torch==2.0.0+cu117) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in c:\python310\lib\site-packages (from sympy->torch==2.0.0+cu117) (1.3.0)
Installing collected packages: torch
Successfully installed torch-2.0.0+cu117

D:\VDisk\Bark>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:59:34_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

D:\VDisk\Bark>python
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("Torch version:", torch.__version__)
Torch version: 2.0.0+cu117
>>> print("CUDA available:", torch.cuda.is_available())
CUDA available: True
>>> print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 1

johlim · 2023-05-06T17:59:42Z

I was struggling with the out of memory error message, but when I typed it correctly in the running console environment, it worked. "set SUNO_USE_SMALL_MODELS=True"

    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 7.26 GiB already allocated; 0 bytes free; 7.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

D:\VDisk\Bark>set SUNO_USE_SMALL_MODELS=True

D:\VDisk\Bark>python main.py
100%|████████████████████████████████████████████████████████████████████████████████| 100/100 [00:01<00:00, 87.93it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 13/13 [00:03<00:00,  3.82it/s]

mrbb · 2023-05-07T19:52:37Z

cuda Download link https://developer.nvidia.com/cuda-downloads

torch Download Link
https://download.pytorch.org/whl/torch/
pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
tourchaudio tourch audio,vision downloadLink

https://download.pytorch.org/whl/torchaudio/ https://download.pytorch.org/whl/torchvision/

If you're using python 3.10 and torch 2.0, you can use torchaudio-2.0.0+cu117-cp310-cp310-win_amd64.whl torchvision-0.15.1+cpu-cp310-cp310-win_amd64.whl

Install Example

 pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu117/torch_stable.html
Collecting torch==2.0.0+cu117
  Using cached https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl (2343.6 MB)
Requirement already satisfied: filelock in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.12.0)
Requirement already satisfied: typing-extensions in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (4.5.0)
Requirement already satisfied: sympy in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (1.12rc1)
Requirement already satisfied: networkx in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1)
Requirement already satisfied: jinja2 in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\python310\lib\site-packages (from jinja2->torch==2.0.0+cu117) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in c:\python310\lib\site-packages (from sympy->torch==2.0.0+cu117) (1.3.0)
Installing collected packages: torch
Successfully installed torch-2.0.0+cu117

D:\VDisk\Bark>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:59:34_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

D:\VDisk\Bark>python
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("Torch version:", torch.__version__)
Torch version: 2.0.0+cu117
>>> print("CUDA available:", torch.cuda.is_available())
CUDA available: True
>>> print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 1

This has saved me a day. It works. Thank you.

Niconord59 · 2023-05-08T11:18:59Z

Hello ! Is it possible to run it with an AMD 6900XT? Because although I have an AMD Ryzen 9 5950X it is slow.

gkucsko · 2023-05-08T19:29:06Z

right now amd is not supported and i can't test without an amd gpu. open to PRs from the community though if anyone can have a look

rubinchyk · 2023-05-09T06:29:26Z

right now amd is not supported and i can't test without an amd gpu. open to PRs from the community though if anyone can have a look

I have AMD and it's working for me with GPU.

gkucsko · 2023-05-11T18:13:13Z

looks like this is fixed for most people. also the topic of the issue is a bit non-descript since it just indicates lack of a gpu

johlim · 2023-05-12T00:05:23Z

@gkucsko
It would be nice to add an example to the ReadMe documentation of using set SUNO_USE_SMALL_MODELS=True in the console before doing the python code.

gkucsko · 2023-05-13T13:18:19Z

yeah, since we didn't really include a command line script i figured from within python is easier, but ya if you launch something it could be useful to do outside. although now we have a preload_models option as well for small models which might be more straighforward for people anyways

builder-main · 2023-05-31T23:17:22Z

I also had to uninstall before reinstall as I installed pytorch before cuda 11.8
pip3 uninstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Before :

$ python info.py 
Torch version: 2.0.1+cpu
CUDA available: False
Number of GPUs: 0
AssertionError: Torch not compiled with CUDA enabled

After

$ python info.py 
Torch version: 2.0.1+cu118
CUDA available: True
Number of GPUs: 1
GPU name: NVIDIA GeForce RTX 3080

JonathanFly · 2023-06-19T20:22:44Z

Hello ! Is it possible to run it with an AMD 6900XT? Because although I have an AMD Ryzen 9 5950X it is slow.

It's not ROCM, but DirectML seems to be better than nothing for Window AMD:

JonathanFly#79 (comment)

https://github.com/JonathanFly/bark/tree/bark_amd_directml_test#-bark-amd-install-test-

jaggzh · 2024-04-12T04:56:54Z

I'm in Linux, but I'm getting lots of memory and cpu use, but not seeing any gpu use at all (unless it suddenly uses it for a split second at the end).
Inference takes a long time -- like 6 minutes for a 24 word sentence.

>>> print("Torch version:", torch.__version__)
Torch version: 2.2.1+cu121
>>> print("CUDA available:", torch.cuda.is_available())
CUDA available: True
>>> print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 1
>>> print("GPU name:", torch.cuda.get_device_name(0))
GPU name: NVIDIA GeForce RTX 3090

$ time ./bark
2024-04-11 21:48:29.666824: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/.../python3.11/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
The attention mask and the pad token id were not set. As a consequence, you mayobserve unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:10000 for open-end generation.

real    5m57.385s
user    20m17.150s
sys     1m14.751s

luke1212 · 2024-04-20T21:44:19Z

RTX3070, same problem...

P.S. This helped #173

This helps！ thanks

gkucsko closed this as completed May 11, 2023

ChamathDilanka mentioned this issue Sep 7, 2023

Can somebody guide me? Bark doesn't Bark after pytorch version update. #453

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"No GPU being used. Careful, inference might be very slow!" #202

"No GPU being used. Careful, inference might be very slow!" #202

dreiscm commented Apr 28, 2023

genetrader commented Apr 28, 2023

gkucsko commented Apr 28, 2023

taktakut commented Apr 29, 2023

rubinchyk commented Apr 29, 2023 •

edited

Loading

SECRET-GUEST commented May 1, 2023

dreiscm commented May 1, 2023

SECRET-GUEST commented May 1, 2023 •

edited

Loading

bdevel commented May 1, 2023

anjy077 commented May 3, 2023

HWiese1980 commented May 3, 2023

HWiese1980 commented May 3, 2023

johlim commented May 6, 2023 •

edited

Loading

johlim commented May 6, 2023 •

edited

Loading

mrbb commented May 7, 2023

Niconord59 commented May 8, 2023 •

edited

Loading

gkucsko commented May 8, 2023

rubinchyk commented May 9, 2023

gkucsko commented May 11, 2023

johlim commented May 12, 2023

gkucsko commented May 13, 2023

builder-main commented May 31, 2023

JonathanFly commented Jun 19, 2023

jaggzh commented Apr 12, 2024

luke1212 commented Apr 20, 2024

"No GPU being used. Careful, inference might be very slow!" #202

"No GPU being used. Careful, inference might be very slow!" #202

Comments

dreiscm commented Apr 28, 2023

genetrader commented Apr 28, 2023

gkucsko commented Apr 28, 2023

taktakut commented Apr 29, 2023

rubinchyk commented Apr 29, 2023 • edited Loading

SECRET-GUEST commented May 1, 2023

dreiscm commented May 1, 2023

SECRET-GUEST commented May 1, 2023 • edited Loading

bdevel commented May 1, 2023

anjy077 commented May 3, 2023

HWiese1980 commented May 3, 2023

HWiese1980 commented May 3, 2023

johlim commented May 6, 2023 • edited Loading

johlim commented May 6, 2023 • edited Loading

mrbb commented May 7, 2023

Niconord59 commented May 8, 2023 • edited Loading

gkucsko commented May 8, 2023

rubinchyk commented May 9, 2023

gkucsko commented May 11, 2023

johlim commented May 12, 2023

gkucsko commented May 13, 2023

builder-main commented May 31, 2023

JonathanFly commented Jun 19, 2023

jaggzh commented Apr 12, 2024

luke1212 commented Apr 20, 2024

rubinchyk commented Apr 29, 2023 •

edited

Loading

SECRET-GUEST commented May 1, 2023 •

edited

Loading

johlim commented May 6, 2023 •

edited

Loading

johlim commented May 6, 2023 •

edited

Loading

Niconord59 commented May 8, 2023 •

edited

Loading