Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error with monkeypatch and model gpt-j and lora #103

Open
ReDXeoL opened this issue May 12, 2023 · 0 comments
Open

error with monkeypatch and model gpt-j and lora #103

ReDXeoL opened this issue May 12, 2023 · 0 comments

Comments

@ReDXeoL
Copy link

ReDXeoL commented May 12, 2023

Hello, I would like to discuss a problem I have with only the 4bit quantized gpt-j models (gpt-j-6B-alpaca-4bit-128g) with the help of the AutoGPTQ method when using the monkeypatch to train with lora

(this only happens with this type of models, models like vicuna or WizardLM work very well with monkeypatch)

ERROR:
Load Model ...
WARNING:The safetensors archive passed at models\bertin-gpt-j-6B-alpaca-4bit-128g\gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata.
Traceback (most recent call last):
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\server.py", line 932, in
shared.model, shared.tokenizer = load_model(shared.model_name)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\models.py", line 153, in load_model
model, _ = load_model_llama(model_name)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\monkey_patch_gptq_lora.py", line 24, in load_model_llama
model, tokenizer = load_llama_model_4bit_low_ram(config_path, model_path, groupsize=shared.args.groupsize, is_v1_model=False)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\repositories\alpaca_lora_4bit\autograd_4bit.py", line 204, in load_llama_model_4bit_low_ram
model = accelerate.load_checkpoint_and_dispatch(
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\big_modeling.py", line 479, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 946, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 135, in set_module_tensor_to_device
if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
AttributeError: 'NoneType' object has no attribute 'device'
Captura
pc: rtx 3060 (12GB) i7 10700 32 ram

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant