how to infer with finetuned model?

[balaji-skoruz]

Got adapter_mode.bin,adapter_config.json after finetuning the vicuna 4bit 128g model ,
and checkpoints folders.

how to use this folders or files to infer the model?

[afnanhabib787]

following

[johnsmith0031]

Use this

from monkeypatch.peft_tuners_lora_monkey_patch import replace_peft_model_with_gptq_lora_model
replace_peft_model_with_gptq_lora_model()
from peft import PeftModel
from monkeypatch.peft_tuners_lora_monkey_patch import Linear4bitLt
model = PeftModel.from_pretrained(model, lora_path, device_map={'': 'cpu'}, torch_dtype=torch.float32, is_trainable=True)

It's inside load_llama_model_4bit_low_ram_and_offload function

[ra-MANUJ-an]

@johnsmith0031 what should be at the place of model, lora_path?

[johnsmith0031]

Yes, lora_path should be point to the path of finetuned lora model.