Unable to filetune falcon-7b with int8

[msinha251]

Hi,

I am trying to finetune falcon with only in8 engine but it's ending up with below error, any idea ?

Also unable to fine-tune base model falcon. Ending up with cuda out of memory error.

Details:
Machine: g5.48xlarge ec2 machine (8 gpu, 22gb each)
xTuring: 0.1.5
Torch: 2.0.1+cu117

Code:

from xturing.datasets.instruction_dataset import InstructionDataset
from xturing.models import BaseModel
import os

instruction_dataset = InstructionDataset("alpaca_data")

# Initializes the model
model = BaseModel.create("falcon_int8")

# Finetuned the model
model.finetune(dataset=instruction_dataset)

# Save the model
model.save("falcon_weights_int8")

error:
RuntimeError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.

[StochasticRomanAgeev]

Hi @msinha251,
Can you please provide full error trace for this model?

[StochasticRomanAgeev]

Hi again @msinha251,
Generic fix for this issue is to set CUDA_VISIBLE_DEVICES=0

[msinha251]

Thanks for the response.