Loss is "nan" while training gpt2

[sankethgadadinni]

model = BaseModel.create("gpt2")
instruction_dataset = InstructionDataset("/content/alpaca_data")
model.finetune(dataset=instruction_dataset)

Am I doing something wrong?

[tushar2407]

Hi @sankethgadadinni , No you are not doing anything wrong, it's just that may be the loss is so high or so low in your case that it cannot be handled and hence you are getting NAN. It's normal to get that in many cases.

[tushar2407]

@sankethgadadinni were you able to load the model after fine-tuning? Does the model work after fine-tuning?
I can help you with walking through the process, I am hold an all-hands session coming Friday, you are welcome to join!!
For more details please head to discord channel: https://discord.gg/xj5j3VJC