Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Finetune] print metrics by rank0 for train and evaluation in fine-tuning #258

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[Finetune] print metrics by rank0 for train and evaluation in fine-tuning #258

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89a5fc7
print metrics by rank0 for train and evalutation in fine-tuning
harborn Jun 20, 2024
402564e
update
harborn Jun 21, 2024
7a89e4f
update
harborn Jun 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions llm_on_ray/finetune/finetune.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ def convert_to_training_args(cls, config: Dict):

# set attr for FSDP
# if accelerate_mode == "FSDP":
# args.updatwe({})
# args.update({})

# set attr for Intel Gaudi
if device == "hpu":
Expand Down Expand Up @@ -171,7 +171,8 @@ def local_load(name, **load_config):
dataset_dict = train_dataset.train_test_split(
test_size=validation_split_percentage / 100
)
dataset_dict["validation"] = dataset_dict["test"]
test_dataset = dataset_dict.pop("test")
dataset_dict["validation"] = test_dataset
return dataset_dict

return datasets.DatasetDict({"train": train_dataset})
Expand All @@ -188,7 +189,8 @@ def local_load(name, **load_config):
dataset_dict = raw_dataset["train"].train_test_split(
test_size=validation_split_percentage / 100
)
dataset_dict["validation"] = dataset_dict["test"]
test_dataset = dataset_dict.pop("test")
dataset_dict["validation"] = test_dataset
return dataset_dict

return raw_dataset
Expand Down Expand Up @@ -367,10 +369,20 @@ def train_func(config: Dict[str, Any]):

training_args, trainer = get_trainer(config, model, tokenizer, tokenized_dataset, data_collator)

common.logger.info("train start")
trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
trainer.save_model()
common.logger.info("train finish")
if training_args.do_train:
common.logger.info("train start")
result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
trainer.save_model()
metrics = result.metrics
metrics["throughput"] = len(tokenized_dataset["train"]) / metrics["train_runtime"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to multiply the number of epoch? Can you please share the result of one of your runs? How does the result compare with the result on Gaudi dashboard?

trainer.log_metrics("train", metrics)
common.logger.info("train finish")

if training_args.do_eval:
common.logger.info("eval start")
metrics = trainer.evaluate()
trainer.log_metrics("eval", metrics)
common.logger.info("eval finish")


def get_finetune_config():
Expand Down
Loading