Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Finetune] Fix evaluation #252

Merged
merged 15 commits into from
Jun 28, 2024
Merged

[Finetune] Fix evaluation #252

merged 15 commits into from
Jun 28, 2024

Conversation

minmingzhu
Copy link
Contributor

@minmingzhu minmingzhu commented Jun 16, 2024

After trying several solutions, I finally used the tokenize_func from neural_chat:

  1. Using the transformer tokenizer with padding and attention_mask, and copying input_ids to labels.
  2. Using the transformer tokenizer with padding and attention_mask, and masking the input in the prompt.
  3. Using the transformer tokenizer with padding and attention_mask, and masking the response in the prompt.
  4. Using neural_chat's tokenize_func for padding, filling attention_mask and labels which masking the input in the prompt.
<style> </style>
Case Average ARC HellaSwag MMLU TruthfulQA Winogrande
Case 1 62.116 55.97 78.39 60.75 45.86 69.61
Case 2 58.99 54.18 72.85 60.23 42.18 65.51
Case 3 59.064 53.24 73.21 60.85 41.56 66.46
Case 4 63.03 53.67 81.47 60.27 43.02 76.72

@minmingzhu minmingzhu changed the title Fix evaluation [Finetune] Fix evaluation Jun 17, 2024
llm_on_ray/finetune/DataPreprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/finetune.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/DataPreprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/DataPreprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/DataPreprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/DataPreprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/DataPreprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/data_preprocess.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/finetune.py Outdated Show resolved Hide resolved
llm_on_ray/finetune/data_preprocess.py Outdated Show resolved Hide resolved
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
2. fix comments

Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
Signed-off-by: minmingzhu <[email protected]>
@minmingzhu minmingzhu merged commit 0b44ac4 into intel:main Jun 28, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants