GRPO: compute rewards over eval set #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sidnarayanan
commented
Feb 14, 2025
Comment on lines
-177
to
-200
|
|
||
| # From https://github.com/huggingface/trl/pull/2700/files | ||
| sync_ref_model: bool = field( | ||
| default=False, | ||
| metadata={ | ||
| "help": "Whether to synchronize the reference model with the active model every `ref_model_sync_steps` " | ||
| "steps, using the `ref_model_mixup_alpha` parameter." | ||
| }, | ||
| ) | ||
| ref_model_mixup_alpha: float = field( | ||
| default=0.9, | ||
| metadata={ | ||
| "help": "α parameter from the TR-DPO paper, which controls the mix between the current policy and the " | ||
| "previous reference policy during updates. The reference policy is updated according to the equation: " | ||
| "`π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`." | ||
| }, | ||
| ) | ||
| ref_model_sync_steps: int = field( | ||
| default=64, | ||
| metadata={ | ||
| "help": "τ parameter from the TR-DPO paper, which determines how frequently the current policy is " | ||
| "synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`." | ||
| }, | ||
| ) |
There was a problem hiding this comment.
This doesn't work for large models, and checkpointing the best epochs (which eval metrics enables) lets us do the reset manually. So I'm removing the configuration.
jamesbraza
approved these changes
Feb 14, 2025
|
sidnarayanan
changed the title
jamesbraza
added a commit
that referenced
this pull request
Feb 26, 2025
* Ported compute_reward_metrics and integrated into GRPOTrainer.__init__ * Decomposed _compute_rewards_per_func out of _generate_and_score_completions * Decomposed _extract_completions from _generate_and_score_completions * Implemented a custom generation_config route in _generate_and_score_completions * Decomposed _generate, and plugged it into prediction_step
jamesbraza
added a commit
that referenced
this pull request
Feb 26, 2025
* Ported compute_reward_metrics and integrated into GRPOTrainer.__init__ * Decomposed _compute_rewards_per_func out of _generate_and_score_completions * Decomposed _extract_completions from _generate_and_score_completions * Implemented a custom generation_config route in _generate_and_score_completions * Decomposed _generate, and plugged it into prediction_step
jamesbraza
added a commit
that referenced
this pull request
Feb 27, 2025
* Ported compute_reward_metrics and integrated into GRPOTrainer.__init__ * Decomposed _compute_rewards_per_func out of _generate_and_score_completions * Decomposed _extract_completions from _generate_and_score_completions * Implemented a custom generation_config route in _generate_and_score_completions * Decomposed _generate, and plugged it into prediction_step
jamesbraza
added a commit
that referenced
this pull request
Feb 28, 2025
* Ported compute_reward_metrics and integrated into GRPOTrainer.__init__ * Decomposed _compute_rewards_per_func out of _generate_and_score_completions * Decomposed _extract_completions from _generate_and_score_completions * Implemented a custom generation_config route in _generate_and_score_completions * Decomposed _generate, and plugged it into prediction_step
jamesbraza
added a commit
that referenced
this pull request
Mar 8, 2025
* Ported compute_reward_metrics and integrated into GRPOTrainer.__init__ * Decomposed _compute_rewards_per_func out of _generate_and_score_completions * Decomposed _extract_completions from _generate_and_score_completions * Implemented a custom generation_config route in _generate_and_score_completions * Decomposed _generate, and plugged it into prediction_step
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Basically what the title says. Trainer's eval loop is built around losses/logits, so I am abusing the return and call signatures of
prediction_stepandcompute_metricsrespectively. This lets us compute+log rewards without rewriting the whole eval loop and logging mechanism.I also pulled in the changes from huggingface#2776, which are needed to unwrap the model for generation in the eval loop.
Most of the code diff is me pulling code out of
compute_lossinto separate methods, so they can be reused inprediction_step