[ Loss Scale ] Enable to define different type of Variable and Gradient for Loss Scale #2562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It add loss scale property as model common property. Signed-off-by: Jiho Chu <[email protected]>
This PR enables the loss scale factor in Weight. . Change the WeightSpec to incluide the loss factor . Add LossScaleForMixed Property as an layer common property, so that it can set the scale factor in initContext. . Add Loss Scale in initContext . Set the LossScaleForMixed Property when there is LossScale Model Property Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This PR split the Variable and Gradient Dim in Var_Grad and Weight. By this way we can set different Variable Type and Gradient in Wegiht. . add dim_g for gradient in WeightSpec. . manager need to update to support WeightSpec. . Create Tensors according to dim_v and dim_g . Create Weight chaged in Weight.h Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
jijoongmoon
requested review from
myungjoo,
again4you,
jaeyun-jung,
leemgs,
wooksong,
helloahn,
kparichay,
gichan-jang,
anyj0527,
zhoonit,
lhs8928,
songgot and
jihochu
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2562. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/. |
jijoongmoon
requested review from
DonghakPark,
SeoHyungjun,
baek2sm,
skykongkong8,
djeong20,
EunjuYang and
a team
as code owners
taos-ci
approved these changes
Apr 30, 2024
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
EunjuYang
reviewed
May 2, 2024
nntrainer/layers/layer_context.h
Outdated
| @@ -171,7 +172,8 @@ class InitLayerContext { | |||
| /** | |||
| * @brief Request a new weight for the layer | |||
| * | |||
| * @param dim dimension of the weight | |||
| * @param dim_v dimension of Variagble of the weight | |||
There was a problem hiding this comment.
Suggested change
| * @param dim_v dimension of Variagble of the weight | |
| * @param dim_v dimension of Variable of the weight |
Also, @param dim_v doesn't seem to match the function prototype.
There was a problem hiding this comment.
yes. My bad. Fixed. Thanks!
Loss Scale is more like Rigid Property of model, rather than flexible property. Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
jijoongmoon
force-pushed
the
add_loss_scale_in_weight
branch
from
8bb54f3 to
cb0f83f
Compare
taos-ci
approved these changes
May 2, 2024
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
myungjoo
approved these changes
May 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR
This PR enables separate definition of variable and gradient of Weight. It's because we have to support different type considering Mixed Precision Training or Quantization.
For example, Variable should be FP16 and Gradient should be FP16 for FP16-FP16 Mixed Precision.
Or Variable should support BCQ3 and Gradient FP16 for BCQ3-FP16 QAT.
Therefore, we need to set update the Variable weight as it is requested.. and for Gradient, we have to set same type of Activation.
This PR enables to set different type between Variable and Gradient.
Commits to be reviewed in this PR
[Property] Add loss scale property
It add loss scale property as model common property.
Signed-off-by: Jiho Chu [email protected]
[ Weight ] Add Loss Scale factor in Weight
This PR enables the loss scale factor in Weight.
. Change the WeightSpec to incluide the loss factor
. Add LossScaleForMixed Property as an layer common property, so that
it can set the scale factor in initContext.
. Add Loss Scale in initContext
. Set the LossScaleForMixed Property when there is LossScale Model
Property
Resolves:
Self evaluation:
Signed-off-by: jijoong.moon [email protected]
[ Weight ] split variable dim and grad dim to set separately
This PR split the Variable and Gradient Dim in Var_Grad and Weight. By this way we can set different Variable Type and Gradient in Wegiht. . add dim_g for gradient in WeightSpec. . manager need to update to support WeightSpec. . Create Tensors according to dim_v and dim_g . Create Weight chaged in Weight.h
Resolves:
Self evaluation:
Signed-off-by: jijoong.moon [email protected]
[ NEURALNET ] change the loss scale property to Rigid Property
Loss Scale is more like Rigid Property of model, rather than flexible property.
Resolves:
Self evaluation:
Signed-off-by: jijoong.moon [email protected]