Draft: [onert] support ReLU6 training #12395
Conversation
| grad_map.array() = | ||
| incoming_map.array() * | ||
| (0.0f < output_map.array() && output_map.array() < 6.0f).template cast<float>(); |
There was a problem hiding this comment.
(note)
array() has no cost - it just change the view of data, for element-wise operation
| const IPortableTensor *getFusedActivationBackprop(const ir::Activation &activation, | ||
| const IPortableTensor *output, | ||
| const IPortableTensor *input_backprop, | ||
| IPortableTensor *output_backprop) | ||
| { | ||
| const IPortableTensor *res; | ||
|
|
||
| switch (activation) | ||
| { | ||
| case ir::Activation::NONE: | ||
| res = input_backprop; | ||
| break; | ||
| case ir::Activation::RELU: | ||
| nnfw::cker::train::ReLUGrad(getShape(output), getBuffer<float>(output), | ||
| getShape(input_backprop), getBuffer<float>(input_backprop), | ||
| getShape(output_backprop), getBuffer<float>(output_backprop)); | ||
| res = output_backprop; | ||
| break; | ||
| case ir::Activation::RELU6: | ||
| nnfw::cker::train::ReLU6Grad(getShape(output), getBuffer<float>(output), | ||
| getShape(input_backprop), getBuffer<float>(input_backprop), | ||
| getShape(output_backprop), getBuffer<float>(output_backprop)); | ||
| res = output_backprop; | ||
| break; | ||
| default: | ||
| throw std::runtime_error("Unsupported activation type yet"); | ||
| } | ||
|
|
||
| return res; | ||
| } |
There was a problem hiding this comment.
While I was supporting relu6,
I found out that each layer(Convolution, FullyConnected and Pool for now) implements activation backpropagation on its own.
I'd like to extract fused activation's backpropagation part as OperationUtil to remove code duplication.
But before going on, I'd like to hear your opinion. Do you think it is fine?
( I'm a bit afraid that there's a reason it has the current implementation. )
There was a problem hiding this comment.
Sorry for the late reply. Perhaps, I missed your comment. It looks good to me.
There was a problem hiding this comment.
No problem!
I didn't ask again, because it wasn't in a hurry.
This draft is to support ReLU6 training feature. - Add ReLU6Grad cker - Extract fused activation gradient calculation part to reuse ONE-DCO-1.0-Signed-off-by: SeungHui Youn <[email protected]>
zetwhite
added
the
DRAFT
|
related code is all merged. so close this Draft :) |
This draft is to support ReLU6 training feature.
ONE-DCO-1.0-Signed-off-by: SeungHui Youn [email protected]
related : #12388