[onert] Apply ReLU6Grad to ElementwiseActivationLayer #12487

zetwhite · 2024-01-17T05:35:05Z

This PR applies ReLU6Grad to ElementwiseActivationLayer.

ONE-DCO-1.0-Signed-off-by: SeungHui Youn [email protected]

zetwhite · 2024-01-17T05:41:34Z

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

+          auto relu_cker = [&alpha]() {
+            if (alpha == std::numeric_limits<float>::infinity())
+              return nnfw::cker::train::ReLUGrad;
+            else if (alpha == 6.0f)
+              return nnfw::cker::train::ReLU6Grad;
+            else
+              throw std::runtime_error{"no supported relu kernel"};
+          }();


(note) select kernel by alpha value

zetwhite · 2024-01-17T05:45:33Z

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

          };
        }
        else
        {
          throw std::runtime_error("train ElementwiseActivationLayer : This layer does not "
-                                   "suppport other ReLU except for ReLU(0-inf)");
+                                   "suppport other ReLU except for ReLU(0-inf) and ReLU6(0-6)");
        }


NOTE

After This PR, the training feature supports (not fused) ReLU6.

I also checked loss value isn't different with tensorflow.

tested model :

saved_model and corresponded circle file : 20240117_1354.zip

TensorFlow

Epoch 1/5 50/50 [==============================] - 0s 617us/step - loss: 0.0794 - mean_squared_error: 0.0794 Epoch 2/5 50/50 [==============================] - 0s 542us/step - loss: 0.0586 - mean_squared_error: 0.0586 Epoch 3/5 50/50 [==============================] - 0s 533us/step - loss: 0.0505 - mean_squared_error: 0.0505 Epoch 4/5 50/50 [==============================] - 0s 542us/step - loss: 0.0461 - mean_squared_error: 0.0461 Epoch 5/5 50/50 [==============================] - 0s 513us/step - loss: 0.0431 - mean_squared_error: 0.0431

onert_train

- learning_rate = 0.001 - batch_size = 20 - loss_info = {loss = mean squared error, reduction = sum over batch size} - optimizer = adam ======================== Epoch 1/5 - time: 3.351ms/step - loss: [0] 0.0794 Epoch 2/5 - time: 3.375ms/step - loss: [0] 0.0586 Epoch 3/5 - time: 3.409ms/step - loss: [0] 0.0505 Epoch 4/5 - time: 3.655ms/step - loss: [0] 0.0461 Epoch 5/5 - time: 3.915ms/step - loss: [0] 0.0431 ===================================

zetwhite · 2024-01-17T05:56:06Z

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

+          _backward_kernel = [relu_cker](const IPortableTensor *output,
+                                         const IPortableTensor *incoming,
+                                         IPortableTensor *outgoing) {
+            relu_cker(getShape(output), getBuffer<float>(output), getShape(incoming),
+                      getBuffer<float>(incoming), getShape(outgoing), getBuffer<float>(outgoing));
          };


(note) create _backend_kernel, by capturing selected relu_cker

ragmani

LGTM

glistening · 2024-01-17T06:46:58Z

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

          };
        }
        else
        {
          throw std::runtime_error("train ElementwiseActivationLayer : This layer does not "
-                                   "suppport other ReLU except for ReLU(0-inf)");
+                                   "suppport other ReLU except for ReLU(0-inf) and ReLU6(0-6)");


(optional) It would be good not to list the supported ReLU types here. We need to update the message whenever supported list is updated. What about something like "Not supported ReLU type activation".

Looks better!
updated :)

glistening

LGTM

jyoungyun

LGTM

This PR applies ReLU6Grad to ElementwiseActivationLayer. ONE-DCO-1.0-Signed-off-by: SeungHui Youn <[email protected]>

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

zetwhite · 2024-01-17T08:23:26Z

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

+              return nnfw::cker::train::ReLU6Grad;
+            else
+              throw std::runtime_error{"no supported relu kernel"};
+          }();


Suggested change

}();

}();

lambda is invoked here.
and it returns a function pointer to relu_cker.

jyoungyun

LGTM

YongseopKim · 2024-01-18T00:50:07Z

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc

+          auto relu_cker = [&alpha]() {
+            if (alpha == std::numeric_limits<float>::infinity())
+              return nnfw::cker::train::ReLUGrad;
+            else if (alpha == 6.0f)


Q. Is it okay to compare float value with float value? I'm not sure....

Could you explain a little more?

At first glance, I failed to catch your worries.
I thought It is natural to compare one float with another.

Generally not, but some range of integer is exactly represented in float format.

For float, it has 24-bit significand. 6 is exactly representalbe.

To make sure,

https://en.wikipedia.org/wiki/Single-precision_floating-point_format

Integers between 0 and 16777216 can be exactly represented

Sorry for not enough explaination 😥

I'm asking about approximate equality

Q.

(0.0f == 0.0f) // true or false?

Almost it returns true but rarely could return false. If you already know this and then code it, it would be okay. But if not, we could consider it. 😀

As far as I know, (6.0f == 6.0f) is always true. They have the same precision since they are float literal.
But when one of them is a double or a long double like (6.0f == 6.0), I'm not sure if they have the same precision.
So I tested it.

#include <iomanip> #include <iostream> #include <limits> #include <typeinfo> #define OUT(x) '\n' << std::setw(16) << #x << x int main() { std::cout << "Literal" "\t" "Printed value" << std::left << std::setprecision(39) << OUT( 3.4028234e38f ) // float << OUT( 3.4028234e38 ) // double << OUT( 3.4028234e38l ) // long double << OUT( 6.0f ) // float << OUT( 6.0 ) // double << OUT( 6.0l ) // long double << '\n'; }

Literal Printed value 3.4028234e38f 340282346638528859811704183484516925440 3.4028234e38 340282339999999992395853996843190976512 3.4028234e38l 340282339999999999995912555211526242304 6.0f 6 6.0 6 6.0l 6

It seems that both exponent value and fraction value of 6.0 and 6.0f are the same.

Dump
6.0f : 0xc0c00000
6.0 : 0xc018000000000000

Hmm... maybe your concern is 6 may be represented in different way for lhs and rhs. Is it right?

To clarify, not concern but curious.

And my point is comparing mantissa of floating value and mantissa of another floating value.

Hmm... maybe your concern is 6 may be represented in different way for lhs and rhs. Is it right?

To clarify, not concern but curious.
And my point is comparing mantissa of floating value and mantissa of another floating value.

I updated about it "It seems that both exponent value and fraction value of 6.0 and 6.0f are the same" in https://github.com/Samsung/ONE/pull/12487#discussion_r1456812278. In short, if comparing value is 6.0, there is no problem.
Is your curiousity resolved?

And I already told the term approximate equality.

if we have a consensus, because of value 6, it is okay.

I was curious what sth happens if the floating point's precision is too big.

https://beta.boost.org/doc/libs/1_68_0/libs/math/doc/html/math_toolkit/float_comparison.html

My comment is not pointing out that this is a problem.

glistening

LGTM

zetwhite force-pushed the 0117/relu6-ewa branch from 45a1b8b to e74df47 Compare January 17, 2024 05:36

zetwhite commented Jan 17, 2024

View reviewed changes

zetwhite requested a review from ragmani January 17, 2024 05:56

zetwhite added approval: 2 Require at least 2 approvals PR/ready for review It is ready to review. Please review it. labels Jan 17, 2024

zetwhite requested a review from a team January 17, 2024 05:56

ragmani previously approved these changes Jan 17, 2024

View reviewed changes

zetwhite requested a review from a team January 17, 2024 06:20

glistening reviewed Jan 17, 2024

View reviewed changes

glistening previously approved these changes Jan 17, 2024

View reviewed changes

jyoungyun previously approved these changes Jan 17, 2024

View reviewed changes

zetwhite dismissed stale reviews from jyoungyun, glistening, and ragmani via ddf3a09 January 17, 2024 07:33

zetwhite requested review from jyoungyun, glistening and ragmani January 17, 2024 07:34

ragmani previously approved these changes Jan 17, 2024

View reviewed changes

zetwhite dismissed ragmani’s stale review via f77ea01 January 17, 2024 08:02

zetwhite force-pushed the 0117/relu6-ewa branch from ddf3a09 to f77ea01 Compare January 17, 2024 08:02

[onert] Apply ReLU6Grad to ElementwiseActivationLayer

f2a8cf0

This PR applies ReLU6Grad to ElementwiseActivationLayer. ONE-DCO-1.0-Signed-off-by: SeungHui Youn <[email protected]>

zetwhite force-pushed the 0117/relu6-ewa branch from f77ea01 to f2a8cf0 Compare January 17, 2024 08:06

ragmani reviewed Jan 17, 2024

View reviewed changes

runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc Show resolved Hide resolved

zetwhite commented Jan 17, 2024

View reviewed changes

ragmani approved these changes Jan 17, 2024

View reviewed changes

jyoungyun approved these changes Jan 17, 2024

View reviewed changes

YongseopKim reviewed Jan 18, 2024

View reviewed changes

glistening approved these changes Jan 18, 2024

View reviewed changes

glistening merged commit 908a3be into Samsung:master Jan 18, 2024
3 checks passed

zetwhite mentioned this pull request Jan 18, 2024

[onert] Support ReLU6 for training #12388

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[onert] Apply ReLU6Grad to ElementwiseActivationLayer #12487

[onert] Apply ReLU6Grad to ElementwiseActivationLayer #12487

zetwhite commented Jan 17, 2024 •

edited

Loading

zetwhite Jan 17, 2024

zetwhite Jan 17, 2024 •

edited

Loading

zetwhite Jan 17, 2024

ragmani left a comment

glistening Jan 17, 2024

zetwhite Jan 17, 2024

glistening left a comment

jyoungyun left a comment

zetwhite Jan 17, 2024

jyoungyun left a comment

YongseopKim Jan 18, 2024

zetwhite Jan 18, 2024

glistening Jan 18, 2024

YongseopKim Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading

ragmani Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading

ragmani Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024

glistening left a comment

[onert] Apply ReLU6Grad to ElementwiseActivationLayer #12487

[onert] Apply ReLU6Grad to ElementwiseActivationLayer #12487

Conversation

zetwhite commented Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

zetwhite Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

NOTE

Choose a reason for hiding this comment

ragmani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glistening left a comment

Choose a reason for hiding this comment

jyoungyun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyoungyun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YongseopKim Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

YongseopKim Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

ragmani Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

YongseopKim Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

ragmani Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

YongseopKim Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glistening left a comment

Choose a reason for hiding this comment

zetwhite commented Jan 17, 2024 •

edited

Loading

zetwhite Jan 17, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading

ragmani Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading

ragmani Jan 18, 2024 •

edited

Loading

YongseopKim Jan 18, 2024 •

edited

Loading