[onert] Introduce Pad op for training

[YongseopKim]

Parent issue: #12325

Let's introduce Pad op for training.

[YongseopKim]

transpose -> Pad

ONERT_LOG_ENABLE_1

[   GraphDumper  ] {
[   GraphDumper  ]     %3 =   @0_Conv2D(  %0,  %1,  %2)
[   GraphDumper  ]     %6 =   @1_DepthwiseConv2D(  %3,  %4,  %5)
[   GraphDumper  ]     %9 =   @2_Conv2D(  %6,  %7,  %8)
[   GraphDumper  ]    %12 =   @3_Conv2D(  %9, %10, %11)
[   GraphDumper  ]    %14 =   @4_Pad( %12, %13)
[   GraphDumper  ]    %17 =   @5_DepthwiseConv2D( %14, %15, %16)
[   GraphDumper  ]    %20 =   @6_Conv2D( %17, %18, %19)
[   GraphDumper  ]    %23 =   @7_Conv2D( %20, %21, %22)
[   GraphDumper  ]    %26 =   @8_DepthwiseConv2D( %23, %24, %25)
[   GraphDumper  ]    %29 =   @9_Conv2D( %26, %27, %28)
[   GraphDumper  ]    %30 =  @10_Add( %20, %29)
[   GraphDumper  ]    %33 =  @11_Conv2D( %30, %31, %32)
[   GraphDumper  ]    %34 =  @12_Pad( %33, %13)
[   GraphDumper  ]    %37 =  @13_DepthwiseConv2D( %34, %35, %36)
[   GraphDumper  ]    %40 =  @14_Conv2D( %37, %38, %39)
[   GraphDumper  ]    %43 =  @15_Conv2D( %40, %41, %42)
[   GraphDumper  ]    %46 =  @16_DepthwiseConv2D( %43, %44, %45)
[   GraphDumper  ]    %49 =  @17_Conv2D( %46, %47, %48)
[   GraphDumper  ]    %50 =  @18_Add( %40, %49)
[   GraphDumper  ]    %53 =  @19_Conv2D( %50, %51, %52)
[   GraphDumper  ]    %56 =  @20_DepthwiseConv2D( %53, %54, %55)
[   GraphDumper  ]    %59 =  @21_Conv2D( %56, %57, %58)
[   GraphDumper  ]    %60 =  @22_Add( %50, %59)
[   GraphDumper  ]    %63 =  @23_Conv2D( %60, %61, %62)
[   GraphDumper  ]    %64 =  @24_Pad( %63, %13)
[   GraphDumper  ]    %67 =  @25_DepthwiseConv2D( %64, %65, %66)
[   GraphDumper  ]    %70 =  @26_Conv2D( %67, %68, %69)
[   GraphDumper  ]    %73 =  @27_Conv2D( %70, %71, %72)
[   GraphDumper  ]    %76 =  @28_DepthwiseConv2D( %73, %74, %75)
[   GraphDumper  ]    %79 =  @29_Conv2D( %76, %77, %78)
[   GraphDumper  ]    %80 =  @30_Add( %70, %79)
[   GraphDumper  ]    %83 =  @31_Conv2D( %80, %81, %82)
[   GraphDumper  ]    %86 =  @32_DepthwiseConv2D( %83, %84, %85)
[   GraphDumper  ]    %89 =  @33_Conv2D( %86, %87, %88)
[   GraphDumper  ]    %90 =  @34_Add( %80, %89)
[   GraphDumper  ]    %93 =  @35_Conv2D( %90, %91, %92)
[   GraphDumper  ]    %96 =  @36_DepthwiseConv2D( %93, %94, %95)
[   GraphDumper  ]    %99 =  @37_Conv2D( %96, %97, %98)
[   GraphDumper  ]   %100 =  @38_Add( %90, %99)
[   GraphDumper  ]   %103 =  @39_Conv2D(%100,%101,%102)
[   GraphDumper  ]   %106 =  @40_DepthwiseConv2D(%103,%104,%105)
[   GraphDumper  ]   %109 =  @41_Conv2D(%106,%107,%108)
[   GraphDumper  ]   %112 =  @42_Conv2D(%109,%110,%111)
[   GraphDumper  ]   %115 =  @43_DepthwiseConv2D(%112,%113,%114)
[   GraphDumper  ]   %118 =  @44_Conv2D(%115,%116,%117)
[   GraphDumper  ]   %119 =  @45_Add(%109,%118)
[   GraphDumper  ]   %122 =  @46_Conv2D(%119,%120,%121)
[   GraphDumper  ]   %125 =  @47_DepthwiseConv2D(%122,%123,%124)
[   GraphDumper  ]   %128 =  @48_Conv2D(%125,%126,%127)
[   GraphDumper  ]   %129 =  @49_Add(%119,%128)
[   GraphDumper  ]   %132 =  @50_Conv2D(%129,%130,%131)
[   GraphDumper  ]   %133 =  @51_Pad(%132, %13)
[   GraphDumper  ]   %136 =  @52_DepthwiseConv2D(%133,%134,%135)
[   GraphDumper  ]   %139 =  @53_Conv2D(%136,%137,%138)
[   GraphDumper  ]   %142 =  @54_Conv2D(%139,%140,%141)
[   GraphDumper  ]   %145 =  @55_DepthwiseConv2D(%142,%143,%144)
[   GraphDumper  ]   %148 =  @56_Conv2D(%145,%146,%147)
[   GraphDumper  ]   %149 =  @57_Add(%139,%148)
[   GraphDumper  ]   %152 =  @58_Conv2D(%149,%150,%151)
[   GraphDumper  ]   %155 =  @59_DepthwiseConv2D(%152,%153,%154)
[   GraphDumper  ]   %158 =  @60_Conv2D(%155,%156,%157)
[   GraphDumper  ]   %159 =  @61_Add(%149,%158)
[   GraphDumper  ]   %162 =  @62_Conv2D(%159,%160,%161)
[   GraphDumper  ]   %165 =  @63_DepthwiseConv2D(%162,%163,%164)
[   GraphDumper  ]   %168 =  @64_Conv2D(%165,%166,%167)
[   GraphDumper  ]   %171 =  @65_Conv2D(%168,%169,%170)
[   GraphDumper  ]   %173 =  @66_ReduceMean(%171,%172)
[   GraphDumper  ]   %176 =  @67_FullyConnected(%173,%174,%175)
[   GraphDumper  ]   %177 =  @68_Softmax(%176)

circledump

Dump: /home/dragon/github/YongseopKim/ONE/mobilenetv2.opt.circle

===================================================================
Model version: 0
 # sub graphs: 1

Operator Codes: [order] OpCodeName (OpCode Enum)
[0] CONV_2D (code: 3, dep_code: 3, version: 1)
[1] DEPTHWISE_CONV_2D (code: 4, dep_code: 4, version: 1)
[2] PAD (code: 34, dep_code: 34, version: 1)
[3] ADD (code: 0, dep_code: 0, version: 1)
[4] MEAN (code: 40, dep_code: 40, version: 1)
[5] FULLY_CONNECTED (code: 9, dep_code: 9, version: 1)
[6] SOFTMAX (code: 25, dep_code: 25, version: 1)

...

[YongseopKim]

While drafting, I change my mind to begin #12473 than this task.

[YongseopKim]

While #12473 , I'm drafting #12499 .

[YongseopKim]

#12499 (comment)

While drafting, I realized that there are ops which don't have to do backwards.

In the case of Pad, AFAIK, pad does nothing at the backward propagation. (because padding doesn't have any params to be tranied.)

Therefore, these ops that have nothing to do at the backward propagation can be extracted as a type such as InterfaceDoNothingAtTheBackwardProgration. (I don't know how to name it...)

FYI

[jyoungyun]

#12499 (comment)

In the case of Pad, AFAIK, pad does nothing at the backward propagation. (because padding doesn't have any params to be tranied.)

Even though there is no weight in Pad Op, it should implement backwarding function for backpropagation.

[YongseopKim]

@jyoungyun yes, therefore, I could say like below

Therefore, these ops that have nothing to do at the backward propagation can be extracted as a type such as InterfaceDoNothingAtTheBackwardProgration. (I don't know how to name it...)

[YongseopKim]

I discussed with @jyoungyun , and I finalize that these impl should be done and making pattern(like NotImplementedYetOperation) is not much useful.

[YongseopKim]

Done