about nvdla small interrupt configuration

[akioolin]

Hi All,

right now, we are tesing on nvdla small configuration. fpga baord is zuc102, sdk is 2018.1, os builder is petalinux. after proper modification of kmd/umd source code, the nvdla_runtime could execute flatbufs tests amples successfully except CONV testbench. what we found is the group 0's interrupt could be received but group 1 is no interrupt response. the following is our interrupt setting in device tree (pl.dtsi) for nvdla small configuration.

nv_small: nv_small_0@b0000000 {
compatible = "xlnx,nv_small";
interrupt-parent = <&axi_intc_0>;
interrupts = <10 2>;
reg = <0x0 0xb0000000 0x0 0x10000>;
};

compare some setting in hw issues lists, here is the respect part which provided by qdchau and mmaciag.

nvdla @a0000000 {
compatible = "nvidia,nvdla_os_initial";
interrupt-parent = <0x4>;
interrupts = <0x0 0x59 0x4 0x0 0x59 0x4>;
reg = <0x0 0xa0000000 0x0 0x40000>;
};

and

u_nvdla: FPI_NVDLA_wrapper@80000000 {
compatible = "xlnx,FPI-NVDLA-wrapper-1.1";
interrupt-parent = <&gic>;
interrupts = <0 89 4>;
reg = <0x0 0x80000000 0x0 0x10000>;
};

where do we missed in interrupt setting in device tree?

Best Regards,
Akio

[ghost]

== Tests for nv_small ==

= Run PDP/PDP_L0_0_small_fbuf
creating new runtime context...
Emulator starting
submitting tasks...
[ 2203.960987] Enter:dla_read_network_config
[ 2203.967666] Exit:dla_read_network_config status=0
[ 2203.973362] Enter: dla_initiate_processors
[ 2203.978475] Enter: dla_submit_operation
[ 2203.983334] Prepare PDP operation index 0 ROI 0 dep_count 0
[ 2203.989980] Enter: dla_prepare_operation
[ 2203.994993] processor:PDP group:0, rdma_group:0 available
[ 2204.001528] Enter: dla_read_config
[ 2204.006088] Exit: dla_read_config
[ 2204.010576] Exit: dla_prepare_operation status=0
[ 2204.016385] Enter: dla_program_operation
[ 2204.021490] Program PDP operation index 0 ROI 0 Group[0]
[ 2204.027968] group id 0 rdma id 0
[ 2204.032353] no desc get due to index==-1
[ 2204.037444] no desc get due to index==-1
[ 2204.042539] no desc get due to index==-1
[ 2204.047628] no desc get due to index==-1
[ 2204.052724] no desc get due to index==-1
[ 2204.057824] no desc get due to index==-1
[ 2204.062928] no desc get due to index==-1
[ 2204.068025] Enter: dla_op_programmed
[ 2204.072789] Exit: dla_op_programmed
[ 2204.077466] Exit: dla_program_operation status=0
[ 2204.083300] Enter: dla_enable_operation
[ 2204.088345] Enable PDP operation index 0 ROI 0
[ 2204.094011] group id 0 rdma id 0
[ 2204.098465] rdma needed 1
[ 2204.102303] Enter: dla_op_enabled
[ 2204.106852] Exit: dla_op_enabled
[ 2204.111328] Exit: dla_enable_operation status=0
[ 2204.117120] Exit: dla_submit_operation
[ 2204.122117] Enter: dla_dequeue_operation
[ 2204.127298] exit PDP as there's no further operation
[ 2204.133541] Exit: dla_dequeue_operation
[ 2204.138656] Exit: dla_initiate_processors status=0
[ 2204.144736] Enter:dla_handle_events, processor:BDMA
[ 2204.150906] Exit:dla_handle_events, ret:0
[ 2204.156214] Enter:dla_handle_events, processor:Convolution
[ 2204.163018] Exit:dla_handle_events, ret:0
[ 2204.168345] Enter:dla_handle_events, processor:SDP
[ 2204.174456] Exit:dla_handle_events, ret:0
[ 2204.179780] Enter:dla_handle_events, processor:PDP
[ 2204.185895] Handle op complete event, processor PDP group 0
[ 2204.192810] Enter:dla_op_completion processor PDP group0
[ 2204.199460] Completed PDP operation index 0 ROI 0
[ 2204.205495] 1 HWLs done, totally 1 layers
[ 2204.210842] Enter: dla_free_op_desc op desc index 0 ROI 0
[ 2204.217604] Exit: dla_free_op_desc
[ 2204.222317] Exit:dla_op_completion processor PDP group0 status=0
[ 2204.229622] Exit:dla_handle_events, ret:0
[ 2204.234921] Enter:dla_handle_events, processor:CDP
[ 2204.241019] Exit:dla_handle_events, ret:0
[ 2204.246320] Enter:dla_handle_events, processor:RUBIK
[ 2204.252580] Exit:dla_handle_events, ret:0
[ 2204.257885] reset engine done
Shutdown signal received, exiting
Test pass
= Run CONV/CONV_D_L0_0_small_fbuf
creating new runtime context...
Emulator starting
submitting tasks...
[ 2204.972905] Enter:dla_read_network_config
[ 2204.979956] Exit:dla_read_network_config status=0
[ 2204.985960] Enter: dla_initiate_processors
[ 2204.991362] Enter: dla_submit_operation
[ 2204.996483] Prepare Convolution operation index 0 ROI 0 dep_count 1
[ 2205.004059] Enter: dla_prepare_operation
[ 2205.009298] processor:Convolution group:0, rdma_group:0 available
[ 2205.016755] Enter: dla_read_config
[ 2205.021543] Exit: dla_read_config
[ 2205.026252] Exit: dla_prepare_operation status=0
[ 2205.032293] Enter: dla_program_operation
[ 2205.037660] Program Convolution operation index 0 ROI 0 Group[0]
[ 2205.045189] no desc get due to index==-1
[ 2205.050614] no desc get due to index==-1
[ 2205.056025] no desc get due to index==-1
[ 2205.061410] no desc get due to index==-1
[ 2205.066776] no desc get due to index==-1
[ 2205.072123] no desc get due to index==-1
[ 2205.077441] Enter: dla_op_programmed
[ 2205.082420] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[ 2205.089955] enable SDP in dla_update_dependency as depdency are resolved
[ 2205.098134] Enter: dla_enable_operation
[ 2205.103446] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[ 2205.114066] Exit: dla_enable_operation status=0
[ 2205.120169] Exit: dla_op_programmed
[ 2205.125213] Exit: dla_program_operation status=0
[ 2205.131382] Exit: dla_submit_operation
[ 2205.136679] Enter: dla_dequeue_operation
[ 2205.142152] exit Convolution as there's no further operation
[ 2205.149378] Exit: dla_dequeue_operation
[ 2205.154765] Enter: dla_submit_operation
[ 2205.160127] Prepare SDP operation index 1 ROI 0 dep_count 0
[ 2205.167248] Enter: dla_prepare_operation
[ 2205.172713] processor:SDP group:0, rdma_group:0 available
[ 2205.179672] Enter: dla_read_config
[ 2205.184617] Exit: dla_read_config
[ 2205.189457] Exit: dla_prepare_operation status=0
[ 2205.195604] Enter: dla_program_operation
[ 2205.201044] Program SDP operation index 1 ROI 0 Group[0]
[ 2205.207893] no desc get due to index==-1
[ 2205.213331] no desc get due to index==-1
[ 2205.218741] no desc get due to index==-1
[ 2205.224111] no desc get due to index==-1
[ 2205.229459] no desc get due to index==-1
[ 2205.234777] no desc get due to index==-1
[ 2205.240067] Enter: dla_op_programmed
[ 2205.245002] Exit: dla_op_programmed
[ 2205.249840] Exit: dla_program_operation status=0
[ 2205.255817] Enter: dla_enable_operation
[ 2205.261000] Enable SDP operation index 1 ROI 0
[ 2205.266789] Enter: dla_op_enabled
[ 2205.271449] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[ 2205.278915] enable Convolution in dla_update_dependency as depdency are resolved
[ 2205.287701] Enter: dla_enable_operation
[ 2205.292905] Enable Convolution operation index 0 ROI 0
[ 2205.299410] Enter: dla_op_enabled
[ 2205.304091] Exit: dla_op_enabled
[ 2205.308679] Exit: dla_enable_operation status=0
[ 2205.314579] Exit: dla_op_enabled
[ 2205.319140] Exit: dla_enable_operation status=0
[ 2205.324998] Exit: dla_submit_operation
[ 2205.330071] Enter: dla_dequeue_operation
[ 2205.335348] exit SDP as there's no further operation
[ 2205.341703] Exit: dla_dequeue_operation
[ 2205.346946] Exit: dla_initiate_processors status=0
[ 2205.353176] Enter:dla_handle_events, processor:BDMA
[ 2205.359497] Exit:dla_handle_events, ret:0
[ 2205.364928] Enter:dla_handle_events, processor:Convolution
[ 2205.371832] Handle cdma weight done event, processor Convolution group 0
[ 2205.379972] Handle cdma data done event, processor Convolution group 0
[ 2205.387900] Handle op complete event, processor Convolution group 0
[ 2205.395541] Enter:dla_op_completion processor Convolution group0
[ 2205.402946] Completed Convolution operation index 0 ROI 0
[ 2205.409760] 1 HWLs done, totally 2 layers
[ 2205.415184] Enter: dla_dequeue_operation
[ 2205.420521] exit Convolution as there's no further operation
[ 2205.427602] Exit: dla_dequeue_operation
[ 2205.432852] Exit:dla_op_completion processor Convolution group0 status=0
[ 2205.441019] Exit:dla_handle_events, ret:0
[ 2205.446477] Enter:dla_handle_events, processor:SDP
[ 2205.452709] Handle op complete event, processor SDP group 0
[ 2205.459728] Enter:dla_op_completion processor SDP group0
[ 2205.466488] Completed SDP operation index 1 ROI 0
[ 2205.472644] 2 HWLs done, totally 2 layers
[ 2205.478115] Enter: dla_free_op_desc op desc index 0 ROI 0
[ 2205.484995] Exit: dla_free_op_desc
[ 2205.489851] Enter: dla_free_op_desc op desc index 1 ROI 0
[ 2205.496730] Exit: dla_free_op_desc
[ 2205.501621] Exit:dla_op_completion processor SDP group0 status=0
[ 2205.509181] Exit:dla_handle_events, ret:0
[ 2205.514755] Enter:dla_handle_events, processor:PDP
[ 2205.521145] Exit:dla_handle_events, ret:0
[ 2205.526770] Enter:dla_handle_events, processor:CDP
[ 2205.533193] Exit:dla_handle_events, ret:0
[ 2205.538838] Enter:dla_handle_events, processor:RUBIK
[ 2205.545453] Exit:dla_handle_events, ret:0
[ 2205.551129] reset engine done
Shutdown signal received, exiting
Test pass
= Run SDP/SDP_X1_L0_0_small_fbuf
creating new runtime context...
Emulator starting
submitting tasks...
[ 2205.982808] Enter:dla_read_network_config
[ 2205.990245] Exit:dla_read_network_config status=0
[ 2205.996613] Enter: dla_initiate_processors
[ 2206.002346] Enter: dla_submit_operation
[ 2206.007775] Prepare SDP operation index 0 ROI 0 dep_count 0
[ 2206.014970] Enter: dla_prepare_operation
[ 2206.020497] processor:SDP group:1, rdma_group:0 available
[ 2206.027530] Enter: dla_read_config
[ 2206.032567] Exit: dla_read_config
[ 2206.037508] Exit: dla_prepare_operation status=0
[ 2206.043769] Enter: dla_program_operation
[ 2206.049343] Program SDP operation index 0 ROI 0 Group[1]
[ 2206.056343] no desc get due to index==-1
[ 2206.061925] no desc get due to index==-1
[ 2206.067478] no desc get due to index==-1
[ 2206.073011] no desc get due to index==-1
[ 2206.078522] no desc get due to index==-1
[ 2206.084002] no desc get due to index==-1
[ 2206.089461] no desc get due to index==-1
[ 2206.094862] Enter: dla_op_programmed
[ 2206.099906] Exit: dla_op_programmed
[ 2206.104843] Exit: dla_program_operation status=0
[ 2206.110894] Enter: dla_enable_operation
[ 2206.116142] Enable SDP operation index 0 ROI 0
[ 2206.121991] Enter: dla_op_enabled
[ 2206.126707] Exit: dla_op_enabled
[ 2206.131315] Exit: dla_enable_operation status=0
[ 2206.137221] Exit: dla_submit_operation
[ 2206.142306] Enter: dla_dequeue_operation
[ 2206.147549] exit SDP as there's no further operation
[ 2206.153846] Exit: dla_dequeue_operation
[ 2206.159005] Exit: dla_initiate_processors status=0
[ 2206.165112] Enter:dla_handle_events, processor:BDMA
[ 2206.171292] Exit:dla_handle_events, ret:0
[ 2206.176594] Enter:dla_handle_events, processor:Convolution
[ 2206.183393] Exit:dla_handle_events, ret:0
[ 2206.188700] Enter:dla_handle_events, processor:SDP
[ 2206.194790] Handle op complete event, processor SDP group 1
[ 2206.201672] Enter:dla_op_completion processor SDP group1
[ 2206.208292] Completed SDP operation index 0 ROI 0
[ 2206.214291] 1 HWLs done, totally 1 layers
[ 2206.219588] Enter: dla_free_op_desc op desc index 0 ROI 0
[ 2206.226297] Exit: dla_free_op_desc
[ 2206.230992] Exit:dla_op_completion processor SDP group1 status=0
[ 2206.238319] Exit:dla_handle_events, ret:0
[ 2206.243633] Enter:dla_handle_events, processor:PDP
[ 2206.249734] Exit:dla_handle_events, ret:0
[ 2206.255054] Enter:dla_handle_events, processor:CDP
[ 2206.261168] Exit:dla_handle_events, ret:0
[ 2206.266494] Enter:dla_handle_events, processor:RUBIK
[ 2206.272787] Exit:dla_handle_events, ret:0
[ 2206.278142] reset engine done
Shutdown signal received, exiting
Test pass
= Run CDP/CDP_L0_0_small_fbuf
creating new runtime context...
Emulator starting
submitting tasks...
[ 2206.992531] Enter:dla_read_network_config
[ 2206.999607] Exit:dla_read_network_config status=0
[ 2207.005635] Enter: dla_initiate_processors
[ 2207.011044] Enter: dla_submit_operation
[ 2207.016189] Prepare CDP operation index 0 ROI 0 dep_count 0
[ 2207.023096] Enter: dla_prepare_operation
[ 2207.028344] processor:CDP group:0, rdma_group:0 available
[ 2207.035090] Enter: dla_read_config
[ 2207.039828] Exit: dla_read_config
[ 2207.044490] Exit: dla_prepare_operation status=0
[ 2207.050480] Enter: dla_program_operation
[ 2207.055798] Program CDP operation index 0 ROI 0 Group[0]
[ 2207.062547] Enter: dla_cdp_program
[ 2207.062550] Enter: processor_cdp_program
[ 2207.072897] Exit: processor_cdp_program
[ 2207.072899] Exit: dla_cdp_program
[ 2207.078202] no desc get due to index==-1
[ 2207.088368] no desc get due to index==-1
[ 2207.093737] no desc get due to index==-1
[ 2207.099082] no desc get due to index==-1
[ 2207.104408] no desc get due to index==-1
[ 2207.109715] no desc get due to index==-1
[ 2207.114999] no desc get due to index==-1
[ 2207.120285] Enter: dla_op_programmed
[ 2207.125218] Exit: dla_op_programmed
[ 2207.130051] Exit: dla_program_operation status=0
[ 2207.136011] Enter: dla_enable_operation
[ 2207.141183] Enable CDP operation index 0 ROI 0
[ 2207.146963] Enter: dla_cdp_enable
[ 2207.151594] Exit: dla_cdp_enable
[ 2207.156124] Enter: dla_op_enabled
[ 2207.160724] Exit: dla_op_enabled
[ 2207.165223] Exit: dla_enable_operation status=0
[ 2207.171013] Exit: dla_submit_operation
[ 2207.175998] Enter: dla_dequeue_operation
[ 2207.181144] exit CDP as there's no further operation
[ 2207.187347] Exit: dla_dequeue_operation
[ 2207.192408] Exit: dla_initiate_processors status=0
[ 2207.198421] Enter:dla_handle_events, processor:BDMA
[ 2207.204503] Exit:dla_handle_events, ret:0
[ 2207.209712] Enter:dla_handle_events, processor:Convolution
[ 2207.216411] Exit:dla_handle_events, ret:0
[ 2207.221622] Enter:dla_handle_events, processor:SDP
[ 2207.227627] Exit:dla_handle_events, ret:0
[ 2207.232836] Enter:dla_handle_events, processor:PDP
[ 2207.238832] Exit:dla_handle_events, ret:0
[ 2207.244039] Enter:dla_handle_events, processor:CDP
[ 2207.250044] Handle op complete event, processor CDP group 0
[ 2207.256850] Enter:dla_op_completion processor CDP group0
[ 2207.263402] Completed CDP operation index 0 ROI 0
[ 2207.269347] 1 HWLs done, totally 1 layers
[ 2207.274594] Enter: dla_free_op_desc op desc index 0 ROI 0
[ 2207.281245] Exit: dla_free_op_desc
[ 2207.285878] Exit:dla_op_completion processor CDP group0 status=0
[ 2207.293139] Exit:dla_handle_events, ret:0
[ 2207.298381] Enter:dla_handle_events, processor:RUBIK
[ 2207.304601] Exit:dla_handle_events, ret:0
[ 2207.309883] reset engine done
Shutdown signal received, exiting
Test pass
= Run NN/NN_L0_1_small_fbuf
creating new runtime context...
Emulator starting
submitting tasks...
[ 2210.896668] Enter:dla_read_network_config
[ 2210.903672] Exit:dla_read_network_config status=0
[ 2210.909643] Enter: dla_initiate_processors
[ 2210.915004] Enter: dla_submit_operation
[ 2210.920099] Prepare Convolution operation index 0 ROI 0 dep_count 1
[ 2210.927667] Enter: dla_prepare_operation
[ 2210.932896] processor:Convolution group:1, rdma_group:0 available
[ 2210.940346] Enter: dla_read_config
[ 2210.945125] Exit: dla_read_config
[ 2210.949831] Exit: dla_prepare_operation status=0
[ 2210.955870] Enter: dla_program_operation
[ 2210.961236] Program Convolution operation index 0 ROI 0 Group[1]
[ 2210.968764] no desc get due to index==-1
[ 2210.974179] no desc get due to index==-1
[ 2210.979577] no desc get due to index==-1
[ 2210.984954] no desc get due to index==-1
[ 2210.990313] no desc get due to index==-1
[ 2210.995644] Enter: dla_op_programmed
[ 2211.000630] Update dependency operation index 2 ROI 0 DEP_COUNT=2
[ 2211.008178] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[ 2211.015723] enable SDP in dla_update_dependency as depdency are resolved
[ 2211.023905] Enter: dla_enable_operation
[ 2211.029220] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[ 2211.039839] Exit: dla_enable_operation status=0
[ 2211.045950] Exit: dla_op_programmed
[ 2211.050997] Exit: dla_program_operation status=0
[ 2211.057176] Exit: dla_submit_operation
[ 2211.062484] Enter: dla_dequeue_operation
[ 2211.067953] Dequeue op from Convolution processor, index=2 ROI=0
[ 2211.075536] Enter: dla_submit_operation
[ 2211.080938] Prepare Convolution operation index 2 ROI 0 dep_count 1
[ 2211.088792] Enter: dla_prepare_operation
[ 2211.094292] processor:Convolution group:0, rdma_group:0 available
[ 2211.101991] Enter: dla_read_config
[ 2211.106995] Exit: dla_read_config
[ 2211.111900] Exit: dla_prepare_operation status=0
[ 2211.118105] Enter: dla_program_operation
[ 2211.123600] Program Convolution operation index 2 ROI 0 Group[0]
[ 2211.131226] no desc get due to index==-1
[ 2211.136737] no desc get due to index==-1
[ 2211.142218] no desc get due to index==-1
[ 2211.147668] no desc get due to index==-1
[ 2211.153083] no desc get due to index==-1
[ 2211.158471] Enter: dla_op_programmed
[ 2211.163499] Update dependency operation index 6 ROI 0 DEP_COUNT=3
[ 2211.171072] Update dependency operation index 3 ROI 0 DEP_COUNT=2
[ 2211.178616] Exit: dla_op_programmed
[ 2211.183552] Exit: dla_program_operation status=0
[ 2211.189625] Exit: dla_submit_operation
[ 2211.194807] Exit: dla_dequeue_operation
[ 2211.200064] Enter: dla_submit_operation
[ 2211.205327] Prepare SDP operation index 1 ROI 0 dep_count 0
[ 2211.212329] Enter: dla_prepare_operation
[ 2211.217670] processor:SDP group:0, rdma_group:1 available
[ 2211.224484] Enter: dla_read_config
[ 2211.229290] Exit: dla_read_config
[ 2211.233999] Exit: dla_prepare_operation status=0
[ 2211.240023] Enter: dla_program_operation
[ 2211.245346] Program SDP operation index 1 ROI 0 Group[0]
[ 2211.252076] no desc get due to index==-1
[ 2211.257384] no desc get due to index==-1
[ 2211.262694] no desc get due to index==-1
[ 2211.268009] no desc get due to index==-1
[ 2211.273323] Enter: dla_op_programmed
[ 2211.278305] Update dependency operation index 3 ROI 0 DEP_COUNT=1
[ 2211.285843] enable SDP in dla_update_dependency as depdency are resolved
[ 2211.293989] Enter: dla_enable_operation
[ 2211.299287] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[ 2211.309866] Exit: dla_enable_operation status=0
[ 2211.315901] Exit: dla_op_programmed
[ 2211.320850] Exit: dla_program_operation status=0
[ 2211.326893] Enter: dla_enable_operation
[ 2211.332148] Enable SDP operation index 1 ROI 0
[ 2211.338022] Enter: dla_op_enabled
[ 2211.342757] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[ 2211.350310] enable Convolution in dla_update_dependency as depdency are resolved
[ 2211.359193] Enter: dla_enable_operation
[ 2211.364502] Enable Convolution operation index 0 ROI 0
[ 2211.371107] Enter: dla_op_enabled
[ 2211.375865] Exit: dla_op_enabled
[ 2211.380508] Exit: dla_enable_operation status=0
[ 2211.386447] Exit: dla_op_enabled
[ 2211.391083] Exit: dla_enable_operation status=0
[ 2211.397023] Exit: dla_submit_operation
[ 2211.402167] Enter: dla_dequeue_operation
[ 2211.407468] Dequeue op from SDP processor, index=3 ROI=0
[ 2211.414162] Enter: dla_submit_operation
[ 2211.419400] Prepare SDP operation index 3 ROI 0 dep_count 0
[ 2211.426425] Enter: dla_prepare_operation
[ 2211.431824] processor:SDP group:1, rdma_group:0 available
[ 2211.438743] Enter: dla_read_config
[ 2211.443667] Exit: dla_read_config
[ 2211.448470] Exit: dla_prepare_operation status=0
[ 2211.454603] Enter: dla_program_operation
[ 2211.460027] Program SDP operation index 3 ROI 0 Group[1]
[ 2211.466866] no desc get due to index==-1
[ 2211.472307] no desc get due to index==-1
[ 2211.477739] no desc get due to index==-1
[ 2211.483155] no desc get due to index==-1
[ 2211.488544] Enter: dla_op_programmed
[ 2211.493573] Update dependency operation index 7 ROI 0 DEP_COUNT=2
[ 2211.501145] Exit: dla_op_programmed
[ 2211.506089] Exit: dla_program_operation status=0
[ 2211.512179] Enter: dla_enable_operation
[ 2211.517476] Enable SDP operation index 3 ROI 0
[ 2211.523382] Enter: dla_op_enabled
[ 2211.528145] Update dependency operation index 2 ROI 0 DEP_COUNT=1
[ 2211.535711] enable Convolution in dla_update_dependency as depdency are resolved
[ 2211.544641] Enter: dla_enable_operation
[ 2211.550034] Enable Convolution operation index 2 ROI 0
[ 2211.556769] Enter: dla_op_enabled
[ 2211.561671] Exit: dla_op_enabled
[ 2211.566466] Exit: dla_enable_operation status=0
[ 2211.572550] Exit: dla_op_enabled
[ 2211.577296] Exit: dla_enable_operation status=0
[ 2211.583307] Exit: dla_submit_operation
[ 2211.588484] Exit: dla_dequeue_operation
[ 2211.593743] Enter: dla_submit_operation
[ 2211.598996] Prepare PDP operation index 5 ROI 0 dep_count 1
[ 2211.606013] Enter: dla_prepare_operation
[ 2211.611383] processor:PDP group:1, rdma_group:1 available
[ 2211.618250] Enter: dla_read_config
[ 2211.623101] Exit: dla_read_config
[ 2211.627832] Exit: dla_prepare_operation status=0
[ 2211.633842] Enter: dla_program_operation
[ 2211.639146] Program PDP operation index 5 ROI 0 Group[1]
[ 2211.645852] group id 1 rdma id 1
[ 2211.650493] no desc get due to index==-1
[ 2211.655813] no desc get due to index==-1
[ 2211.661119] no desc get due to index==-1
[ 2211.666410] no desc get due to index==-1
[ 2211.671689] no desc get due to index==-1
[ 2211.676943] Enter: dla_op_programmed
[ 2211.681837] Update dependency operation index 11 ROI 0 DEP_COUNT=2
[ 2211.689363] Exit: dla_op_programmed
[ 2211.694182] Exit: dla_program_operation status=0
[ 2211.700123] Exit: dla_submit_operation
[ 2211.705173] Enter: dla_dequeue_operation
[ 2211.710387] Dequeue op from PDP processor, index=11 ROI=0
[ 2211.717109] Enter: dla_submit_operation
[ 2211.722266] Prepare PDP operation index 11 ROI 0 dep_count 1
[ 2211.729268] Enter: dla_prepare_operation
[ 2211.734523] processor:PDP group:0, rdma_group:0 available
[ 2211.741271] Enter: dla_read_config
[ 2211.746045] Exit: dla_read_config
[ 2211.750733] Exit: dla_prepare_operation status=0
[ 2211.756763] Enter: dla_program_operation
[ 2211.762096] Program PDP operation index 11 ROI 0 Group[0]
[ 2211.768910] group id 0 rdma id 0
[ 2211.773538] no desc get due to index==-1
[ 2211.778850] no desc get due to index==-1
[ 2211.784149] no desc get due to index==-1
[ 2211.789427] no desc get due to index==-1
[ 2211.794682] no desc get due to index==-1
[ 2211.799906] Enter: dla_op_programmed
[ 2211.804748] Update dependency operation index 22 ROI 0 DEP_COUNT=2
[ 2211.812207] Exit: dla_op_programmed
[ 2211.816961] Exit: dla_program_operation status=0
[ 2211.822846] Exit: dla_submit_operation
[ 2211.827860] Exit: dla_dequeue_operation
[ 2211.832958] Enter: dla_submit_operation
[ 2211.838041] Prepare CDP operation index 4 ROI 0 dep_count 2
[ 2211.844894] Enter: dla_prepare_operation
[ 2211.850106] processor:CDP group:1, rdma_group:1 available
[ 2211.856828] Enter: dla_read_config
[ 2211.861562] Exit: dla_read_config
[ 2211.866189] Exit: dla_prepare_operation status=0
[ 2211.872116] Enter: dla_program_operation
[ 2211.877340] Program CDP operation index 4 ROI 0 Group[1]
[ 2211.883957] Enter: dla_cdp_program
[ 2211.883960] Enter: processor_cdp_program
[ 2211.893984] Exit: processor_cdp_program
[ 2211.893985] Exit: dla_cdp_program
[ 2211.899117] no desc get due to index==-1
[ 2211.908890] no desc get due to index==-1
[ 2211.914053] no desc get due to index==-1
[ 2211.919231] no desc get due to index==-1
[ 2211.924404] no desc get due to index==-1
[ 2211.929580] Enter: dla_op_programmed
[ 2211.934423] Update dependency operation index 10 ROI 0 DEP_COUNT=3
[ 2211.941907] Exit: dla_op_programmed
[ 2211.946681] Exit: dla_program_operation status=0
[ 2211.952573] Exit: dla_submit_operation
[ 2211.957589] Enter: dla_dequeue_operation
[ 2211.962782] Dequeue op from CDP processor, index=10 ROI=0
[ 2211.969474] Enter: dla_submit_operation
[ 2211.974592] Prepare CDP operation index 10 ROI 0 dep_count 2
[ 2211.981545] Enter: dla_prepare_operation
[ 2211.986748] processor:CDP group:0, rdma_group:0 available
[ 2211.993436] Enter: dla_read_config
[ 2211.998119] Exit: dla_read_config
[ 2212.002687] Exit: dla_prepare_operation status=0
[ 2212.008579] Exit: dla_submit_operation
[ 2212.013591] Exit: dla_dequeue_operation
[ 2212.018685] Exit: dla_initiate_processors status=0
[ 2212.024728] Enter:dla_handle_events, processor:BDMA
[ 2212.030851] Exit:dla_handle_events, ret:0
[ 2212.036104] Enter:dla_handle_events, processor:Convolution
[ 2212.042876] Exit:dla_handle_events, ret:0
[ 2212.048188] Enter:dla_handle_events, processor:SDP
[ 2212.054317] Exit:dla_handle_events, ret:0
[ 2212.059688] Enter:dla_handle_events, processor:PDP
[ 2212.065855] Exit:dla_handle_events, ret:0
[ 2212.071217] Enter:dla_handle_events, processor:CDP
[ 2212.077357] Exit:dla_handle_events, ret:0
[ 2212.082717] Enter:dla_handle_events, processor:RUBIK
[ 2212.089054] Exit:dla_handle_events, ret:0
[ 2212.094454] Enter:dla_handle_events, processor:BDMA
[ 2212.100733] Exit:dla_handle_events, ret:0
[ 2212.106130] Enter:dla_handle_events, processor:Convolution
[ 2212.113015] Exit:dla_handle_events, ret:0
[ 2212.118409] Enter:dla_handle_events, processor:SDP
[ 2212.124584] Exit:dla_handle_events, ret:0
[ 2212.129985] Enter:dla_handle_events, processor:PDP
[ 2212.136181] Exit:dla_handle_events, ret:0
[ 2212.141594] Enter:dla_handle_events, processor:CDP
[ 2212.147788] Exit:dla_handle_events, ret:0
[ 2212.153183] Enter:dla_handle_events, processor:RUBIK
[ 2212.159561] Exit:dla_handle_events, ret:0

Dead here, can't keep running.

What could be the cause of this problem? Has anyone ever had the same problem? How to fix it.