Fix/ Indexer chunk size reduce

[bogdanfazakas]

Fixes #354 #366 #388 .

Changes proposed in this PR:

• update reduce chunk size logic
• split retrieve and process logic for chunk size reduction
• added lock to loop so we wait util processing certain blocks before starting with new ones(that may start with the same block)
• refactored the way we used last indexed block inside the indexer thread

[paulo-ocean]

hey @bogdanfazakas
can you check the integration test "should store the ddo state in the db with no errors and retrieve it using did"
this seems to be failing since the merge of the "fix ddo state endpoint PR"
thanks

[bogdanfazakas]

hey @bogdanfazakas can you check the integration test "should store the ddo state in the db with no errors and retrieve it using did" this seems to be failing since the merge of the "fix ddo state endpoint PR" thanks

yeah checking it now, strange things going on, was not failing on the branch and also the tests are passing locally 😃
so a bit more to debug

[paulo-ocean]

i was actually reviewing this while you were changing it :-)
anyway, just noticed that your lockProccessing variable is always false, not what you wanted for sure :-)

[bogdanfazakas]

@paulo-ocean & @mariacarmina @jamiehewitt15 this should be ready and should fix also our integration test issues

[paulo-ocean]

lgtm

[jamiehewitt15]

Looks good to me