blockdb: bound max deleted blocks per blockdb sync

[algorandskiy]

Summary

If a node switches from Archival to non-archival then the blockdb cleanup can take a long time blocking the blockdb in a write tx. This PR addresses this by scoping max blocks deleted per sync op.

Earliest                       MinToSave         Latest

   |                               |                 |
   +-------------------------------+-----------------+
   ^            ^                  ^                 ^
   |            |                  +-----------------+
   +------------+
   | maxDeletionBatchSize          ^     MaxBlockHistoryLookback
   +-------------------------------+

              Deletion Range (Old)

Test Plan

Added a unit test.
Additionally fixed a data race seen in TestVotersReloadFromDiskAfterOneStateProofCommitted.

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (877090b) 55.14% compared to head (0cb72c6) 55.97%.

Files	Patch %	Lines
ledger/blockqueue.go	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5910      +/-   ##
==========================================
+ Coverage   55.14%   55.97%   +0.83%     
==========================================
  Files         478      478              
  Lines       67602    67612      +10     
==========================================
+ Hits        37280    37848     +568     
+ Misses      27757    27201     -556     
+ Partials     2565     2563       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[onetechnical]

From my testing, this looks good. algod remains available, and it sheds 10,000 rounds at a time, though it takes about 26 seconds per batch. This means that for going archival -> non-archival it would take about a day to completely cut through 35 million rounds. I think this is acceptable, since it remains responsive in the mean time.

[jannotti]

it takes about 26 seconds per batch. This means that for going archival -> non-archival it would take about a day to completely cut through 35 million rounds. I think this is acceptable, since it remains responsive in the mean time.

I think this means that since a sync takes 26 seconds or so, we will only be calling sync every 8 rounds or so. If that's the case, I don't see any point in even going as high as 10,000. If we do 1,000 I guess we might be syncing fast enough to sync almost every round. And 8 rounds or so of 1,000 deletions is nearly as good as doing 10,000 rounds every 26 seconds. It'll still clean up in about a day.

Keep in mind, I may be misunderstanding how sync works. I had thought we wrote a block every round.

[algorandskiy]

My intuition is bigger batches should take a bit less overall time than smaller batches because of the index update. Not sure tho if there any optimizations in sqlite for sequential removal from indices.