Archival bucketlist (#4403)

# Description

Resolves #4394

Depends on XDR changes in
stellar/stellar-xdr#200.

This change refactors the BucketList so that we can have multiple Bucket
Lists simultaneously. This is necessary for State Archival, where
validators will maintain a live BucketList, hot archive BucketList, and
cold archive BucketList. This change does not yet support writing the
new archival BucketLists to history,

Each BucketList has small differences wrt to the entry types it stores,
merge logic, and how lookups are done, but the overall BucketList level
logic is the same. Because of this, it seemed easiest to template the
Bucket related classes and specialize a few functions. The difference
are as follows (I'll be updating the Archival CAP with this info soon)

Entry Types:
Live BucketList: `BucketEntry`
Hot Archive BucketList: `HotArchiveBucketEntry`.

This change was necessary due to how LedgerKeys are stored. In the Live
BucketList, only `DEADENTRY` store plain `LedgerKey`. `DEADENTRY` is
equivalent to null, so these LedgerKeys are dropped at the bottom level
bucket.

Hot Archives require two types that store LedgerKeys: `HA_LIVE` and
`HA_DELETED`. `HA_LIVE` is equivalent to the "null" type in the Live
BucketList, and is dropped in the bottom bucket. However, `HA_DELETED`
needs to be persisted.

Merging:
In the hot archive, `LedgerEntry` are never updated. They may be
overwritten by another `HotArchiveBucketEntry` type, but the
`LedgerEntry` contents itself never change. This means that the
`INITENTRY`, `LIVEENTRY`, and `DEADENTRY` abstraction doesn't really
make sense. This makes the merge logic for Hot Archive buckets very
simple, the newer entry always overwrites the older entry.

This PR adds the concept of a "Tombstone" entry. A tombstone entry
essentially represents null in the given BucketList and can be dropped
once it reaches the bottom bucket. On the live BucketList, DEADENTRY is
the tombstone type. On the Hot Archive BucketList, `HA_LIVE` is the
tombstone.

BucketListDB lookup:
The live BucketList only ever returns `LedgerEntry` types when queried.
Any DEADENTRY LedgerKey is simply omitted from the return, as this is
the "null" tombstone type.

Hot archive lookups must return both LedgerKey and LedgerEntry types.
HA_LIVE entries are of type LedgerKey and can be omitted from the return
since they are the tombstone type. However, HA_ARCHIVED entries are not
tombstone entries and must be returned when found. Do to this, Hot
Archive lookups return `HotArchiveBucketEntry` instead of `LedgerEntry`
when queried.

Future updates will add more functionality to Hot Archives, further
distinguishing it from the live BucketList, and add a third BucketList
type called the cold archive. This cold archive will add a third
BucketEntry type and have different merge logic as well, so I think the
templated Bucket classes, while verbose, are warranted.

This is currently a draft until the XDR changes are merged.

# Checklist
- [x] Reviewed the
[contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes)
document
- [x] Rebased on top of master (no merge commits)
- [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio
extension)
- [x] Compiles
- [x] Ran all tests
- [ ] If change impacts performance, include supporting evidence per the
[performance
document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)

src/bucket/BucketApplicator.cpp

@@ -4,8 +4,8 @@
 
 #include "util/asio.h" // IWYU pragma: keep
 #include "bucket/BucketApplicator.h"
-#include "bucket/Bucket.h"
-#include "bucket/BucketList.h"
+#include "bucket/LiveBucket.h"
+#include "bucket/LiveBucketList.h"
 #include "ledger/LedgerTxn.h"
 #include "ledger/LedgerTxnEntry.h"
 #include "main/Application.h"
@@ -20,7 +20,7 @@ BucketApplicator::BucketApplicator(Application& app,
                                    uint32_t maxProtocolVersion,
                                    uint32_t minProtocolVersionSeen,
                                    uint32_t level,
-                                   std::shared_ptr<Bucket const> bucket,
+                                   std::shared_ptr<LiveBucket const> bucket,
                                    std::function<bool(LedgerEntryType)> filter,
                                    std::unordered_set<LedgerKey>& seenKeys)
     : mApp(app)
@@ -135,7 +135,7 @@ BucketApplicator::advance(BucketApplicator::Counters& counters)
         }
 
         BucketEntry const& e = *mBucketIter;
-        Bucket::checkProtocolLegality(e, mMaxProtocolVersion);
+        LiveBucket::checkProtocolLegality(e, mMaxProtocolVersion);
 
         if (shouldApplyEntry(mEntryTypeFilter, e))
         {
@@ -167,15 +167,15 @@ BucketApplicator::advance(BucketApplicator::Counters& counters)
                 // The last level can have live entries, but at that point we
                 // know that they are actually init entries because the earliest
                 // state of all entries is init, so we mark them as such here
-                if (mLevel == BucketList::kNumLevels - 1 &&
+                if (mLevel == LiveBucketList::kNumLevels - 1 &&
                     e.type() == LIVEENTRY)
                 {
                     ltx->createWithoutLoading(e.liveEntry());
                 }
                 else if (
                     protocolVersionIsBefore(
                         mMinProtocolVersionSeen,
-                        Bucket::
+                        LiveBucket::
                             FIRST_PROTOCOL_SUPPORTING_INITENTRY_AND_METAENTRY))
                 {
                     // Prior to protocol 11, INITENTRY didn't exist, so we need
@@ -207,7 +207,7 @@ BucketApplicator::advance(BucketApplicator::Counters& counters)
                 releaseAssertOrThrow(!isUsingBucketListDB);
                 if (protocolVersionIsBefore(
                         mMinProtocolVersionSeen,
-                        Bucket::
+                        LiveBucket::
                             FIRST_PROTOCOL_SUPPORTING_INITENTRY_AND_METAENTRY))
                 {
                     // Prior to protocol 11, DEAD entries could exist

src/bucket/BucketApplicator.h

@@ -4,8 +4,8 @@
 // under the Apache License, Version 2.0. See the COPYING file at the root
 // of this distribution or at http://www.apache.org/licenses/LICENSE-2.0
 
-#include "bucket/Bucket.h"
 #include "bucket/BucketInputIterator.h"
+#include "bucket/LiveBucket.h"
 #include "util/Timer.h"
 #include <memory>
 
@@ -24,7 +24,7 @@ class BucketApplicator
     uint32_t mMaxProtocolVersion;
     uint32_t mMinProtocolVersionSeen;
     uint32_t mLevel;
-    BucketInputIterator mBucketIter;
+    LiveBucketInputIterator mBucketIter;
     size_t mCount{0};
     std::function<bool(LedgerEntryType)> mEntryTypeFilter;
     std::unordered_set<LedgerKey>& mSeenKeys;
@@ -72,7 +72,7 @@ class BucketApplicator
     // When this flag is set, each offer key read is added to seenKeys
     BucketApplicator(Application& app, uint32_t maxProtocolVersion,
                      uint32_t minProtocolVersionSeen, uint32_t level,
-                     std::shared_ptr<Bucket const> bucket,
+                     std::shared_ptr<LiveBucket const> bucket,
                      std::function<bool(LedgerEntryType)> filter,
                      std::unordered_set<LedgerKey>& seenKeys);
     operator bool() const;