ingest/ledgerbackend: Add ledger backend which replays ledgers #5584
Conversation
| return bytes.Equal(serialized, otherSerialized), nil | ||
| } | ||
|
|
||
| func changeIsEqual(a, b Change) (bool, error) { |
There was a problem hiding this comment.
IMO this shud be moved from here to be a receiver function in ingest/change.go
Something like func (a Change)IsEqual(b Change)
I remember that I had some usecase for comparing changes when I was working on some change related issues, and I was needing such a compare function
There was a problem hiding this comment.
This function deliberately doesn't compare all the fields in the change struct so in that sense it is tailored specifically to the replay backend's needs. That's why I did not expose it as a public utility function.
The changeIsEqual function compares the xdr values of the pre and post but it does not compare the the transaction, operation, or ledger because it is used to determine that the changes from merging two input ledgers is equal to changes from the first of the input ledgers followed by the changes from the second input ledgers.
|
@karthikiyer56 I think I have addressed your code review feedback, could you take another look? |
| "github.com/stellar/go/xdr" | ||
| ) | ||
|
|
||
| type ReplayBackend struct { |
There was a problem hiding this comment.
I would probably call this LocalFileBackend or FsBackend to indicate it comes from a local file.
| if err := unmarshallCompressedXDRFile(config.LedgerEntriesFilePath, &generatedLedgerEntries); err != nil { | ||
| return nil, err | ||
| } | ||
| if err := unmarshallCompressedXDRFile(config.LedgersFilePath, &generatedLedgers); err != nil { | ||
| return nil, err | ||
| } |
There was a problem hiding this comment.
It may be better to stream the decoding instead of doing it upfront. It allows for larger files not fitting memory and allows parelellizing the decompression and decoding in the future.
| return nil | ||
| } | ||
|
|
||
| func xdrEquals(a, b encoding.BinaryMarshaler) (bool, error) { |
There was a problem hiding this comment.
Shouldn't this live in the XDR package?
| return true, nil | ||
| } | ||
|
|
||
| func changesAreEqual(a, b map[string][]Change) (bool, error) { |
There was a problem hiding this comment.
Shouldn't this live Change is defined?
| } | ||
|
|
||
| // MergeLedgers merges two xdr.LedgerCloseMeta instances. | ||
| func MergeLedgers(networkPassphrase string, dst *xdr.LedgerCloseMeta, src xdr.LedgerCloseMeta) error { |
There was a problem hiding this comment.
Shouldn't this live in the XDR package?
| ) | ||
|
|
||
| type ReplayBackend struct { | ||
| ledgerBackend ledgerbackend.LedgerBackend |
There was a problem hiding this comment.
This ledger backend has a strange role. It composes in a strange way with the actual backend, implementing only certain operations. Also in order for it to work, the passphrase of the passed backend need to be the same, so it's duplicated.
I wonder if we can simply mock some parameters so that passing the backend isn't necessary.
|
All my comments are optional. I am mostly concerned about the role of the passed ledger backend, which is a bit unintuitive to me. |
PR Checklist
PR Structure
otherwise).
services/friendbot, orallordocif the changes are broad or impact manypackages.
Thoroughness
.mdfiles, etc... affected by this change). Take a look in the
docsfolder for a given service,like this one.
Release planning
CHANGELOG.mdwithin the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
Close #5480
Following #5556 , this PR implements a ledger backend which takes synthetically generated ledgers, merges them with real ledgers, and then replays them at a configurable rate.
Why
This ledger backend will be used to benchmark ingestion duration for horizon / rpc when ingesting ledgers with a high volume of soroban transactions. These benchmarks will inform us of horizon and rpc's ceiling when it comes to ingestion throughput.
Known limitations
[N/A]