feat(pending): pending block sealed instead of re-added to mempool

[Trantorian1]

Important

Most of the changes in this pr are just me adding some tests. Actual logic changes are pretty small.

Pull Request type

• Feature

What is the current behavior?

Pending block transactions are re-added back to the mempool and re-executed upon startup. This happens in case of graceful or ungraceful shutdown in the middle of block production: the pending block will be saved to db to avoid loss of state.

This leads to several issues:

1. Transactions are re-executed, and might provided different outputs or hashes.
2. Transactions from the pending block are treated like any other and could be dropped by the mempool under certain edge cases.

Resolves: #458

What is the new behavior?

Pending block, alongside the pending state diff, visited segments and declared classes are retrieved from db and closed into a new finalized block.

This means that if a sequencer node is shutdown mid-production, the state contained in the pending block will automatically be persisted upon startup as it own finalized block without re-executing transactions. This also guarantees that any changes to the config will not be applied to transactions executed under different parameters (these will only affect the next block).

Does this introduce a breaking change?

No.

[Mohiiit]

Great work! The test cases are really clean and well-structured. I have a couple of thoughts:

• Should we consider adding E2E tests for this? For example, we could simulate spawning Madara, shutting it down mid-process, and then restarting it to verify that everything works as expected.
• I found the stages a bit confusing. In most test cases, when we store the pending block in stage 2, it’s mentioned that this simulates stopping and restarting the node. However, isn’t that more related to the block production task? Maybe I’m misunderstanding, but this part could use some clarification.

[Trantorian1]

• I found the stages a bit confusing. In most test cases, when we store the pending block in stage 2, it’s mentioned that this simulates stopping and restarting the node. However, isn’t that more related to the block production task? Maybe I’m misunderstanding, but this part could use some clarification.

So, the block production functions by storing un-finalized blocks as pending. This is the only form of data we can recover without re-execution as everything else is stored in RAM (mempool transactions which have not yet been polled yet are also stored in db for retrieval, but these haven't been executed anyways). This means that if ever the node crashes, we will only be able to retrieve whatever data was stored in the pending block. This is done atomically so we never commit partial data to the database and only a full pending block can ever be stored.

We are therefore "simulate[ing] stopping and restarting the node", since:

1. This is the only pending data that can persist a node restart, and it cannot be partially valid (we still test failing cases though).
2. Upon restart, this is what the block production would be looking to seal.

I will be adding some if this in comments to the tests.