reward: Fix state regeneration with post-Kaia staking info

[blukat29]

Proposed changes

• Affects

  • debug_trace* APIs on full-mode nodes may have returned incorrect results if the traced transaction touches any validator reward address. e.g. interaction with PublicDelegation contracts. Archive-mode nodes are unaffected.

• Background

  • State regeneration (or reexec) is a process of generating the world state at a given block when the state is not in the database. It works by re-executing the transactions (and post-transaction state transitions by Finalize()), starting from the nearest block in which the state exists in the database.
  • State regen is primarily used in debug trace APIs. In those APIs, a block or a transaction is executed against a historic block state. To support this API even in full nodes, the block state is regenerated if not exist in the database.
  • During state regen, the Finalize() queries the staking info for the block to calculate rewards. Without the staking info, we cannot regenerate the correct state.
  • Since Kaia hardfork, the staking info is calculated from the previous block's state.

• Problem

  • During state regen, the staking info is supposed to be available because block states are sequentially generated. Suppose you want to regen from block 10 to 12 (block 10's state is in database)
    • Load state at 10.
    • Execute block 11 to regen state at 11. Staking info is from state at 10.
    • Execute block 12 to regen state at 12. Staking info is from state at 11.
  • But in reality, when executing block 12, StakingManager silently fails to read staking info from block 11 state.
    • Regenerated state is saved in an ephemeral database (https://github.com/kaiachain/kaia/blob/v1.0.1-rc.1/node/cn/state_accessor.go#L82-L84)
    • StakingManager reads from the official database (https://github.com/kaiachain/kaia/blob/v1.0.1-rc.1/node/cn/backend.go#L364)
    • So StakingManager fails at https://github.com/kaiachain/kaia/blob/v1.0.1-rc.1/blockchain/system/multicall.go#L79.

• Solution

  • During state regen, preload staking info from the ephemeral state.
    • Load state at 10. Preload staking info from state at 10.
    • Execute block 11 to regen state at 11. Use preloaded staking info. Preload staking info from state at 11.
    • Execute block 12 to regen state at 12. Use preloaded staking info. Preload staking info from state at 12.
    • Delete preloaded staking info because it's only for regen.
  • Because preloaded staking info can be concurrently used by multiple threads (e.g. concurrent debug API calls), it is implemented with a reference counting data structure.

Types of changes

• Bugfix
• New feature or enhancement
• Others

Checklist

• I have read the CONTRIBUTING GUIDELINES doc
• I have read the CLA and signed by comment I have read the CLA Document and I hereby sign the CLA in first time contribute
• Lint and unit tests pass locally with my changes ($ make test)
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in downstream modules

Related issues

Further comments

[blukat29]

@yoomee1313 @hyeonLewis @hyunsooda PTAL. Tell me if you need further context.

[hyunsooda]

I think cache seems more appropriate rather than preload, which seems ambiguous. Can you please provide motivation for such a name?

[blukat29]

@hyunsooda Thanks for asking that. Preload is a strictly distinct feature from the cache. The difference is eviction.

• Cache (stakingInfoCache) can be safely evicted because we can read from database upon cache miss. It is for optimization purpose and non-essential for correct operation.
• Preload (preloadedInfo) must not be evicted because there's no way we can re-calculate from the state. It is for correct operation.

When I named the feature, I ruled out 'cache' because it implies it can be evicted. So I used the name 'preload'. I also considered 'prefetch' but didn't choose it because prefetch also implies optimization.