Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor sync out of block_store.rs #2089

Open
wants to merge 99 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
b161579
feat: added new ExternalMessage::RequestFromHeight and ExternalMessag…
shawn-zil Dec 24, 2024
515d4ff
feat: initial blockstore.rs skeleton.
shawn-zil Dec 25, 2024
333542e
feat: added request/response skeleton.
shawn-zil Dec 26, 2024
c9cace2
feat: hook up initial wiring of blockstore with consensus.
shawn-zil Dec 26, 2024
efe631f
feat: added blockstore.rs.
shawn-zil Dec 26, 2024
b814ecc
feat: added in-flight check.
shawn-zil Dec 27, 2024
4488630
feat: added debug/warn/trace messages.
shawn-zil Dec 27, 2024
56f87f3
feat: initial requests firing.
shawn-zil Dec 27, 2024
e3f73a7
feat: convert config value to usize for simplicity.
shawn-zil Dec 27, 2024
bf56c1b
feat: added blockstore::handle_response_from_height().
shawn-zil Dec 27, 2024
292e397
feat: checkpoint, successful RequestFromHeight-ResponseFromHeight.
shawn-zil Dec 27, 2024
1392793
feat: direct insert into DB, without receipts/touched/state.
shawn-zil Dec 27, 2024
28cab10
feat: successfully injecting blocks/state_trie
shawn-zil Dec 27, 2024
48ae6d7
feat: small refactor blockstore.rs
shawn-zil Dec 28, 2024
6334c80
sec: make RequestId random, to mitigate response injections.
shawn-zil Dec 28, 2024
36c667c
feat: minor reorg.
shawn-zil Dec 28, 2024
c7247e7
feat: disable speculative requests for now, until we have a better wa…
shawn-zil Dec 28, 2024
74728bf
feat: re-enabled speculative fetch.
shawn-zil Dec 28, 2024
680e25f
feat: use InjectedProposals instead of ProcessProposals.
shawn-zil Dec 30, 2024
e5b95b6
chore: minor cleanups.
shawn-zil Dec 30, 2024
9dca837
feat: avoid single source of truth.
shawn-zil Dec 30, 2024
72e6ae1
fix: insufficient peers in GCP.
shawn-zil Dec 30, 2024
6ae8701
feat: only inject blocks sourced from two peers - impossible to sync,…
shawn-zil Dec 30, 2024
7df24d9
feat: sort-of working sync with multiple sources of truth.
shawn-zil Dec 30, 2024
cd8a8e5
feat: pre-allocate enough capacity; corrected block_gap check.
shawn-zil Dec 30, 2024
d8c0788
feat: replace non-corroborated blocks in cache.
shawn-zil Dec 30, 2024
5c4b04b
chore: clippy. checkpoint. [corroborate proposals]
shawn-zil Dec 30, 2024
bb9b181
feat: [speculative fetch]
shawn-zil Dec 30, 2024
d7832ec
feat: remove peer check, which allows it to proceed under circumstanc…
shawn-zil Dec 30, 2024
22ba183
chore: clippy.
shawn-zil Dec 30, 2024
7c0f3fd
feat: added handle_metadata_request/response().
shawn-zil Jan 2, 2025
9da4843
feat: [checkpoint - retrieve chain metadata].
shawn-zil Jan 3, 2025
652d281
feat: added handle_multiblock_request/response().
shawn-zil Jan 3, 2025
cf8e62b
feat: [checkpoint - multi_block_request/response; never quite catchin…
shawn-zil Jan 3, 2025
50e8cc9
chore: clippy.
shawn-zil Jan 3, 2025
f83c01a
feat: sync phase#3 - zip it up. works for syncing new nodes.
shawn-zil Jan 4, 2025
1f9e893
feat: rename blockstore.rs to sync.rs - makes clear that its job is t…
shawn-zil Jan 6, 2025
b4dc764
nit: minor refactor - removing previous strategy.
shawn-zil Jan 6, 2025
fecab2d
feat: request multi-blocks from original meta-data peer.
shawn-zil Jan 6, 2025
2df7d31
feat: validates the chain metadata as it is retrieved.
shawn-zil Jan 6, 2025
2f826f4
chore: minor cleanup.
shawn-zil Jan 6, 2025
c9926a0
feat: perform checks to ensure multi-block response matches multi-blo…
shawn-zil Jan 6, 2025
4c92394
feat: allow retries of request_missing_blocks().
shawn-zil Jan 6, 2025
0012261
feat: added ability to retry phase 1, during phase 2 error.
shawn-zil Jan 7, 2025
bfd7f5a
feat: combined p1_cursor/p2_cursor into a self.state value.
shawn-zil Jan 7, 2025
73f5029
feat: restructure sync_proposal() to make it legible.
shawn-zil Jan 7, 2025
b5a592d
checkpoint: working sync with state machine.
shawn-zil Jan 7, 2025
87f038a
Revert "sec: make RequestId random, to mitigate response injections."
shawn-zil Jan 8, 2025
f00530f
feat: make fixed-sized queue size, configurable.
shawn-zil Jan 8, 2025
4a4c709
feat: v1 sync compatibility.
shawn-zil Jan 9, 2025
ba2c6ee
feat: use ChainMetaData as the main state variable structure.
shawn-zil Jan 9, 2025
3777c46
feat: make sync compatible with older nodes.
shawn-zil Jan 9, 2025
05293fb
feat: default to V1 peer; upgrade to V2 peer upon getting invalid res…
shawn-zil Jan 9, 2025
d3553c2
feat: filter V1 responses for gaps and forks.
shawn-zil Jan 10, 2025
a7c6458
feat: working phase 1 with protomainnet.
shawn-zil Jan 10, 2025
c2d8f4b
feat: removed sending BlockRequest from block_store.rs
shawn-zil Jan 10, 2025
ad9c357
chore: comments, cleanup.
shawn-zil Jan 10, 2025
935e4de
fix: correct Phase 2 range, the stored value is accurate.
shawn-zil Jan 10, 2025
21ec72b
feat: ensure unique peers.
shawn-zil Jan 11, 2025
6c88c4e
feat: output rate stats in block/s
shawn-zil Jan 11, 2025
e61fa87
feat: minor reorg, logging.
shawn-zil Jan 11, 2025
e06ade8
feat: added saving of metadata/segments to DB - allows continuation.
shawn-zil Jan 12, 2025
93107b5
feat: added stateful sync algorithm feature - can continue after rest…
shawn-zil Jan 12, 2025
dd7da23
feat: rebuilt the algorithm to use DB for state, instead of in-memory.
shawn-zil Jan 13, 2025
d3a791e
feat: added PeerVer info to DB.
shawn-zil Jan 13, 2025
daa4c77
chore: post-rebase.
shawn-zil Jan 14, 2025
c3711ec
feat: removed block_store.rs
shawn-zil Jan 16, 2025
25cbd17
feat: added sync from timeout, not just proposals.
shawn-zil Jan 16, 2025
6a65afb
feat: made the batch_size dynamic, so that it can get past a larger r…
shawn-zil Jan 16, 2025
8ae5db0
feat: added dynamic_batch_sizing() which is reactive, not pro-active.
shawn-zil Jan 17, 2025
c62aa92
feat: make dynamic_batch_sizing() work per-peer, not per sync.
shawn-zil Jan 17, 2025
36a305e
fix: wire up peers in test Network.
shawn-zil Jan 17, 2025
8df3540
fix: handle when V2 BlockResponse is late.
shawn-zil Jan 17, 2025
0c970a1
feat: sync batch_size should depend on the current request_range, not…
shawn-zil Jan 17, 2025
52f3bcb
feat: simplified the request_missing_metadata() match selector.
shawn-zil Jan 17, 2025
c05d533
fix: improve test sync, added Network::run_until_synced();
shawn-zil Jan 20, 2025
b14e0fc
fix: fixed unreliable::blocks_are_produced_while_a_node_restarts() test.
shawn-zil Jan 20, 2025
298db1a
fix: staking::validators_can_join_and_become_proposer() test.
shawn-zil Jan 20, 2025
524040e
fix: tests.
shawn-zil Jan 20, 2025
87cce64
nit: use Db::contains_block() instead of Db::get_block_by_hash().
shawn-zil Jan 21, 2025
5ac7137
fix: tests.
shawn-zil Jan 21, 2025
166f8b5
feat: retry sync against upgraded Peer, immediately.
shawn-zil Jan 21, 2025
7ba8320
fix: checkpoints_test(), randomized add_peers() for tests.
shawn-zil Jan 21, 2025
9b53ba2
fix: handle_forking(), validators_can_join() test.
shawn-zil Jan 22, 2025
ef42826
fix: experiment.
shawn-zil Jan 22, 2025
313b1b9
fix: checkpoints_test().
shawn-zil Jan 23, 2025
2c01a42
feat: check for checkpoint block, not just history.
shawn-zil Jan 23, 2025
a380986
fix: undoing some test timeouts.
shawn-zil Jan 23, 2025
907a438
feat: replace ChainMetaData with BlockHeader.
shawn-zil Jan 23, 2025
df1c977
feat: changed BlockRequestV2 from 'hash'-based to 'height'-based.
shawn-zil Jan 23, 2025
f8ddef0
feat: simplify checkpointed check.
shawn-zil Jan 24, 2025
945c040
nit: make sync_data temporary.
shawn-zil Jan 24, 2025
008248d
fix: better error handling for committee_for_hash().
shawn-zil Jan 24, 2025
ccecd73
feat: sets starting_block during Sync::new().
shawn-zil Jan 24, 2025
12616a3
feat: store gas_used as a proxy for block size.
shawn-zil Jan 24, 2025
201d472
feat: reordered handle_metadata_response() to allow for micro-segment…
shawn-zil Jan 24, 2025
bb159de
fix: removed dynamic_batch_sizing() as it should be unnecessary until…
shawn-zil Jan 24, 2025
99b527e
feat: shifts txn verification from server side to client side.
shawn-zil Jan 24, 2025
9507067
fix: validators_can_join..()
shawn-zil Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 3 additions & 19 deletions z2/src/converter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,14 @@ use bitvec::{bitarr, order::Msb0};
use eth_trie::{EthTrie, MemoryDB, Trie};
use indicatif::{ProgressBar, ProgressFinish, ProgressIterator, ProgressStyle};
use itertools::Itertools;
use libp2p::PeerId;
use sha2::{Digest, Sha256};
use tokio::sync::mpsc;
use tracing::{debug, trace, warn};
use zilliqa::{
block_store::BlockStore,
cfg::{scilla_ext_libs_path_default, Amount, Config, NodeConfig},
crypto::{Hash, SecretKey},
db::Db,
exec::store_external_libraries,
message::{Block, QuorumCertificate, Vote, MAX_COMMITTEE_SIZE},
node::{MessageSender, RequestId},
schnorr,
scilla::{storage_key, CheckOutput, ParamValue, Transition},
state::{Account, Code, ContractInit, State},
Expand Down Expand Up @@ -346,27 +342,15 @@ pub async fn convert_persistence(
"{msg} {wide_bar} [{per_sec}] {human_pos}/~{human_len} ({elapsed}/~{duration})",
)?;

let (outbound_message_sender, _a) = mpsc::unbounded_channel();
let (local_message_sender, _b) = mpsc::unbounded_channel();
let message_sender = MessageSender {
our_shard: 0,
our_peer_id: PeerId::random(),
outbound_channel: outbound_message_sender,
local_channel: local_message_sender,
request_id: RequestId::default(),
};
// let (outbound_message_sender, _a) = mpsc::unbounded_channel();
// let (local_message_sender, _b) = mpsc::unbounded_channel();

let zq2_db = Arc::new(zq2_db);
let node_config = &zq2_config.nodes[0];
let block_store = Arc::new(BlockStore::new(
node_config,
zq2_db.clone(),
message_sender.clone(),
)?);
let mut state = State::new_with_genesis(
zq2_db.clone().state_trie()?,
node_config.clone(),
block_store,
zq2_db.clone(),
)?;

let mut scilla_docker = run_scilla_docker()?;
Expand Down
14 changes: 3 additions & 11 deletions zilliqa/src/api/zilliqa.rs
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,7 @@ fn get_blockchain_info(_: Params, node: &Arc<Mutex<Node>>) -> Result<BlockchainI
let num_peers = node.get_peer_num();
let num_tx_blocks = node.get_chain_tip();
let num_ds_blocks = (num_tx_blocks / TX_BLOCKS_PER_DS_BLOCK) + 1;
let num_transactions = node.consensus.block_store.get_num_transactions()?;
let num_transactions = node.consensus.get_num_transactions()?;
let ds_block_rate = tx_block_rate / TX_BLOCKS_PER_DS_BLOCK as f64;

// num_txns_ds_epoch
Expand All @@ -518,7 +518,6 @@ fn get_blockchain_info(_: Params, node: &Arc<Mutex<Node>>) -> Result<BlockchainI
for i in current_epoch_first..node.get_chain_tip() {
let block = node
.consensus
.block_store
.get_canonical_block_by_number(i)?
.ok_or_else(|| anyhow!("Block not found"))?;
num_txns_ds_epoch += block.transactions.len();
Expand All @@ -527,7 +526,6 @@ fn get_blockchain_info(_: Params, node: &Arc<Mutex<Node>>) -> Result<BlockchainI
// num_txns_tx_epoch
let latest_block = node
.consensus
.block_store
.get_canonical_block_by_number(node.get_chain_tip())?;
let num_txns_tx_epoch = match latest_block {
Some(block) => block.transactions.len(),
Expand Down Expand Up @@ -1247,11 +1245,7 @@ fn get_recent_transactions(
let mut txns = Vec::new();
let mut blocks_searched = 0;
while block_number > 0 && txns.len() < 100 && blocks_searched < 100 {
let block = match node
.consensus
.block_store
.get_canonical_block_by_number(block_number)?
{
let block = match node.consensus.get_canonical_block_by_number(block_number)? {
Some(block) => block,
None => continue,
};
Expand All @@ -1274,7 +1268,7 @@ fn get_recent_transactions(
// GetNumTransactions
fn get_num_transactions(_params: Params, node: &Arc<Mutex<Node>>) -> Result<String> {
let node = node.lock().unwrap();
let num_transactions = node.consensus.block_store.get_num_transactions()?;
let num_transactions = node.consensus.get_num_transactions()?;
Ok(num_transactions.to_string())
}

Expand All @@ -1283,7 +1277,6 @@ fn get_num_txns_tx_epoch(_params: Params, node: &Arc<Mutex<Node>>) -> Result<Str
let node = node.lock().unwrap();
let latest_block = node
.consensus
.block_store
.get_canonical_block_by_number(node.get_chain_tip())?;
let num_transactions = match latest_block {
Some(block) => block.transactions.len(),
Expand All @@ -1302,7 +1295,6 @@ fn get_num_txns_ds_epoch(_params: Params, node: &Arc<Mutex<Node>>) -> Result<Str
for i in current_epoch_first..node.get_chain_tip() {
let block = node
.consensus
.block_store
.get_canonical_block_by_number(i)?
.ok_or_else(|| anyhow!("Block not found"))?;
num_txns_epoch += block.transactions.len();
Expand Down
Loading
Loading