Dynamically choose which plan to run based on archive chain id

[cronokirby]

This also adds a plan for the testnet.

Closes #23.

[conorsch]

This also adds a plan for the testnet.

To test, I attempted an archive from historical state backups, followed by a regen to get a full database. The archive step ostensibly worked great, and I have all blocks in sqlite3 up to block 2157610 on penumbra-testnet-phobos-2; download URL here. However, when I run the regen command based on that archive, it fails immediately:

full logs from regen attempt

❯ RUST_LOG=debug penumbra-reindexer regen --database-url 'postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable' --working-dir /home/conor/testnet-regen/regen
2025-01-27T00:39:37.069626Z DEBUG new: penumbra_reindexer::storage: initializing archive database path="/home/conor/.penumbra/network_data/node0/reindexer_archive.bin"
2025-01-27T00:39:37.070340Z DEBUG new: sqlx::query: summary="PRAGMA foreign_keys = ON; …" db.statement="\n\nPRAGMA foreign_keys = ON;\nPRAGMA synchronous = OFF;\n" rows_affected=0 rows_returned=0 elapsed=186.978µs elapsed_secs=0.000186978
2025-01-27T00:39:37.070402Z DEBUG new: penumbra_reindexer::storage: creating archive tables
2025-01-27T00:39:37.070621Z DEBUG new: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS metadata (\n  id INTEGER PRIMARY KEY CHECK (id = 0),\n  version TEXT NOT NULL UNIQUE,\n  chain_id TEXT NOT NULL UNIQUE\n);\n" rows_affected=0 rows_returned=0 elapsed=25.085µs elapsed_secs=2.5085e-5
2025-01-27T00:39:37.071017Z DEBUG new: sqlx::query: summary="PRAGMA foreign_keys = ON; …" db.statement="\n\nPRAGMA foreign_keys = ON;\nPRAGMA synchronous = OFF;\n" rows_affected=0 rows_returned=0 elapsed=104.019µs elapsed_secs=0.000104019
2025-01-27T00:39:37.071110Z DEBUG new: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS blobs (\n  data BLOB NOT NULL\n)\n" rows_affected=0 rows_returned=0 elapsed=19.79µs elapsed_secs=1.979e-5
2025-01-27T00:39:37.071292Z DEBUG new: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS blocks (\n  height INTEGER NOT NULL PRIMARY KEY,\n  data_id INTEGER NOT NULL\n)\n" rows_affected=0 rows_returned=0 elapsed=20.107µs elapsed_secs=2.0107e-5
2025-01-27T00:39:37.071409Z DEBUG new: sqlx::query: summary="CREATE UNIQUE INDEX IF …" db.statement="\n\nCREATE UNIQUE INDEX IF NOT EXISTS idx_blocks_data_id ON blocks(data_id)\n" rows_affected=0 rows_returned=0 elapsed=27.877µs elapsed_secs=2.7877e-5
2025-01-27T00:39:37.071544Z DEBUG new: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS geneses (\n  initial_height INTEGER NOT NULL PRIMARY KEY,\n  data_id INTEGER NOT NULL\n)\n" rows_affected=0 rows_returned=0 elapsed=17.377µs elapsed_secs=1.7377e-5
2025-01-27T00:39:37.071628Z DEBUG new: sqlx::query: summary="CREATE UNIQUE INDEX IF …" db.statement="\n\nCREATE UNIQUE INDEX IF NOT EXISTS idx_geneses_data_id ON geneses(data_id)\n" rows_affected=0 rows_returned=0 elapsed=15.315µs elapsed_secs=1.5315e-5
2025-01-27T00:39:37.071761Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: penumbra_reindexer::indexer: initializing database
2025-01-27T00:39:37.071763Z DEBUG new: sqlx::query: summary="SELECT version, chain_id FROM …" db.statement="\n\nSELECT\n  version,\n  chain_id\nFROM\n  metadata\n" rows_affected=0 rows_returned=1 elapsed=36.105µs elapsed_secs=3.6105e-5
2025-01-27T00:39:37.076487Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::postgres::notice: relation "blocks" already exists, skipping
2025-01-27T00:39:37.076583Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="-- I had to …" db.statement="\n\n-- I had to remove all the comments to make this work :(\nCREATE TABLE IF NOT EXISTS blocks (\n  rowid BIGSERIAL PRIMARY KEY,\n  height BIGINT NOT NULL,\n  chain_id VARCHAR NOT NULL,\n  created_at TIMESTAMPTZ NOT NULL,\n  UNIQUE (height, chain_id)\n)\n" rows_affected=0 rows_returned=0 elapsed=264.529µs elapsed_secs=0.000264529
2025-01-27T00:39:37.077032Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::postgres::notice: relation "idx_blocks_height_chain" already exists, skipping
2025-01-27T00:39:37.077068Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE INDEX IF NOT …" db.statement="\n\nCREATE INDEX IF NOT EXISTS idx_blocks_height_chain ON blocks(height, chain_id)\n" rows_affected=0 rows_returned=0 elapsed=441.361µs elapsed_secs=0.000441361
2025-01-27T00:39:37.077212Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::postgres::notice: relation "tx_results" already exists, skipping
2025-01-27T00:39:37.077305Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS tx_results (\n  rowid BIGSERIAL PRIMARY KEY,\n  block_id BIGINT NOT NULL REFERENCES blocks(rowid),\n  index INTEGER NOT NULL,\n  created_at TIMESTAMPTZ NOT NULL,\n  tx_hash VARCHAR NOT NULL,\n  tx_result BYTEA NOT NULL,\n  UNIQUE (block_id, index)\n)\n" rows_affected=0 rows_returned=0 elapsed=140.46µs elapsed_secs=0.00014046
2025-01-27T00:39:37.077432Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::postgres::notice: relation "events" already exists, skipping
2025-01-27T00:39:37.077495Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS events (\n  rowid BIGSERIAL PRIMARY KEY,\n  block_id BIGINT NOT NULL REFERENCES blocks(rowid),\n  tx_id BIGINT NULL REFERENCES tx_results(rowid),\n  type VARCHAR NOT NULL\n)\n" rows_affected=0 rows_returned=0 elapsed=123.049µs elapsed_secs=0.000123049
2025-01-27T00:39:37.077605Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::postgres::notice: relation "attributes" already exists, skipping
2025-01-27T00:39:37.077671Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE TABLE IF NOT …" db.statement="\n\nCREATE TABLE IF NOT EXISTS attributes (\n  event_id BIGINT NOT NULL REFERENCES events(rowid),\n  key VARCHAR NOT NULL,\n  composite_key VARCHAR NOT NULL,\n  value VARCHAR NULL,\n  UNIQUE (event_id, key)\n)\n" rows_affected=0 rows_returned=0 elapsed=106.34µs elapsed_secs=0.00010634
2025-01-27T00:39:37.077812Z  INFO init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::postgres::notice: relation "idx_attributes_event_id" already exists, skipping
2025-01-27T00:39:37.077838Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE INDEX IF NOT …" db.statement="\n\nCREATE INDEX IF NOT EXISTS idx_attributes_event_id ON attributes(event_id)\n" rows_affected=0 rows_returned=0 elapsed=137.171µs elapsed_secs=0.000137171
2025-01-27T00:39:37.078731Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE OR REPLACE VIEW …" db.statement="\n\nCREATE\nOR REPLACE VIEW event_attributes AS\nSELECT\n  block_id,\n  tx_id,\n  type,\n  key,\n  composite_key,\n  value\nFROM\n  events\n  LEFT JOIN attributes ON (events.rowid = attributes.event_id)\n" rows_affected=0 rows_returned=0 elapsed=829.604µs elapsed_secs=0.000829604
2025-01-27T00:39:37.079274Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE OR REPLACE VIEW …" db.statement="\n\nCREATE\nOR REPLACE VIEW block_events AS\nSELECT\n  blocks.rowid as block_id,\n  height,\n  chain_id,\n  type,\n  key,\n  composite_key,\n  value\nFROM\n  blocks\n  JOIN event_attributes ON (blocks.rowid = event_attributes.block_id)\nWHERE\n  event_attributes.tx_id IS NULL\n" rows_affected=0 rows_returned=0 elapsed=458.698µs elapsed_secs=0.000458698
2025-01-27T00:39:37.080002Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="CREATE OR REPLACE VIEW …" db.statement="\n\nCREATE\nOR REPLACE VIEW tx_events AS\nSELECT\n  height,\n  index,\n  chain_id,\n  type,\n  key,\n  composite_key,\n  value,\n  tx_results.created_at\nFROM\n  blocks\n  JOIN tx_results ON (blocks.rowid = tx_results.block_id)\n  JOIN event_attributes ON (tx_results.rowid = event_attributes.tx_id)\nWHERE\n  event_attributes.tx_id IS NOT NULL\n" rows_affected=0 rows_returned=0 elapsed=629.032
 s elapsed_secs=0.000629032
2025-01-27T00:39:37.080095Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary=" …" db.statement="\n\n\n" rows_affected=0 rows_returned=0 elapsed=77.096µs elapsed_secs=7.7096e-5
2025-01-27T00:39:37.080487Z DEBUG init{database_url="postgresql://penumbra:[email protected]:5432/penumbra_testnet?sslmode=disable"}: sqlx::query: summary="COMMIT" db.statement="" rows_affected=0 rows_returned=0 elapsed=379.317µs elapsed_secs=0.000379317
2025-01-27T00:39:37.080636Z DEBUG sqlx::query: summary="SELECT chain_id FROM metadata" db.statement="" rows_affected=0 rows_returned=1 elapsed=63.279µs elapsed_secs=6.3279e-5
2025-01-27T00:39:37.081007Z  INFO cnidarium::storage: opening rocksdb config column path="/home/conor/testnet-regen/regen"
2025-01-27T00:39:37.102018Z  INFO cnidarium::storage: reading prefixes from config column family
2025-01-27T00:39:37.103433Z  INFO cnidarium::storage: initializing global store config
2025-01-27T00:39:37.103448Z  INFO cnidarium::storage: creating substore config for prefix prefix="cometbft-data"
2025-01-27T00:39:37.103462Z  INFO cnidarium::storage: creating substore config for prefix prefix="ibc-data"
2025-01-27T00:39:37.103479Z  INFO cnidarium::storage: opening rocksdb path="/home/conor/testnet-regen/regen"
2025-01-27T00:39:37.121817Z DEBUG cnidarium::storage: initializing substore substore_prefix="cometbft-data" substore_version=18446744073709551615
2025-01-27T00:39:37.121841Z DEBUG cnidarium::storage: initializing substore substore_prefix="ibc-data" substore_version=18446744073709551615
2025-01-27T00:39:37.121852Z DEBUG cnidarium::storage: initializing main store jmt_version=18446744073709551615
2025-01-27T00:39:37.121934Z  INFO cnidarium::storage: snapshot dispatcher task has started
2025-01-27T00:39:37.121938Z DEBUG new: penumbra_app::app: initializing App instance
2025-01-27T00:39:37.121986Z DEBUG jmt: hashed jmt key key=b"sct/block_manager/block_height" key_hash=KeyHash("ff40c0c874dd5b23838ce8062a66562f0f8e714f13eef677c5314ea2570e1320")
2025-01-27T00:39:37.122055Z DEBUG penumbra_reindexer::penumbra: error while fetching current metadata: Missing block_height version=V0o79
2025-01-27T00:39:37.122655Z  INFO cnidarium::storage: opening rocksdb config column path="/home/conor/testnet-regen/regen"
2025-01-27T00:39:37.140635Z  INFO cnidarium::storage: reading prefixes from config column family
2025-01-27T00:39:37.141103Z  INFO cnidarium::storage: initializing global store config
2025-01-27T00:39:37.141117Z  INFO cnidarium::storage: creating substore config for prefix prefix="cometbft-data"
2025-01-27T00:39:37.141130Z  INFO cnidarium::storage: creating substore config for prefix prefix="ibc-data"
2025-01-27T00:39:37.141154Z  INFO cnidarium::storage: opening rocksdb path="/home/conor/testnet-regen/regen"
2025-01-27T00:39:37.160092Z DEBUG cnidarium::storage: initializing substore substore_prefix="cometbft-data" substore_version=18446744073709551615
2025-01-27T00:39:37.160119Z DEBUG cnidarium::storage: initializing substore substore_prefix="ibc-data" substore_version=18446744073709551615
2025-01-27T00:39:37.160129Z DEBUG cnidarium::storage: initializing main store jmt_version=18446744073709551615
2025-01-27T00:39:37.160205Z  INFO cnidarium::storage: snapshot dispatcher task has started
2025-01-27T00:39:37.160219Z DEBUG new: penumbra_app::app: initializing App instance
2025-01-27T00:39:37.160271Z DEBUG jmt: hashed jmt key key=b"sct/block_manager/block_height" key_hash=KeyHash("ff40c0c874dd5b23838ce8062a66562f0f8e714f13eef677c5314ea2570e1320")
2025-01-27T00:39:37.160348Z DEBUG penumbra_reindexer::penumbra: error while fetching current metadata: Missing block_height version=V0o80
2025-01-27T00:39:37.160904Z  INFO penumbra_reindexer::penumbra: plan for penumbra-testnet-phobos-2 truncated between None..=None: RegenerationPlan { steps: [(0, InitThenRunTo { genesis_height: 1, version: V0o80, last_block: Some(1459799) }), (1459799, Migrate { from: V0o80, to: V0o81 }), (1459799, InitThenRunTo { genesis_height: 1459800, version: V0o80, last_block: None })] }
2025-01-27T00:39:37.160935Z  INFO init_then_run_to{genesis_height=1 version=V0o80 first_block=1 last_block=Some(1459799)}: penumbra_reindexer::penumbra: regeneration step
2025-01-27T00:39:37.161211Z DEBUG init_then_run_to{genesis_height=1 version=V0o80 first_block=1 last_block=Some(1459799)}: sqlx::query: summary="SELECT (data) FROM geneses …" db.statement="\n\nSELECT\n  (data)\nFROM\n  geneses\n  JOIN blobs ON data_id = blobs.rowid\nWHERE\n  initial_height = ?\n" rows_affected=0 rows_returned=0 elapsed=121.782µs elapsed_secs=0.000121782
2025-01-27T00:39:37.161279Z DEBUG penumbra_reindexer::storage: closing archive database
Error: expected genesis before height 1

Not sure whether the error message "expected genesis before height 1" indicates that the sqlite3 db was malformed. I double-checked for off-by-one errors as seen in #21, and couldn't find any. Will need to try again.

[cronokirby]

delta.bin.gz

Seems like it was missing the genesis file, try applying the patch above with:

gzip -d delta.bin.gz
rdiff patch <original>.sqlite3 delta.bin new.sqlite3
sha256sum new.sqlite3

should produce:

6f4749811386fa58f2ecbd1f705bb30bf27296164594bc700a1ff6e5a5f8e63

[conorsch]

Gorgeous: with your patch, I'm able to regen up to current height. Not sure where I went wrong on the initial archive step, will redo that on a rainy day. For now, unblocked. Thanks for the super terse solution.

[conorsch]

This patch worked for regenerating testnet dbs on penumbra-testnet-phobos-2, so I'm merging.