upgrades: failing to release lock on storage

[conorsch]

On the latest round of migration-testing for Testnet 76 (#4402), I've started seeing:

2024-05-21T22:13:45.041770Z  INFO pd_migrate:migrate{self=Testnet76 comet_home=Some("/penumbra-config/penumbra-devnet-val/node0/cometbft")}:migrate{storage=Storage { .. } pd_home="/penumbra-config/penumbra-devnet-val/node0/pd" genesis_start=Some(Time(2024-05-21 22:05:35.659383898))}: cnidarium
::storage: opening rocksdb config column path="/penumbra-config/penumbra-devnet-val/node0/pd/rocksdb"
thread 'tokio-runtime-worker' panicked at /usr/src/penumbra/crates/cnidarium/src/storage.rs:73:60:
can open database: Error { message: "IO error: lock hold by current process, acquire time 1716329625 acquiring thread 29: /penumbra-config/penumbra-devnet-val/node0/pd/rocksdb/LOCK: No locks available" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: failed to upgrade state

Caused by:
    task 19 panicked
command terminated with exit code 1

This looks like #4344, so it's likely a recent merge forgot to drop a storage handle.

[conorsch]

Found one missing:

diff --git a/crates/bin/pd/src/migrate/testnet76.rs b/crates/bin/pd/src/migrate/testnet76.rs
index b02b2b559..5c8e6ea15 100644
--- a/crates/bin/pd/src/migrate/testnet76.rs
+++ b/crates/bin/pd/src/migrate/testnet76.rs
@@ -121,6 +121,7 @@ pub async fn migrate(
     let rocksdb_dir = pd_home.join("rocksdb");
     let storage = Storage::load(rocksdb_dir, SUBSTORE_PREFIXES.to_vec()).await?;
     let migrated_state = storage.latest_snapshot();
+    storage.release().await;

     // The migration is complete, now we need to generate a genesis file. To do this, we need
     // to lookup a validator view from the chain, and specify the post-upgrade app hash and

Checking for more...

[conorsch]

diff --git a/crates/bin/pd/src/migrate/reset_halt_bit.rs b/crates/bin/pd/src/migrate/reset_halt_bit.rs
index b9877378c..4e0060592 100644
--- a/crates/bin/pd/src/migrate/reset_halt_bit.rs
+++ b/crates/bin/pd/src/migrate/reset_halt_bit.rs
@@ -14,6 +14,7 @@ pub async fn migrate(
     let mut delta = StateDelta::new(export_state);
     delta.ready_to_start();
     let _ = storage.commit_in_place(delta).await?;
+    storage.release().await;
     tracing::info!("migration completed: halt bit is turned off, chain is ready to start");
     Ok(())
 }

[erwanor]

I had removed it, it must have slipped back in during a rebase of #4137. I will push a pr.

[conorsch]

Those two aren't sufficient to resolve, so I'm still digging around. Happy to pair or review, I've got a slick testing setup to validate.

[erwanor]

@conorsch I believe this should fix it: fb1b14e