Externalities not allowed to fail within runtime

[Unrender]

Good morning,

This isn't the first time I've had this error, but I don't know where it could come from.
And it occurs at different times.
The first time, the node had been running for more than 50 days.
I had it again 2 weeks ago and again last night.

The only way I've found to quickly restart the node is to completely delete the directory containing the data.

And so, when that happens, I have to connect to the web interface of my VPS to restart it first because I no longer have access via ssh (I can't even ping the VPS) .

But it's really boring.

Have you ever had the same thing and above all, do you know how to prevent it?

(I use the 1.3.2 binary file that I downloaded, and it's an archive node)

May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Consumed 12.273s CPU time.
May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Scheduled restart job, restart counter is at 61.
May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: Stopped Node Ternoa.
May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Consumed 12.273s CPU time.
May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: Started Node Ternoa.
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 Ternoa Node
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ✌️ version 1.1.1
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ❤️ by Ternoa, 2020-2024
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 📋 Chain specification: Ternoa Mainnet
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 🏷 Node name: NodePainNodeGain
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 👤 Role: FULL
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 💾 Database: RocksDb at /blocks/chains/mainnet-live/db/full
May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ⛓ Native runtime: mainnet-21 (mainnet-0.tx1.au1)
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: ====================
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: Version: 1.1.1
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 0: sp_panic_handler::set::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 1: std::panicking::rust_panic_with_hook
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:702:17
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 2: std::panicking::begin_panic_handler::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:588:13
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 3: std::sys_common::backtrace::__rust_end_short_backtrace
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/sys_common/backtrace.rs:138:18
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 4: rust_begin_unwind
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:584:5
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 5: core::panicking::panic_fmt
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/panicking.rs:142:14
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 6: core::result::unwrap_failed
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/result.rs:1814:5
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 8: <&mut dyn sp_externalities::Externalities as sp_io::storage::Storage>::get_version_1
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 9: sp_externalities::scope_limited::with_externalities
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 10: tracing::span::Span::in_scope
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 11: sp_io::storage::get_version_1
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 12: sp_io::storage::ExtStorageGetVersion1::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 13: <sc_executor_wasmtime::imports::Registry as sp_wasm_interface::HostFunctionRegistry>::with_function_context
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 14: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 15: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller,A1),R>>::into_func::wasm_to_host_shim
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 16:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 17:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 18:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 19:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 20: wasmtime_runtime::traphandlers::catch_traps::call_closure
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 21: wasmtime_setjmp
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 22: wasmtime_runtime::traphandlers::::with
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 23: wasmtime_runtime::traphandlers::catch_traps
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 24: wasmtime::func::invoke_wasm_and_catch_traps
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 25: wasmtime::func::typed::TypedFunc<Params,Results>::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 26: sc_executor_wasmtime::instance_wrapper::EntryPoint::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 27: sc_executor_wasmtime::runtime::perform_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 28: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 29: sc_executor_common::wasm_runtime::WasmInstance::call_export
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 30: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 31: std::thread::local::LocalKey::with
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 32: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 33: sc_executor::wasm_runtime::RuntimeCache::with_instance
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 34: <sc_executor::native_executor::NativeElseWasmExecutor as sp_core::traits::CodeExecutor>::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 35: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 36: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 37: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor>::contextual_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 38: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt>::call_api_at
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 39: <mainnet_runtime::RuntimeApiImpl<SR_API_BLOCK,RuntimeApiImplCall> as sp_api::Core<SR_API_BLOCK>>::__runtime_api_internal_call_api_at
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 40: sp_consensus_babe::BabeApi::configuration
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 41: ternoa_service::new_partial
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 42: ternoa_service::new_full
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 43: tokio::runtime::park::CachedParkThread::block_on
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 44: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 45: tokio::runtime::runtime::Runtime::block_on
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 46: sc_cli::runner::Runner::run_node_until_exit
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 47: ternoa_cli::command::run
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 48: ternoa::main
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 49: std::sys_common::backtrace::__rust_begin_short_backtrace
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 50: std::rt::lang_start::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 51: core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/ops/function.rs:280:13
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try::do_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:492:40
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:456:19
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panic::catch_unwind
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panic.rs:137:14
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::rt::lang_start_internal::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/rt.rs:128:48
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try::do_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:492:40
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:456:19
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panic::catch_unwind
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panic.rs:137:14
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::rt::lang_start_internal
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/rt.rs:128:20
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 52: main
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 53:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 54: __libc_start_main
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 55: _start
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: Thread 'main' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0xce06c4bff55e0fef101af3ed89f4bbda49a9b459f3d9f598fd18f28c9498c16a"', /home/peshwar9/.cargo/git/checkouts/substrate-7e08433d4c370a21/a3ed011/primitives/state-machine/src/ext.rs:189
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: This is a bug. Please report it at:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: https://github.com/capsule-corp-ternoa/chain/issues
May 12 10:21:28 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Main process exited, code=killed, status=11/SEGV
May 12 10:21:28 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Failed with result 'signal'.