Create new cluster from snapshot

[fischerman]

Can I create a new cluster from a snapshot? I don't know the cluster token anymore. The old cluster is completely gone. The only thing left is the snapshot. I think the old cluster was newer than rke2 v.1.20.9. On a fresh node I get an error that the token is required.

# /opt/rke2/bin/rke2 server   --cluster-reset   --cluster-reset-restore-path=/home/core/etcd-snapshot-cairo-1678449600 
WARN[0000] not running in CIS mode                      
WARN[0000] remove /var/lib/rancher/rke2/agent/etc/rke2-agent-load-balancer.json: no such file or directory 
WARN[0000] remove /var/lib/rancher/rke2/agent/etc/rke2-api-server-agent-load-balancer.json: no such file or directory 
FATA[0000] /var/lib/rancher/rke2/server/token does not exist, please pass --token to complete the restoration

If I start the server first and try to attempt a restore afterwards, I get the following:

# /opt/rke2/bin/rke2 server   --cluster-reset   --cluster-reset-restore-path=/home/core/etcd-snapshot-cairo-1678449600
...
INFO[0031] ETCD server is now running                   
INFO[0031] rke2 is up and running                       
panic: bootstrap data already found and encrypted with different token

goroutine 418 [running]:
github.com/k3s-io/k3s/pkg/cluster.(*Cluster).Start.func1()
	/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cluster/cluster.go:106 +0xff
created by github.com/k3s-io/k3s/pkg/cluster.(*Cluster).Start
	/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cluster/cluster.go:101 +0x6aa

I don't quite understand how the token and the etcd snapshot are related. Any chance to make use of the snapshot?

[fischerman]

Possibly related to #1898. Is this still accurate today? From the docs:

For all versions of rke2 v.1.20.9 and prior, you will need to back up and restore certificates first due to a known issue in which bootstrap data might not save on restore

The deleted cluster was newer at the time of deletion (through upgrade). However, it was initially bootstrapped with a version below 1.20.

[brandond]

Can I create a new cluster from a snapshot? I don't know the cluster token anymore.
I don't quite understand how the token and the etcd snapshot are related.

Not really, no. While you might be able to access some of the data in etcd, the secrets-encryption configuration, along with the cluster root CA certificates, are all encrypted with the token. If you don't know the token, that data will be inaccessible and the cluster essentially broken and unusable.

[fischerman]

Maybe I missed it, but is this anywhere documented?

I was able to retrieve what I needed using https://github.com/openshift/origin/tree/master/tools/etcdhelper

[brandond]

We do mention needing the token value or file in a couple places within the backup/restore docs, but we could probbly be more explicit about it. I'm currently finishing up some major docs work for K3s and will do the same for RKE2 shortly.