How to rotate ca cert with new root and --force

[nonessentialprototype]

I am on rke2 2.24.17, and I am trying to rotate the ca cert with a newly generated ca certs. We want to invalidate the old certs/kube config so we do NOT want it cross signed.

I've followed the step here modified for my needs and am running this on the first server
https://docs.rke2.io/security/certificates#rotating-custom-ca-certificates

mkdir -p /tmp/rke2/server/tls
cp  /var/lib/rancher/rke2/server/tls/service.key /tmp/rke2/server/tls

DATA_DIR=/tmp/rke2 PRODUCT=rke2 ./generate-custom-ca-certs.sh

rke2 certificate rotate-ca --path=/tmp/rke2/server --server=https://127.0.0.1:9345 --force

The rotate is successfully but after I run systemctl restart rke2-server.service the server never comes back up and and logs show

CA cert validation failed: Get "https://127.0.0.1:9345/cacerts" tls: failed to verify certificate x509 certificate signed by unknown authority.

This section also mentions

If you used the --force option or changed the root CA, ensure that any nodes that were joined with a secure token are reconfigured to use the new token value, prior to being restarted. The token may be stored in a .env file, systemd unit, or config.yaml, depending on how the node was configured during initial installation.

which seems to be a copy paste error cause the generate-custom-ca-certs does not output a new token.

I then tried these steps modifying the rotate script to not copy and use the old ones.
https://docs.rke2.io/security/certificates#rotating-self-signed-ca-certificates
with the same result, it rotates successfully but 509s on restart.

[brandond]

generate-custom-ca-certs does not output a new token.

yes, generate-custom-ca-certs was intended to be used before initial cluster startup. Using that that to switch from the default self-signed certs, to custom certs, is not something that we've tested. You'll need to calculate the hash yourself, and update the hash in the token files by hand.

[nonessentialprototype]

If that is the case the documentation here https://docs.rke2.io/security/certificates#using-the-example-script-1
, and the script should be updated as it even prints out

  echo "To update certificates on an existing cluster, you may now run:"
  echo "    k3s certificate rotate-ca --path=${DATA_DIR}/server"

[brandond]

Yes, it prints that because it is intended to also be used to update the custom certs when they need to be renewed. It is just use of that script to switch from default to custom that isn't a tested path.

[nonessentialprototype]

I have been able to rotate successfully once, subsequent attempts after all the nodes are up fail.

I ran across this comment k3s-io/k3s#8952 (comment)

Which got me thinking to modify the rotate-default-ca-certs.sh

Adding an If around ${TYPE}-ca.crt creation

if [[ $TYPE = "client" ]]; then
  cat ${TYPE}-ca.pem \
  ${TYPE}-root-ssigned.pem > ${TYPE}-ca.crt

else
    cat ${TYPE}-ca.pem \
      ${TYPE}-root-ssigned.pem \
      ${TYPE}-root-xsigned.pem \
      ${TYPE}-root-old.pem > ${TYPE}-ca.crt
fi

then running

PRODUCT=rke2 ./rotate-default-ca-certs.sh
rke2 certificate rotate-ca --path=/var/lib/rancher/rke2/server/rotate-ca --server=https://127.0.0.1:9345 --force
rke2 certificate rotate
systemctl restart rke2-server

once up, updating the config on each server/agent to the new token and then rebooting them.

Once they are all rebooted, the original kubeconfig has become invalidated.

[brandond]

If that file is exposed your cluster is compromised, even if you rotate certs. That kubeconfig has full admin access and may have already pivoted to nodes or other accessible infrastructure. The threat model of compromised access to the admin kubeconfig does not change by eliminating client certs or making it easier to revoke existing certs.

[nonessentialprototype]

No but there is a way to do a rotation and force a new cert thus invalidating the exiting cert based rbac it's not straightforward. My whole point is the kubeconfig rke2 exposes to the end user after a cluster is set up shouldn't be the cert one.

[hdong69]

It was the 'etc/rancher/rke2/config.yaml' on each agent and node with the new token generated above.

I dont have access currently but we were able to rotate to successfully multiple times in a row.

Either way it's more of a pain then it should be and rke2 should really not use the cert based kubeconfig as the default

@nonessentialprototype

1. For the image below, these red blocks should be updated in the config.yaml, right ?
   

PRODUCT=rke2 ./rotate-default-ca-certs.sh rke2 certificate rotate-ca --path=/var/lib/rancher/rke2/server/rotate-ca --server=https://127.0.0.1:9345 --force rke2 certificate rotate systemctl restart rke2-server

For your script you provided, do i need to run it on all master nodes ?

[nonessentialprototype]

Yes for 1
I believe it was just on the first node then would you update the masters configs and restart those.

[hdong69]

many thanks @nonessentialprototype