From bb10a1c8b878da54269890429e3c5a4a9abd71b8 Mon Sep 17 00:00:00 2001 From: Adam Dyess Date: Thu, 9 Jan 2025 02:23:03 -0600 Subject: [PATCH] Charm Upgrade documentation (#881) --------- Co-authored-by: Mateo Florido --- .../config/vocabularies/Canonical/accept.txt | 3 + docs/src/.custom_wordlist.txt | 3 + docs/src/charm/howto/index.md | 4 +- docs/src/charm/howto/upgrade-minor.md | 214 ++++++++++++++++++ docs/src/charm/howto/upgrade-patch.md | 204 +++++++++++++++++ docs/src/charm/howto/validate.md | 187 +++++++++++++++ 6 files changed, 614 insertions(+), 1 deletion(-) create mode 100644 docs/src/charm/howto/upgrade-minor.md create mode 100644 docs/src/charm/howto/upgrade-patch.md create mode 100644 docs/src/charm/howto/validate.md diff --git a/docs/canonicalk8s/styles/config/vocabularies/Canonical/accept.txt b/docs/canonicalk8s/styles/config/vocabularies/Canonical/accept.txt index f69bdb63b..178b966a9 100644 --- a/docs/canonicalk8s/styles/config/vocabularies/Canonical/accept.txt +++ b/docs/canonicalk8s/styles/config/vocabularies/Canonical/accept.txt @@ -221,6 +221,7 @@ linux Lite's LoadBalancer localhost +logfile Lookaside lookups loopback @@ -281,6 +282,7 @@ PPSs proc programmatically provisioner +Pre PRs PV qdisc @@ -339,6 +341,7 @@ UIDs unix unschedulable unsquashed +uuid Velero vf VF diff --git a/docs/src/.custom_wordlist.txt b/docs/src/.custom_wordlist.txt index 909e5f376..eb8c81834 100644 --- a/docs/src/.custom_wordlist.txt +++ b/docs/src/.custom_wordlist.txt @@ -147,6 +147,7 @@ linux Lite's LoadBalancer localhost +logfile Lookaside lookups loopback @@ -204,6 +205,7 @@ PPA proc programmatically provisioner +Pre PRs PV qdisc @@ -258,6 +260,7 @@ ubuntu unix unschedulable unsquashed +uuid Velero vf VF diff --git a/docs/src/charm/howto/index.md b/docs/src/charm/howto/index.md index 4c09218d0..0e271f306 100644 --- a/docs/src/charm/howto/index.md +++ b/docs/src/charm/howto/index.md @@ -26,7 +26,9 @@ proxy cos-lite contribute custom-registry - +Upgrade patch version +Upgrade minor version +Validate the cluster ``` --- diff --git a/docs/src/charm/howto/upgrade-minor.md b/docs/src/charm/howto/upgrade-minor.md new file mode 100644 index 000000000..5d3d614df --- /dev/null +++ b/docs/src/charm/howto/upgrade-minor.md @@ -0,0 +1,214 @@ +# How to Upgrade {{product}} to the next minor revision + +It is recommended that you keep your Kubernetes deployment +updated to the latest available stable version. You should +also update the other applications deployed in your Kubernetes +cluster. Keeping up-to-date ensures you have the latest bug-fixes +and security patches for smooth operation of your cluster. + +New minor versions of Kubernetes are set to release three +times per year. Check the latest release version +on the Kubernetes release page on GitHub. + +```{note} Kubernetes will not automatically handle minor +release upgrades. The cluster will not perform an unattended +automatic upgrade between minor versions, e.g. 1.30.1 to 1.31.0. +Attended upgrades are required when you wish to upgrade +whether to a patch or minor version. +``` + +Determine which version of each application is currently deployed by running: + + +```sh +juju status +``` + + +The ‘App’ section of the output lists each application and its +version number. Note that this is the version of the upstream +application deployed. The version of the Juju charm is indicated +under the column titled ‘Rev’. The charms may be updated in +between new versions of the application. + + +``` +Model Controller Cloud/Region Version SLA Timestamp +my-cluster canonicaws aws/us-east-1 3.6.0 unsupported 16:02:18-05:00 + +App Version Status Scale Charm Channel Rev Exposed Message +k8s 1.31.3 active 3 k8s 1.31/stable 123 yes Ready + +Unit Workload Agent Machine Public address Ports Message +k8s/0 active idle 0 54.89.153.117 6443/tcp Ready +k8s/1* active idle 1 3.238.230.3 6443/tcp Ready +k8s/2 active idle 2 34.229.202.243 6443/tcp Ready + +Machine State Address Inst id Base AZ Message +0 started 54.89.153.117 i-0b6fc845c28864913 ubuntu@22.04 us-east-1f running +1 started 3.238.230.3 i-05439714c88bea35f ubuntu@22.04 us-east-1f running +2 started 34.229.202.243 i-07ecf97ed29860334 ubuntu@22.04 us-east-1c running +``` + + + +## Before you begin + +As with all upgrades, there is a possibility that there may be +unforeseen difficulties. It is highly recommended to make +a backup of any important data, including any running workloads. +For more details on creating backups, see the separate +[docs on backups][backup-restore]. + + +Verify that: + +* The machine from which you will perform the backup has sufficient + internet access to retrieve updated software +* Your cluster is running normally +* Your Juju client and controller/models are running the same, + stable version of Juju (see the [Juju docs][juju-docs]) +* You read the [Upstream release notes][upstream-notes] for details + of Kubernetes deprecation notices and API changes that may impact + your workloads + + +It is also important to understand that Kubernetes will only +upgrade and if necessary migrate, components relating specifically +to elements of Kubernetes installed and configured as part of Kubernetes. +This may not include any customized configuration of Kubernetes, +or non-built-in generated objects (e.g. storage classes) or deployments which +rely on deprecated APIs. + +## Specific upgrade instructions + +### Deciding if an upgrade is available + +Juju will contact [charmhub] daily to find new revisions of charms +deployed in your models. To see if the `k8s` or `k8s-worker` charms +can be upgraded, set with the following: + +```sh +juju status --format=json | \ + jq '.applications | + to_entries[] | { + application: .key, + "charm-name": .value["charm-name"], + "charm-channel": .value["charm-channel"], + "charm-rev": .value["charm-rev"], + "can-upgrade-to": .value["can-upgrade-to"] + }' +``` + +This outputs a list of applications in the model: +* the name of the application (ex. `k8s`) +* the charm used by the application (ex. `k8s`) +* the kubernetes channel this charm follows (ex. `1.31/stable`) +* the current charm revision (ex. `1001`) +* the next potential charm revision (ex. `ch:amd64/k8s-1002`) + +If the `can-upgrade-to` revision is `null`, the charm is on the most +stable release within this channel. Continue with the +[Pre Upgrade Check](#the-pre-upgrade-check). + +Otherwise, complete the [Upgrade Patch](upgrade-patch) instructions first. + + +```{caution} Only update the charm to the next minor version. +If the current `charm-channel` is `1.31/stable`, it's critical +to refresh to the `1.32/stable`. Skipping channels (eg 1.31 -> 1.33) +will result in the units blocking and indicating they cannot upgrade. + +See Kubernetes' [Version Skew Policy][version-skew-policy] +``` + +### The pre-upgrade-check + +Before running an upgrade, check that the cluster is +steady and ready for upgrade. The charm will perform checks +necessary to confirm the cluster is in safe working order before +upgrading. + +```sh +juju run k8s/leader pre-upgrade-check +``` + +If no error appears, the `pre-upgrade-check` completed successfully. + +### Refreshing charm applications + +#### Control Plane units (k8s) + +Following the `pre-upgrade-check` update the control-plane nodes. + +```sh +juju refresh k8s --channel ${NEXT_CHANNEL} +juju status k8s --watch 5s +``` + +The `refresh` command instructs the juju controller to follow a new +charm `channel`. The Kubernetes charm will be upgraded to the lasted +revision within that channel. The charm code is simultaneously replaced +on each unit, then the `k8s` snap is updated unit-by-unit in order to +maintain a highly-available kube-api-server endpoint, starting with the +Juju leader unit for each application. + +During the upgrade process, the application status message and the +`k8s` leader unit message will display the current progress, +listing the `k8s` and `k8s-worker` units still pending upgrades. + +After the `k8s` charm is upgraded, the application `Version` from `juju status` +will reflect the updated version of the control-plane nodes making up the cluster. + +#### Worker units (k8s-worker) + +After updating the control-plane applications, worker nodes may be upgraded +by running the `pre-upgrade-check` action. + +```sh +juju run k8s-worker/leader pre-upgrade-check +juju refresh k8s-worker --channel ${NEXT_CHANNEL} +juju status k8s-worker --watch 5s +``` + +The `refresh` command instructs the juju controller to follow a new +charm channel related to the Kubernetes release and use the new charm +revision of the application's channel to upgrade each unit. The +charm code is simultaneously replaced on each unit, then the `k8s` +snap is updated unit-by-unit, starting with the Juju leader unit for the +application. + +After the `k8s-worker` charm is upgraded, the application `Version` from `juju status` +will reflect the updated version of the worker nodes making up the cluster. + +```{note} Repeat [this section](#worker-units-k8s-worker) for every +application using the k8s-worker charm, if multiple k8s-worker +applications appear in the same model. +``` + +## Verify an Upgrade + +Once an upgrade is complete, confirm the successful upgrade by running: + + +```sh +juju status +``` + + +... should indicate that all units are active/idle and the correct +version of **Kubernetes** is listed in the application's **Version** + +Run a [cluster validation][cluster-validation] +to ensure that the cluster is fully functional. + + + +[backup-restore]: ../../snap/howto/backup-restore +[charmhub]: https://charmhub.io/k8s +[cluster-validation]: ./validate +[juju-docs]: https://juju.is/docs/juju/upgrade-models +[release-notes]: ../reference/releases +[upgrade-notes]: ../reference/upgrade-notes +[upstream-notes]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.32.md#deprecation +[version-skew-policy]: https://kubernetes.io/releases/version-skew-policy/ diff --git a/docs/src/charm/howto/upgrade-patch.md b/docs/src/charm/howto/upgrade-patch.md new file mode 100644 index 000000000..4133a68a5 --- /dev/null +++ b/docs/src/charm/howto/upgrade-patch.md @@ -0,0 +1,204 @@ +# How to Upgrade {{product}} to the next patch revision + +It is recommended that you keep your Kubernetes deployment +updated to the latest available stable version. You should +also update the other applications deployed in your Kubernetes +cluster. Keeping up-to-date ensures you have the latest bug-fixes +and security patches for smooth operation of your cluster. + +Kubernetes releases patch versions approximately every month. These updates +focus on making minor improvements without introducing new features. You +can check the latest release version on the Kubernetes release page on GitHub. + +```{note} Kubernetes will not automatically handle patch +releases. The cluster will not perform an unattended automatic +upgrade between patch versions, e.g. 1.31.0 to 1.31.1. +Attended upgrades are required when you wish to upgrade +whether to a patch or minor version. +``` + +You can see which version of each application is currently deployed by running: + + +```sh +juju status +``` + + +The ‘App’ section of the output lists each application and its +version number. Note that this is the version of the upstream +application deployed. The version of the Juju charm is indicated +under the column titled ‘Rev’. The charms may be updated in +between new versions of the application. + + +``` +Model Controller Cloud/Region Version SLA Timestamp +my-cluster canonicaws aws/us-east-1 3.6.0 unsupported 16:02:18-05:00 + +App Version Status Scale Charm Channel Rev Exposed Message +k8s 1.31.3 active 3 k8s 1.31/stable 123 yes Ready + +Unit Workload Agent Machine Public address Ports Message +k8s/0 active idle 0 54.89.153.117 6443/tcp Ready +k8s/1* active idle 1 3.238.230.3 6443/tcp Ready +k8s/2 active idle 2 34.229.202.243 6443/tcp Ready + +Machine State Address Inst id Base AZ Message +0 started 54.89.153.117 i-0b6fc845c28864913 ubuntu@22.04 us-east-1f running +1 started 3.238.230.3 i-05439714c88bea35f ubuntu@22.04 us-east-1f running +2 started 34.229.202.243 i-07ecf97ed29860334 ubuntu@22.04 us-east-1c running +``` + + + +## Before you begin + +As with all upgrades, there is a possibility that there may be +unforeseen difficulties. It is highly recommended that you make +a backup of any important data, including any running workloads. +For more details on creating backups, see the separate +[docs on backups][backup-restore]. + + +You should also make sure: + +* The machine from which you will perform the backup has sufficient + internet access to retrieve updated software +* Your cluster is running normally +* Your Juju client and controller/models are running the same, + stable version of Juju (see the [Juju docs][juju-docs]) +* You read the [Upstream release notes][upstream-notes] for details + of Kubernetes deprecation notices and API changes that may impact + your workloads + + +It is also important to understand that Kubernetes will only +upgrade and if necessary migrate, components relating specifically +to elements of Kubernetes installed and configured as part of Kubernetes. +This may not include any customized configuration of Kubernetes, +or no-build-in generated objects (e.g. storage classes) or deployments which +rely on deprecated APIs. + +## Specific upgrade instructions + +### Deciding if an upgrade is available + +Juju will contact [charmhub] daily to find new revisions of charms +deployed in your models. To see if the `k8s` or `k8s-worker` charms +can be upgraded, set with the following: + +```sh +juju status --format=json | \ + jq '.applications | + to_entries[] | { + application: .key, + "charm-name": .value["charm-name"], + "charm-channel": .value["charm-channel"], + "charm-rev": .value["charm-rev"], + "can-upgrade-to": .value["can-upgrade-to"] + }' +``` + +This outputs list of applications in the model: +* the name of the application (ex. `k8s`) +* the charm used by the application (ex. `k8s`) +* the kubernetes channel this charm follows (ex. `1.31/stable`) +* the current charm revision (ex. `1001`) +* the next potential charm revision (ex. `ch:amd64/k8s-1002`) + +If the `can-upgrade-to` revision is `null`, the charm is on the most +stable release within this channel and there is no patch upgrade to +complete. + +Otherwise continue with the [Pre Upgrade Check](#the-pre-upgrade-check) + + +### The pre-upgrade-check + +Before running an upgrade, check that the cluster is +steady and ready for upgrade. The charm will perform checks +necessary to confirm the cluster is in safe working order before +upgrading. + +```sh +juju run k8s/leader pre-upgrade-check +``` + +If no error appears, the `pre-upgrade-check` completed successfully. + + +### Refreshing charm applications + +#### Control Plane units (k8s) + +Following the `pre-upgrade-check` update the control-plane nodes. + +```sh +juju refresh k8s +juju status k8s --watch 5s +``` + +The `refresh` command instructs the juju controller to use the new charm +revision within the current charm `channel`. The charm code is simultaneously +replaced on each unit, then the `k8s` snap is updated unit-by-unit in order +to maintain a highly-available kube-api-server endpoint, starting with the +Juju leader unit for the application. + +During the upgrade process, the application status message and the +`k8s` leader unit message will display the current progress, +listing the `k8s` and `k8s-worker` units still pending upgrades. + +After the `k8s` charm is upgraded, the application `Version` from `juju status` +will reflect the updated version of the control-plane nodes making up the cluster. + +#### Worker units (k8s-worker) + +After updating the control-plane applications, worker nodes may be upgraded +by running the `pre-upgrade-check` action. + +```sh +juju run k8s-worker/leader pre-upgrade-check +juju refresh k8s-worker +juju status k8s-worker --watch 5s +``` + +The `refresh` command instructs the juju controller to use the new charm +revision of the application's charm channel to upgrade each unit. The +charm code is simultaneously replaced on each unit, then the `k8s` +snap is updated unit-by-unit starting with the Juju leader unit for the +application. + +After the `k8s-worker` charm is upgraded, the application `Version` from `juju status` +will reflect the updated version of the worker nodes making up the cluster. + +```{note} Repeat [this section](#worker-units-k8s-worker) for every +application using the k8s-worker charm, if multiple k8s-worker +applications appear in the same model. +``` + + +## Verify an Upgrade + +Once an upgrade is complete, confirm the successful upgrade by running: + + +```sh +juju status +``` + + +... should indicate that all units are active/idle and the correct +version of **Kubernetes** is listed in the application's **Version** + +It is recommended that you run a [cluster validation][cluster-validation] +to ensure that the cluster is fully functional. + + + +[backup-restore]: ../../snap/howto/backup-restore +[charmhub]: https://charmhub.io/k8s +[cluster-validation]: ./validate +[juju-docs]: https://juju.is/docs/juju/upgrade-models +[release-notes]: ../reference/releases +[upstream-notes]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.32.md#deprecation diff --git a/docs/src/charm/howto/validate.md b/docs/src/charm/howto/validate.md new file mode 100644 index 000000000..bef5ae8d3 --- /dev/null +++ b/docs/src/charm/howto/validate.md @@ -0,0 +1,187 @@ +# How to Validate {{product}} + +End-to-end (e2e) tests for **Kubernetes** provide a mechanism +to test the behavior of the system. This is a useful indicator +that the cluster is performing properly, as well as a good +validation of any code changes. + +For **{{product}}**, these tests are encapsulated +in an additional **Juju** charm which can be added to your cluster. +Actual testing is then run through the charm's actions. + + +```{caution} Your cluster will need to have at least two +running worker units for the `e2e` test to run properly. +``` + +## Deploying the kubernetes-e2e charm + +Add the charm to your cluster: + +```bash +juju deploy kubernetes-e2e --constraints mem=4G +``` + +We also need to configure the charm to select the appropriate version of tests. +This relates to the installed version of Kubernetes. You can check which +version your cluster is set to by running: + +```bash +juju status k8s +``` + +The output will be in the form of `version.number/risk`, e.g `1.31/stable`. You should set +the `kubernetes-e2e` channel to the same value. + +``` +juju config kubernetes-e2e channel=1.31/stable +``` + +Finally we relate the charm to `k8s`: + +```bash +juju integrate kubernetes-e2e:kube-control k8s:kube-control +``` + +It may take some moments for these relations to establish. Once the connections are made, the charm will update its status to "Ready to test." + +## Running the default tests + +The tests are configured as a **Juju** _action_. To run the default tests: + +```bash +juju run kubernetes-e2e/0 test --background +``` + +The command will return with a number for that specific action operation. +```console +Scheduled operation 25 with task 26 +Check operation status with 'juju show-operation 25' +Check task status with 'juju show-task 26' +``` + +See the section on [Test output](#test-output) below for details. + +## Running specific tests + +The complete set of **Kubernetes** e2e tests is more fully described in the +[upstream Kubernetes documentation][e2e-upstream]. In some cases you may wish +to omit particular groups of tests. This is possible by applying a regular +expression defining the tests to omit when initiating the action. + +By default, the standard tests marked `[Flaky]` or `[Serial]` are skipped. To +also omit the tests marked as `[Slow]`, you could run: + +```bash +juju run kubernetes-e2e/0 test skip='\[(Flaky|Slow|Serial)\]' --background +``` + +Note that the brackets for the regex need to be escaped as shown. + +Running this command will return a uuid for that specific test run, as with the default case. + +## Test output + +You can check on the current status of the test by running: + +```bash +juju show-operation 25 +``` + +where `25` is the id of the scheduled operation when the test was initiated. +This will return YAML output indicating the current status, +which can be either `running`, `completed` or `failed`. + +```yaml +summary: test run on unit-kubernetes-e2e-0 +status: running +action: + name: test + parameters: {} +timing: + enqueued: 2023-03-13 11:01:34 -0500 CDT + started: 2023-03-13 11:01:34 -0500 CDT +tasks: + "26": + host: kubernetes-e2e/0 + status: running + timing: + enqueued: 2023-03-13 11:01:34 -0500 CDT + started: 2023-03-13 11:01:34 -0500 CDT +``` + +Once completed, you can see more detail on the timing by running: + +```bash +juju show-operation 25 +``` + +Which will return output similar to: + +```yaml +summary: test run on unit-kubernetes-e2e-0 +status: running +action: + name: test + parameters: {} +timing: + enqueued: 2023-03-13 11:01:34 -0500 CDT + started: 2023-03-13 11:01:34 -0500 CDT + completed: 2023-03-13 11:10:15 -0500 CDT +tasks: + "26": + host: kubernetes-e2e/0 + status: completed + timing: + enqueued: 2023-03-13 11:01:34 -0500 CDT + started: 2023-03-13 11:01:34 -0500 CDT + completed: 2023-03-13 11:10:15 -0500 CDT + results: + junit: /home/ubuntu/26-junit.tar.gz + log: /home/ubuntu/26.log.tar.gz +``` + +If the tests fail, or you want to look through the detail of each test, you can examine the +detailed log. + +## Viewing test logs + +The test logfile is stored as a file on the test instance. The filename +corresponds to the id of the action which created it, with a '.log' +extension, and it is stored in the `/home/ubuntu/` directory of the machine +where the tests are running. A compressed version is also stored with the +extension `.log.tar.gz` + +This log can be copied to your local machine for easier viewing: + +```bash +juju scp kubernetes-e2e/0:26.log . +``` + +Note that the captured test logfile uses ANSI output, and is best viewed with +`cat`, `tail` or a similar command which can handle this type of output. +Alternatively, you can strip the ANSI parts of the output: + +```shell +fn= +echo -ne $(cat $fn | sed 's/$/\\n/' | sed 's/\x1B\[[0-9]*\w//g') +``` + +```{note} If you are running regular tests in this way, +it is advisable to remove the generated logs from the test unit. +The uncompressed logs in particular can be very large and quickly +fill up storage. +``` + +## Upgrading the e2e tests + +When an update is available, the `kubernetes-e2e` charm can be upgraded with the command: + +```bash +juju refresh kubernetes-e2e --channel=${release} +``` + + + +[e2e-upstream]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md +