docs: how-to-purge-data (#473)

* docs: how-to-purge-data

Add document how to purge data from nova database shadow tables.

docs/.custom_wordlist.txt

@@ -48,3 +48,10 @@ VMs
 VPN
 websockets
 yaml
+STSStack
+sshuttle
+vpn
+SQLAlchemy
+Caracal
+Xena
+opendev

docs/.wordlist.txt

@@ -44,6 +44,3 @@ UI
 UUID
 VM
 YAML
-STSStack
-sshuttle
-vpn

docs/explanation/index.rst

@@ -12,3 +12,4 @@ and the implementation.
    planning
    upgrade
    upgrade-groups
+   nova-data-migration

docs/explanation/nova-data-migration.rst

@@ -0,0 +1,85 @@
+==========================================
+Data Migration on nova-cloud-controller
+==========================================
+
+This document explains the details of the database migration for nova-cloud-controller.
+
+nova-cloud-controller Database Migration Details
+------------------------------------------------
+
+When users upgrade the Juju application nova-cloud-controller, the commands **nova-manage db sync** and **nova-manage db online_data_migration** will be executed.
+
+* **nova-manage db sync** - Upgrades the main database schema to the most recent version.
+* **nova-manage db online_data_migration** - Performs data migration to update all live data.
+
+db sync
+~~~~~~~
+
+Before the Yoga version, nova-cloud-controller used SQLAlchemy to handle database migrations. Starting from Yoga, it switched to Alembic, a migrations tool for SQLAlchemy.
+
+Below is a table showing the migration versions from Ussuri to Caracal.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Version
+     - Migration Version
+   * - Ussuri
+     - 407
+   * - Victoria
+     - 412
+   * - Wallaby
+     - 417
+   * - Xena
+     - 422
+   * - Yoga
+     - 422
+       - (move to Alembic)
+       - ``8f2f1571d55b_initial_version``
+       - ``16f1fbcab42b_resolve_shadow_table_diffs``
+   * - Zed
+     - ``ccb0fa1a2252_add_encryption_fields_to``
+   * - 2023.1 (Antelope)
+     - ``960aac0e09ea_de_duplicate_indexes_in_instances``
+   * - 2023.2 (Bobcat)
+     - ``1acf2c98e646_add_compute_id_to_instance``
+       - ``1b91788ec3a6_drop_legacy_migrate_version_table``
+   * - 2024.1 (Caracal)
+     - ``13863f4e1612_create_share_mapping_table``
+
+Details of each migration can be found at:
+
+- `unmaintained/yoga - nova/nova/db/main/legacy_migrations/versions`_
+- `2024.1 - nova/nova/db/main/migrations/versions`_
+
+db online_data_migration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The list of online data migrations can be found at `nova_online_migrations`_.
+There are only two online migration cases after Victoria:
+
+- ``pci_device_obj.PciDevice.populate_dev_uuids``, added in Victoria
+- ``instance_obj.populate_instance_compute_id``, added in 2023.2
+
+On COU
+------
+
+Generally, the data migration operation load is not too high, as observed from previous information. COU provides two optional steps, **purge** and **archive**, to reduce the possible load during the upgrade. These two COU steps run the following commands on the nova-cloud-controller Juju unit:
+
+* **db archive_deleted_rows** - Run during the archive step; this command moves deleted rows from production tables to shadow tables.
+* **db purge** - Run during the purge step; this command deletes rows from shadow tables.
+
+Make sure to check the details of database schema migrations and online data migrations before each upgrade.
+
+More Information
+----------------
+
+- `opendev/openstack/nova`_
+- :doc:`Archive old data <../how-to/archive-old-data>`
+- :doc:`Purge data on shadow table <../how-to/purge-data-on-shadow-table>`
+
+.. LINKS
+.. _unmaintained/yoga - nova/nova/db/main/legacy_migrations/versions: https://opendev.org/openstack/nova/src/branch/unmaintained/yoga/nova/db/main/legacy_migrations/versions
+.. _2024.1 - nova/nova/db/main/migrations/versions: https://opendev.org/openstack/nova/src/branch/stable/2024.1/nova/db/main/migrations/versions
+.. _opendev/openstack/nova: https://opendev.org/openstack/nova
+.. _nova_online_migrations: https://opendev.org/openstack/nova/src/commit/fcda90460f6831b67027c19ded655b5e7c5e5a1e/nova/cmd/manage.py#L195

docs/how-to/archive-old-data.rst

@@ -54,6 +54,8 @@ More information
 - `nova-cloud-controller charm actions`_
 - `nova-manage reference`_ - see `archive_deleted_rows` subcommand
 - OpenStack upgrade guide information on `archiving old database data`_
+- :doc:`Purge data on shadow table <purge-data-on-shadow-table>`
+- :doc:`Nova data migration <../explanation/nova-data-migration>`
 
 
 .. LINKS

docs/how-to/index.rst

@@ -17,3 +17,4 @@ are possible with **COU**.
    no-backup
    archive-old-data
    verbosity
+   purge-data-on-shadow-table

docs/how-to/purge-data-on-shadow-table.rst

@@ -0,0 +1,30 @@
+==========================================
+Purge data on shadow tables
+==========================================
+
+The purge step is disable by default. When enabled, it will run before proceeding to actual upgrade steps.
+This can be enabled with the ``--purge`` flag.
+
+This purge step is a performance optimisation, delete data from the shadow tables in nova database. The behaviour is equal to run juju action ``purge`` on nova-cloud-controller unit, which help to reduce the size of the database, make queries faster, backups efficiency, and follow the data retention policies.
+
+The ``purge-before-date`` flag is supported to delete the data older than the date provided. The date string format should be ``YYYY-MM-DD[HH:mm][:ss]``. This flag is useful to retain recent data for debugging or data retention policies.
+
+
+Usage examples
+--------------
+
+.. terminal::
+   :input cou plan --purge --purge-before-date 2000-01-02
+
+
+More information
+----------------
+
+- `nova-cloud-controller charm actions`_
+- `nova-manage reference`_ - see `purge` subcommand
+- :doc:`Archive old data <archive-old-data>`
+- :doc:`Nova data migration <../explanation/nova-data-migration>`
+
+.. LINKS
+.. _nova-cloud-controller charm actions: https://charmhub.io/nova-cloud-controller/actions
+.. _nova-manage reference: https://docs.openstack.org/nova/rocky/cli/nova-manage.html