Make PostgreSQL PVC storage requirements immutable

[rooftopcellist]

SUMMARY

k8s does not allow a user to modify a PVC's size after it has been created, so we should prevent users from trying to change this on the AWX CR after the initial deployment. Currently, if a user does change the requests or limits on the db PVC, the operator will crashloop with the following error:

Forbidden: updates to statefulset spec for fields other than \\'replicas\\', \\'template\\', \\'updateStrategy\\', \\'persistentVolumeClaimRetentionPolicy\\' and \\'minReadySeconds\\' are forbidden\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity"}      

ISSUE TYPE

• Bug, Docs Fix or other nominal change

[rooftopcellist]

I added in this so that make bundle doesn't create diff for it every time.

[tylergmuir]

I ran into this same error that you mentioned and documented it in #1642. That said, I think we should do the reverse and still allow it in the CRD, but then in the ansible code that the operator runs, have it better handle that change to the PVC. There is already error catching in that step, the issue is that the error catching doesn't quite work as expected.

The issue I see with making this immutable is that there is no process to increase the size without doing a lot of manual work. Alternatively, if the field is mutable and the error handling in the playbook works as expected, then you still need to manually resize the PVC (due to Kubernetes not automatically resizing PVC from statefulstatesets) but then you have a resized PVC.

If we do decide to go the route of making the fields immutable, we need to also include much better documentation on how to migrate to a larger PVC.

[rooftopcellist]

Marking this as a draft PR for now because I think we need to workshop how best to handle this and weigh up our options more.

[rooftopcellist]

@tylergmuir You bring up a good point, if a user has a storage class that does support dynamic expansion, this would make things more difficult for them to utilize it.

It would be great to take the time to figure out the steps needed for expanding the db PVC and document them in a markdown file here in the repo. I think we would want to cover two scenarios:

A) How to migrate if your storageclass supports dynamic expansion (currently modify the AWX CR spec postgres_storage_requirements setting, then delete the StatefulSet and wait for the next reconciliation loop to re-create it)

B) How to use the migration.md logic to migrate the awx instance to a fresh instance and provision a larger PVC in the process.

I will close this PR and open an issue for adding documentation for this. It would be nice to find a way to prevent users without SC's that support dynamic expansion from falling into this reconcilation fail loop... Maybe the option described here could help with that.

[rooftopcellist]

I created an issue for documenting the steps currently needed to expand a PVC.

• Add Documentation for expanding the DB PVC of an existing AWX #1655