Old Secrets Deleted on Failed Update in BerglasSecretController

[Hi-king]

Description

We've encountered a critical issue with the BerglasSecretController in our Kubernetes setup where old secrets are deleted whenever an update fails. This behavior results in an inability to roll back to a previous state since the old secrets are no longer available.

Steps to Reproduce

1. Deploy an incorrectly configured BerglasSecret (e.g., invalid permissions).
2. Introduce a failure in the update process.
3. Observe that the old secret is deleted despite the update failure.

Expected Behavior

The expected behavior would be that if an update fails, the old secrets should not be deleted, preserving the ability to roll back to a previous state.

Actual Behavior

When an update fails, the old secrets are deleted, leaving no way to recover the previous secret values.

Possible Solution

A potential solution could be to implement a transactional mechanism where the old secrets are only deleted upon a successful update.

berglas-secret-controller/controllers/secret.go

Line 105 in dc69a1c

err = r.Delete(ctx, secret)

Here, the deletion is followed by a create operation, and failure at this step is problematic. Could we consider using a create-then-replace approach to simulate a transaction-like mechanism?