Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InvalidBackingStoreScaling Error on infra, acm-metrics-backing-store #745

Closed
5 tasks done
schwesig opened this issue Sep 23, 2024 · 1 comment
Closed
5 tasks done

InvalidBackingStoreScaling Error on infra, acm-metrics-backing-store #745

schwesig opened this issue Sep 23, 2024 · 1 comment
Assignees
@schwesig
Labels
bug Something isn't working observability openshift This issue pertains to NERC OpenShift

Comments

@schwesig
Copy link
Member

schwesig commented Sep 23, 2024 •
edited
Loading

InvalidBackingStoreScaling Error on infra, acm-metrics-backing-store

Motivation

The backing store ("acm-metrics-backing-store") using ocs-external-storagecluster-ceph-rbd StorageClass is showing an "InvalidBackingStoreScaling" error.
We’re not trying to scale down anything. And deleting the storage and restarting didn't solve anything and showed that there are some persistent naming issues.
After working with Red Hat support and internal calls, we decided to try creating a new backing store with a different name and see if the issue happens again.

Completion Criteria

  • A new backing store with a different name has been set up.
  • The scaling down error doesn’t appear anymore, or shows up again to search to "the real problem".

Description

Completion dates

Desired - 2024-09-25
Required - 2024-09-27

Following this to create acm-metrics at the beginning

  1. PR 462
    changed files:

    • cluster-scope/overlays/nerc-ocp-infra/acm-metrics-storage/backingstore.yaml
    • cluster-scope/overlays/nerc-ocp-infra/acm-metrics-storage/bucketclass.yaml
    • cluster-scope/overlays/nerc-ocp-infra/acm-metrics-storage/kustomization.yaml
    • cluster-scope/overlays/nerc-ocp-infra/acm-metrics-storage/storageclass.yaml
    • cluster-scope/overlays/nerc-ocp-infra/kustomization.yaml

  2. PR 463
    changed files:

    • cluster-scope/base/objectbucket.io/objectbucketclaims/acm-metrics/kustomization.yaml
    • cluster-scope/base/objectbucket.io/objectbucketclaims/acm-metrics/objectbucketclaim.yaml
    • cluster-scope/bundles/acm-observability/kustomization.yaml

  3. PR 473
    changed files:

    • cluster-scope/base/external-secrets.io/externalsecrets/open-cluster-management-observability-acm-metrics-object-storage/externalsecret.yaml
    • cluster-scope/base/external-secrets.io/externalsecrets/open-cluster-management-observability-acm-metrics-object-storage/kustomization.yaml
    • cluster-scope/bundles/acm-observability/kustomization.yaml
    • cluster-scope/overlays/nerc-ocp-infra/externalsecrets/open-cluster-management-observability-acm-metrics-object-storage_patch.yaml
    • cluster-scope/overlays/nerc-ocp-infra/kustomization.yaml

  4. PR 474
    changed files:

    • cluster-scope/base/observability.open-cluster-management.io/multiclusterobservabilities/observability/multiclusterobservability.yaml

  5. PR 477
    changed files:

    • cluster-scope/overlays/nerc-ocp-infra/externalsecrets/open-cluster-management-observability-acm-metrics-object-storage_patch.yaml

  6. PR 493
    changed files:

    • cluster-scope/overlays/nerc-ocp-infra/acm-metrics-storage/backingstore.yaml
@schwesig schwesig added bug Something isn't working openshift This issue pertains to NERC OpenShift observability labels Sep 23, 2024
@schwesig schwesig self-assigned this Sep 23, 2024
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 23, 2024
@schwesig
Add ACM metrics 2ti object bucket claim & storage
73aa567 
fixes partly nerc-project/operations#745

This PR tackles part of the issue by adding acm-metrics-2ti and updating existing observability and storage settings.
The changes try to solve or reproduce the errors in #745.
But this is only the first step, see #745.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig schwesig mentioned this issue Sep 23, 2024
Merged
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 23, 2024
@schwesig
Add ACM metrics 2ti object bucket claim & storage
6bbc817 
fixes partly nerc-project/operations#745

This PR tackles part of the issue by adding acm-metrics-2ti and updating existing observability and storage settings.
The changes try to solve or reproduce the errors in #745.
But this is only the first step, see #745.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 23, 2024
@schwesig
Add ACM metrics 2ti object bucket claim & storage
b0a9c45 
fixes partly nerc-project/operations#745

This PR tackles part of the issue by adding acm-metrics-2ti and updating existing observability and storage settings.
The changes try to solve or reproduce the errors in #745.
But this is only the first step, see #745.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
schwesig added a commit to OCP-on-NERC/nerc-ocp-config that referenced this issue Sep 23, 2024
@schwesig
Add ACM metrics 2ti object bucket claim & storage (#540)
be96fcc 
fixes partly nerc-project/operations#745

This PR tackles part of the issue by adding acm-metrics-2ti and updating existing observability and storage settings.
The changes try to solve or reproduce the errors in #745.
But this is only the first step, see #745.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig schwesig mentioned this issue Sep 25, 2024
Closed
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 26, 2024
@schwesig
feat: add ExtSecret for acm-metrics-2Ti objstore
0214dfe 
partly fixing nerc-project/operations#745

- New ExternalSecret for ACM metrics 2Ti object storage in `open-cluster-management-observability` namespace.
- Updated the ACM observability bundle to include the new resource in the kustomization.
- Patch for the infra overlay to map the bucket, endpoint, access_key, and secret_key for ACM metrics 2Ti object storage.
- Manually, outside of github: added bucket, endpoint, access_key, and secret_key to vault infra.

This new storage is for testing our problems with nooba, scaling storage pools, and ODF operator.
nerc-project/operations#688
To reproduce the error or solve it.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig schwesig mentioned this issue Sep 26, 2024
Merged
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 26, 2024
@schwesig
feat: add ExtSecret for acm-metrics-2Ti objstore
22bb634 
partly fixing nerc-project/operations#745

- New ExternalSecret for ACM metrics 2Ti object storage in `open-cluster-management-observability` namespace.
- Updated the ACM observability bundle to include the new resource in the kustomization.
- Patch for the infra overlay to map the bucket, endpoint, access_key, and secret_key for ACM metrics 2Ti object storage.
- Manually, outside of github: added bucket, endpoint, access_key, and secret_key to vault infra.

This new storage is for testing our problems with nooba, scaling storage pools, and ODF operator.
nerc-project/operations#688
To reproduce the error or solve it.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 26, 2024
@schwesig
feat: switch to acm-metrics-2ti object storage
99b2399 
partly fixing nerc-project/operations#745

Update the configuration for ACM metrics object storage to use
- the acm-metrics-2ti.yaml file
- and the acm-metrics-2ti-object-storage name.

This new storage is for testing our problems with nooba, scaling storage pools, and ODF operator.
nerc-project/operations#688
To reproduce the error or solve it.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig schwesig mentioned this issue Sep 26, 2024
Merged
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Sep 26, 2024
@schwesig
feat: switch to acm-metrics-2ti object storage
d544346 
partly fixing nerc-project/operations#745

Update the configuration for ACM metrics object storage to use
- the acm-metrics-2ti.yaml file
- and the acm-metrics-2ti-object-storage name.

This new storage is for testing our problems with nooba, scaling storage pools, and ODF operator.
nerc-project/operations#688
To reproduce the error or solve it.

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig
Copy link
Member Author

schwesig commented Oct 1, 2024

follow up in #751

@schwesig schwesig closed this as completed Oct 1, 2024
@schwesig schwesig mentioned this issue Oct 1, 2024
Closed
9 tasks
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Oct 7, 2024
@schwesig
cleanup: after creating acm-metrics-2ti
1a67ffa 
- reversing acm-metrics storage, after:
- succcesfully created new acm-metrics-2ti
  - nerc-project/operations#745
- steps from:
  - OCP-on-NERC#462
  - OCP-on-NERC#463
  - OCP-on-NERC#473
  - OCP-on-NERC#474
  - OCP-on-NERC#477
  - OCP-on-NERC#493

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig schwesig mentioned this issue Oct 7, 2024
Merged
schwesig added a commit to schwesig/OCP-on-NERC_nerc-ocp-config that referenced this issue Oct 7, 2024
@schwesig
cleanup: after creating acm-metrics-2ti
327db08 
- reversing acm-metrics storage, after:
- succcesfully created new acm-metrics-2ti
  - nerc-project/operations#745
- steps from:
  - OCP-on-NERC#462
  - OCP-on-NERC#463
  - OCP-on-NERC#473
  - OCP-on-NERC#474
  - OCP-on-NERC#477
  - OCP-on-NERC#493
 - also deleted: vault key
   - nerc-ocp-infra/open-cluster-management-observability/acm-metrics/object-storage

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
schwesig added a commit to OCP-on-NERC/nerc-ocp-config that referenced this issue Oct 7, 2024
@schwesig
cleanup: after creating acm-metrics-2ti (#559)
602644c 
- reversing acm-metrics storage, after:
- succcesfully created new acm-metrics-2ti
  - nerc-project/operations#745
- steps from:
  - #462
  - #463
  - #473
  - #474
  - #477
  - #493
 - also deleted: vault key
   - nerc-ocp-infra/open-cluster-management-observability/acm-metrics/object-storage

Signed-off-by: ​/Thor(sten)?/ Schwesig <[email protected]>
@schwesig schwesig mentioned this issue Oct 31, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@schwesig schwesig

Labels
bug Something isn't working observability openshift This issue pertains to NERC OpenShift
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@schwesig