Helm install is not completing but velero is installed

[lsacco-nutreense]

What steps did you take and what happened:

I recently updated my terraform helm install to upgrade to 7.1.0 version of Velero. After making modications for the new parameters, I can get it to install, but it will fail my terraform build regardless how I modify it. I then tried to reinstall Velero outside of terraform and experience a similar issue with helm install. Finally, after a semisuccessful helm install, I created the backupStorageLocation and volumeSnapshotLocation as separate config yaml and applied them successfully with kubectl. This successfully installs it albeit with manual intervention.

Here's the configuration I used in Terraform to attempt the helm install:

resource "helm_release" "velero" {
  chart            = "velero"
  name             = "velero"
  repository       = "https://vmware-tanzu.github.io/helm-charts"
  namespace        = "velero"
  create_namespace = true
  version          = var.velero_helm_chart_version

  set {
    name  = "initContainers[0].name"
    value = "velero-plugin-for-aws"
  }

  set {
    name  = "initContainers[0].image"
    value = "velero/velero-plugin-for-aws:v1.10.0"
  }

  set {
    name  = "initContainers[0].volumeMounts[0].mountPath"
    value = "/target"
  }

  set {
    name  = "initContainers[0].volumeMounts[0].name"
    value = "plugins"
  }

  set {
    name  = "initContainers[0].imagePullPolicy"
    value = "IfNotPresent"
  }

  set {
    name  = "configuration.backupStorageLocation[0].bucket"
    value = var.eks_backup_bucket
  }

  set {
    name  = "configuration.backupStorageLocation[0].provider"
    value = "aws"
  }

  set {
    name  = "configuration.backupStorageLocation[0].config.region"
    value = var.aws_region
  }

  set {
    name  = "configuration.backupStorageLocation[0].config.profile"
    value = "default"
  }

  set {
    name  = "configuration.volumeSnapshotLocations[0].name"
    value = "default"
  }

  set {
    name  = "configuration.volumeSnapshotLocations[0].provider"
    value = "aws"
  }

  set {
    name  = "configuration.volumeSnapshotLocations[0].config.region"
    value = var.aws_region
  }

  set {
    name  = "configuration.volumeSnapshotLocations[0].config.profile"
    value = "default"
  }

  set {
    name  = "securityContext.fsGroup"
    value = "1337"
  }

  set {
    name  = "installCRDs"
    value = "true"
  }

  set {
    name  = "rbac.create"
    value = "true"
  }

  set {
    name  = "rbac.clusterAdministrator"
    value = "true"
  }

  set {
    name  = "serviceAccount.server.create"
    value = "true"
  }

  set {
    name  = "serviceAccount.server.name"
    value = "velero"
  }

  set {
    name  = "serviceAccount.server.annotations.eks\\.amazonaws\\.com/role-arn"
    value = var.velero_role_arn
  }

  set {
    name  = "credentials.useSecret"
    value = "false"
  }

  set {
    name  = "backupsEnabled"
    value = "true"
  }

  set {
    name  = "snapshotsEnabled"
    value = "true"
  }

  set {
    name  = "deployNodeAgent"
    value = "false"
  }

  set {
    name  = "image.repository"
    value = "velero/velero"
  }

  set {
    name  = "image.tag"
    value = var.velero_image_version
  }
}

Here's my the error I see from helm:

$ helm install velero vmware-tanzu/velero --namespace velero -f velero-values.yaml

Error: INSTALLATION FAILED: 2 errors occurred:
	* BackupStorageLocation.velero.io "default" is invalid: [spec.objectStorage.bucket: Required value, spec.provider: Required value]
	* VolumeSnapshotLocation.velero.io "default" is invalid: spec.provider: Required value

Here's my yaml file:

initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.10.0
    volumeMounts:
      - mountPath: /target
        name: plugins

configuration:
  backupStorageLocations:
    - name: default
      spec:
        provider: aws
        objectStorage:
          bucket: usw2-nonprod-eks-backup
        config:
          region: us-west-2
          profile: "default"
  volumeSnapshotLocations:
    - name: default
      spec:
        provider: aws
        config:
          region: us-west-2
          profile: "default"

serviceAccount:
  server:
    create: false
    name: velero

credentials:
  useSecret: false

These are the two yaml's I successfully applied after unsuccessfully installing them using helm and notice how there is no difference:

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: usw2-nonprod-eks-backup
  config:
    region: us-west-2
    s3ForcePathStyle: "true"
-----
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  config:
    region: us-west-2

What did you expect to happen:

I would have expected Velero to install like it had previously using terraform helm script. I would love to know if something is misconfigured in my terraform script above, but judging by the separate helm install I don't think that is the case unless the values I specified are wrong as well.

The following information will help us better understand what's going on:

There is no issue with backups, only install.

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero

time="2024-07-29T19:00:39Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-07-29T19:00:39Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=79954 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-07-29T19:00:55Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=79962 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-07-29T19:01:48Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141"
time="2024-07-29T19:01:48Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-07-29T19:01:48Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=79969 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-07-29T19:01:55Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=79976 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-07-29T19:02:48Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141"
time="2024-07-29T19:02:49Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-07-29T19:02:49Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=79984 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-07-29T19:02:55Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=79992 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-07-29T19:03:55Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=80000 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws

The following commands are pertinent because the backups run fine after manual installation.

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):

velero version
Client:
	Version: v1.14.0
	Git commit: -
Server:
	Version: v1.14.0

• Velero features (use velero client config get features):

features: <NOT SET>

• Kubernetes version (use kubectl version):

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9-eks-036c24b

• Kubernetes installer & version: brew
• Cloud provider or hardware configuration: AWS EKS 1.28 Kubernetes
• OS (e.g. from /etc/os-release): AWS Linux 2023

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

[raviumishra]

Using Helm install
helm install velero vmware-tanzu/velero --namespace velero --create-namespace --values ./values.yml

credentials:
  secretContents:
    cloud: ./credentials-velero.txt

backupStorageLocations:
  - name: velerobackups
    bucket: velerobackups # Make sure this matches your Azure Blob container name
    provider: azure
    config:
      resourceGroup: rg-name
      storageAccount: velerobackups
      subscriptionId: ************
      location: eastus

volumeSnapshotLocations:
  - name: velerobackups
    provider: azure
    config:
      resourceGroup: rg-name
      storageAccount: velerobackups
      subscriptionId: ************
      location: eastus

Error: INSTALLATION FAILED: 2 errors occurred:
        * BackupStorageLocation.velero.io "default" is invalid: [spec.objectStorage.bucket: Required value, spec.provider: Required value]
        * VolumeSnapshotLocation.velero.io "default" is invalid: spec.provider: Required value

[raviumishra]

@Lyndon-Li any ideas ?

[vkukk]

Still happens with latest Helm install

$ helm install velero vmware-tanzu/velero --namespace velero --create-namespace --set-file credentials.secretContents.cloud=secret/credentials --values values.yaml 
Error: INSTALLATION FAILED: 2 errors occurred:
        * BackupStorageLocation.velero.io "default" is invalid: [spec.objectStorage.bucket: Required value, spec.provider: Required value]
        * VolumeSnapshotLocation.velero.io "default" is invalid: spec.provider: Required value

[steled]

Hi,

I had a similar problem and IIRC I needed to set defaultBackupStorageLocation and defaultVolumeSnapshotLocations with the name configured here.

Maybe this helps.

[adamcirillo]

i'm getting the same error too trying today

 ~  helm upgrade --install velero vmware-tanzu/velero `
>   --namespace velero --create-namespace `
>   --set-file credentials.secretContents.cloud=./credentials-velero `
>   -f velero-values.yaml
Release "velero" does not exist. Installing it now.
Error: 2 errors occurred:
        * BackupStorageLocation.velero.io "default" is invalid: [spec.objectStorage.bucket: Required value, spec.provider: Required value]
        * VolumeSnapshotLocation.velero.io "default" is invalid: spec.provider: Required value

helm values

configuration:
  backupStorageLocations:
    - name: default
      provider: "aws"
      objectStorage:
        bucket: "bucket"
      config:
        profile: "default"
        region: "us-west-004"
        s3ForcePathStyle: "true"
        s3Url: "https://s3.us-west-004.backblazeb2.com"
  volumeSnapshotLocations:
    - name: default
      provider: "aws"
      config:
        region: "us-west-004"
        s3ForcePathStyle: "true"
        s3Url: "https://s3.us-west-004.backblazeb2.com"
initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.11.1
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

[adamcirillo]

i tried adding the config in as crds it worked correctly so to me it seems something is wrong with the helm chart

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: bucket
  config:
    region: us-west-004
    s3ForcePathStyle: "true"
    s3Url: "https://s3.us-west-004.backblazeb2.com"
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  config:
    region: us-west-004
    s3ForcePathStyle: "true"
    s3Url: "https://s3.us-west-004.backblazeb2.com"