Getting Warning message "No volume ID returned by volume snapshotter for persistent volume"

[davidmkrtchian]

What steps did you take and what happened:
I want to create backups for my PVC using CSI driver as provided in documentation.
https://velero.io/docs/v1.13/csi/

I am installing Velero with this configuration and with 2 init container with images velero/velero-plugin-for-aws:v1.9.0 and velero/velero-plugin-for-csi:v0.7.0

configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: xxx-backups
      prefix: rax/sandbox/volume-velero-test/
      default: true
      accessMode: ReadWrite
      config:
        region: us-east-1 
  volumeSnapshotLocation:
  - provider: aws
    config:
      region: us-east-1    
defaultSnapshotMoveData:
  features: EnableCSI  

Here is how I am creating my PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: velero-test
  namespace: velero
  annotations:
    velero.io/csi-volumesnapshot-class: "test-snapclass"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi  
  storageClassName: netapp-nfs3-aggr01   

After that by following documentation I am creating an VolumeSnapshotClass using driver csi.trident.netapp.io

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  namespace: velero
  name: test-snapclass
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: csi.trident.netapp.io
deletionPolicy: Delete  

After that trying to create an backup with this configuration

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: backup
  namespace: velero
  annotations:
    velero.io/csi-volumesnapshot-class_disk.csi.cloud.com: "test-snapclass"
spec:
  includedNamespaces:
    - velero

The Backup is completing but I am getting this

Cluster: resource: /persistentvolumes name: /pvc-337f3234-a829-49ff-b220-20296ad8d64f message: /No volume ID returned by volume snapshotter for persistent volume

What did you expect to happen:
I understand that by approaching it this way, I am backing up the entire namespace rather than just the PVC or PV. However, my initial priority is to address this issue before moving forward with the backup process.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help
bundle-2024-02-13-16-27-33.tar.gz

[shubham-pampattiwar]

Can you also paste the yaml spec corresponding to the storageclass (netapp-nfs3-aggr01 ) used for creating the PVC ?

[davidmkrtchian]

Can you also paste the yaml spec corresponding to the storageclass (netapp-nfs3-aggr01 ) used for creating the PVC ?

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-nfs3-aggr01
  uid: d3c895fe-7ebf-4cfd-a89e-6f32430a4cca
  resourceVersion: '594498'
  creationTimestamp: '2022-11-03T12:17:09Z'
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
  managedFields:
    - manager: HashiCorp
      operation: Update
      apiVersion: storage.k8s.io/v1
      time: '2022-11-03T12:17:09Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:allowVolumeExpansion: {}
        f:metadata:
          f:annotations:
            .: {}
            f:storageclass.kubernetes.io/is-default-class: {}
        f:parameters:
          .: {}
          f:backendType: {}
          f:fsType: {}
          f:media: {}
          f:provisioningType: {}
          f:snapshots: {}
          f:storagePools: {}
        f:provisioner: {}
        f:reclaimPolicy: {}
        f:volumeBindingMode: {}
  selfLink: /apis/storage.k8s.io/v1/storageclasses/netapp-nfs3-aggr01
provisioner: csi.trident.netapp.io
parameters:
  backendType: ontap-nas
  fsType: xfs
  media: ssd
  provisioningType: thin
  snapshots: 'true'
  storagePools: netapp-nfs3-aggr01:aggr01_n01_SSD
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

[ywk253100]

Seems you have an incorrect indent for features: EnableCSI which caused the CSI isn't enabled so the native snapshot mechanism is used to back up the PVs.

Remove the indent to enable the CSI feature and try it again:

defaultSnapshotMoveData:
features: EnableCSI 

[davidmkrtchian]

Seems you have an incorrect indent for features: EnableCSI which caused the CSI isn't enabled so the native snapshot mechanism is used to back up the PVs.

Remove the indent to enable the CSI feature and try it again:

defaultSnapshotMoveData:
features: EnableCSI 

I apologize for my delayed response.

Thank you for your reply. I've corrected the mistake, but there seems to be no change; I'm still receiving "No Volume ID" after creating the backup.
I would be grateful if you could suggest any other solutions.

[ywk253100]

Please run the velero debug command again and upload the debug bundle

[davidmkrtchian]

Please run the velero debug command again and upload the debug bundle

I made some changes in my code and got the next situation. When the configuration is looking in this way

 configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: bostongene-backups
      prefix: rax/sandbox/volume-velero-test/
      default: true
      accessMode: ReadWrite
      config:
        region: us-east-1
  volumeSnapshotLocation:
  - provider: aws
    config:
      region: us-east-1
  features: EnableCSI  (This feature is with indent) 

I can see the features: EnableCSI in the arguments for my deployment. Only in this way it is working.

      args:
        - server
        - '--uploader-type=restic'
        - '--features=EnableCSI'

But after creating backup, it is failing with timeout
Waiting for CSI driver to reconcile volumesnapshot velero/velero-velero-test-lfsrd. Retrying in 5s

Also here is the bundle file

bundle-2024-02-21-22-12-16.tar.gz

[ywk253100]

I can see the features: EnableCSI in the arguments for my deployment. Only in this way it is working.

This is the correct format: https://github.com/vmware-tanzu/helm-charts/blob/main/charts/velero/values.yaml#L440

For the timeout error, refer to #7427 (comment) for further investigation.

[davidmkrtchian]

I can see the features: EnableCSI in the arguments for my deployment. Only in this way it is working.

This is the correct format: https://github.com/vmware-tanzu/helm-charts/blob/main/charts/velero/values.yaml#L440

For the timeout error, refer to #7427 (comment) for further investigation.

I've reviewed your comment. In my scenario, the volume snapshot is being created successfully after running the backup, but it is not creating the volumesnapshotcontent. Despite attempting the solution of adding the CSI snapshot timeout, it hasn't resolved the problem for me.
Could you confirm whether my configuration for creating backups and the snapshot class appears correct? The logs only indicate a timeout error. Your assistance in resolving this matter is greatly appreciated. Thank you.

[ywk253100]

If the VolumeSnapshot is created, then this seems isn't the issue of Velero.
You can check the status of the VolumeSnapshot and check whether it contains any useful information for why the VolumeSnapshotContent isn't created.

If the status of VolumeSnapshot contains nothing useful, you'd better contact the driver provider and get some support.

[Saddique-NY]

I have been facing the same issue. I want to take the volumsnapshot on vpshere storage. let me share what process I followed. I enabled "volumesnapshots" CSI by deploying the resources in kube-system namespace for kubernetes. Then I deployed velero v1.10.2 in velero namespace by following the velero-k8s matrix.
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
name: minio
namespace: velero
spec:
config:
region: minio
credential:
key: 123
name: minio-ds
provider: aws

and

• args:
  - server
  - '--uploader-type=restic'
  - '--default-backup-storage-location=minio'
  could someone tell me what am i missing?

[revoltez]

In my case the problem was that i wasn't properly indenting features: EnableCSI & defaultSnapshotMoveData:
they should be indented under Configuration not top level

configuration:#8622 
  # backup & snaphot configs here as well
  defaultSnapshotMoveData:
  features: EnableCSI