Velero doesnt do well with Minio bucket versioning #8466
Comments
Lyndon-Li
added
the
Area/Storage/Minio
|
Thanks for the sharing, this is a helpful troubleshooting. Velero is calling standard S3 API, any objects returned by it are regarded valid by Velero, so Velero has no way to filter out some objects in this scenarios. |
|
on the contrary. I made a pr in pkg/persistence/object_store.go func (s *objectBackupStore) ListBackups() ([]string, error) {} some time ago - but it never made it into your code. func (s *objectBackupStore) ListBackups() ([]string, error) {
prefixes, err := s.objectStore.ListCommonPrefixes(s.bucket, s.layout.subdirs["backups"], "/")
if err != nil {
return nil, err
}
if len(prefixes) == 0 {
return []string{}, nil
}
output := make([]string, 0, len(prefixes))
for _, prefix := range prefixes {
// values returned from a call to ObjectStore's
// ListCommonPrefixes method return the *full* prefix, inclusive
// of s.backupsPrefix, and include the delimiter ("/") as a suffix. Trim
// each of those off to get the backup name.
backupName := strings.TrimSuffix(strings.TrimPrefix(prefix, s.layout.subdirs["backups"]), "/")
// if a bucket is minio and versioned, the s3 folder still exists, even if all s3 objects have been deleted.
// We should only take s3 folders having a velero-backup.json object.
if ok, errObjectExists := s.objectStore.ObjectExists(s.bucket, s.layout.getBackupMetadataKey(backupName)); !ok || errObjectExists != nil {
// velero-backup.json or error do not add to list
if errObjectExists != nil {
s.logger.WithError(errObjectExists).Debugf("could not check if %s/%s exists", s.bucket, s.layout.getBackupMetadataKey(backupName))
}
continue
}
output = append(output, backupName)
}
return output, nil
}It simply checks, if a folder contains the anyway needed velero-backup.json file inside a backup folder. We are running this patch since version v1.12.1 without any problems. I could create another pull request if someone is interested. |
|
Sure please do. We will review. |
Eventually, the problem comes down to minio side. |
…inio, if bucket is versioned.
…inio, if bucket is versioned. Signed-off-by: Christian Jürges <[email protected]>
What steps did you take and what happened:
This will probably not make it past triage, but I hit this issue and solved it, someone else might benefit from this.
I am using helm to install minio, have been reusing the same values.yaml for a couple of years now.
With this deployment and velero 15.1.0 I created a backup with tth 2hours.
Once the backup expired velero kept trying to sync the expired backup:
I started digging around.
mc(minio client) shows the folder in the bucket as empty.If I exec into the minio container I see something very different:
Turns out versioning on the bucket is
suspendednotun-versioned, even though the helm values sayversioning: falseI listed the default values of the minio helm chart and behold the magic of satan:
# versioning: false # remove this key if you do not want versioning feature<--- 👎 👎 👎 Remove the line, setting it to false will suspend versioning, not disable it completely.What did you expect to happen:
Velero should either work with versioned s3 buckets or not pickup the metadata from the versioned blobs if a file was deleted (ex:
mcworks correctly with versions and does not list any deleted file that has "active" versions)The following information will help us better understand what's going on:
https://transfer.kronsoft.cloud/yp7er0/bundle-2024-11-27-11-42-30.tar.gz -> expires in 7d
Anything else you would like to add:
Environment:
velero version):velero client config get features):features: <NOT SET>kubectl version):/etc/os-release):Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
The text was updated successfully, but these errors were encountered: