backups fail constantly

[cooperspencer]

What steps did you take and what happened:
We run backups via a schedule. The backups are in a failed state and we can't view any logs.
The backup shows that all the items got backed up but the phase is failed nonetheless:

Name:         regular-20250121230037
Namespace:    velero
Labels:       velero.io/schedule-name=regular
              velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.30.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=30

Phase:  Failed (run `velero backup logs regular-20250121230037` for more information)

Errors:    0
Warnings:  66

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  false
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2025-01-22 00:00:37 +0100 CET
Completed:  <n/a>

Expiration:  2025-02-21 00:00:37 +0100 CET

Total items to be backed up:  3600
Items backed up:              3600

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included or not detectable>

  Pod Volume Backups - kopia (specify --details for more information):
    Completed:  70

HooksAttempted:  0
HooksFailed:     0

When I try to get the logs of the backup, I get An error occurred: file not found.

We are in the oracle cloud and the destination is an s3 bucket in oracle.

What did you expect to happen:
I expected it to not fail.

The following information will help us better understand what's going on:

bundle-2025-01-22-09-47-28.tar.gz

Anything else you would like to add:

Environment:

• Velero version (use velero version): v1.15.2
• Velero features (use velero client config get features): v1.15.2
• Kubernetes version (use kubectl version): v1.31.3
• Kubernetes installer & version: v1.30.1
• Cloud provider or hardware configuration: Oracle
• OS (e.g. from /etc/os-release): Oracle Linux 7

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

The error seen from the log bundle:
rpc error: code = Unknown desc = error putting object frzy7lu2gi8y/backups/test-backup/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 400, RequestID: fra-1:pU6AObrk2D9VFepwRu_R5p6OOtBm4P-PYY_Ml-7qhWbwz_iY4BY0pxK-xs2JbxId, HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD or a valid sha256 value.

Velero failed to write the backup data to object store because the S3 API PutObject returned an error.

Do you see the failure happened for every backup?

[cooperspencer]

See this issue #8013 discussion for details. You need to use a lower version of AWS plugin for Velero (1.18), as the new versions has upgraded to AWS SDK V2

[cooperspencer]

THANK YOU!! I downgraded the aws-plugin to 1.8 and now it works.