Aws plugin 1.11.1 constantly exiting with error for unclear reason #8653
Comments
|
It doesn't look like an error because the log level is If you see any failures, please share velero log bundle. Otherwise, this log could be ignored. |
|
Thank you for your reply. It's mostly about the errors about this plugin. I didn't find any other errors that relate to this one. It's bugging me as the plugin is constantly throwing info level "plugin process exited" in the logs and I'm unable to identify why. Log bundle of a freshly started container: |
|
Debug logs are meant for developers. These logs should not worry you. In case you are wondering where it came from. It's not meant to have additional info other than, a plugin exited. |
|
The plugin will exit any time the Reconcile() loop finishes. It looks like this is from the BSL controller. By default this gets validated every minute, so I would expect a plugin to exit once a minute. If you're seeing this more often, then either you've changed the BSL validation frequency, or something is going wrong in your env or your velero instance that's causing constant validation. |
|
As described... every minute. I would suggest removing this from the info logs if this is the expected (correct) behaviour. For me it was misleading and I was searching for the cause. Thank you for the help. |
|
@chieftec9 these logs were added in hashicorp/go-plugin#195 I believe it is to help assist developers diagnose issues when user report them. If you are ok with redoing a backup/restore with these logs then perhaps we can have user opt into have these logs silenced in the main case? |
What steps did you take and what happened:
I'm getting constant error messages that aws plugin is exiting with no clear reason or possibility to identify why. Backups seem to be successfully created as files are being put to S3 and snapshots created over CSI.
Error:
{"backup-storage-location":"velero/default","controller":"backup-storage-location","level":"info","logSource":"pkg/controller/backup_storage_location_controller.go:142","msg":"Validating BackupStorageLocation","time":"2025-01-25T13:36:25Z"} {"backup-storage-location":"velero/default","controller":"backup-storage-location","level":"info","logSource":"pkg/controller/backup_storage_location_controller.go:127","msg":"BackupStorageLocations is valid, marking as available","time":"2025-01-25T13:36:25Z"} {"backup-storage-location":"velero/default","cmd":"/plugins/velero-plugin-for-aws","controller":"backup-storage-location","err":"rpc error: code = Unavailable desc = error reading from server: EOF","level":"debug","logSource":"pkg/plugin/clientmgmt/process/logrus_adapter.go:75","msg":"received EOF, stopping recv loop","pluginName":"stdio","time":"2025-01-25T13:36:25Z"} {"backup-storage-location":"velero/default","cmd":"/plugins/velero-plugin-for-aws","controller":"backup-storage-location","id":"56594","level":"info","logSource":"pkg/plugin/clientmgmt/process/logrus_adapter.go:80","msg":"plugin process exited","plugin":"/plugins/velero-plugin-for-aws","time":"2025-01-25T13:36:25Z"} {"backup-storage-location":"velero/default","cmd":"/plugins/velero-plugin-for-aws","controller":"backup-storage-location","level":"debug","logSource":"pkg/plugin/clientmgmt/process/logrus_adapter.go:75","msg":"plugin exited","time":"2025-01-25T13:36:25Z"}I tested also with a blank S3 (no pre-existing backups), editing config for backupstoragelocation like forcestylepath and trying to hardcode url, but nothing worked.
Would be really grateful on where to look further to resolve this.
What did you expect to happen:
I assume this error should not persists and there is some reason for it being triggered.
The following information will help us better understand what's going on:
If you are using velero v1.7.0+:
Please use
velero debug --backup <backupname> --restore <restorename>to generate the support bundle, and attach to this issue, more options please refer tovelero debug --helpIf you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs deployment/velero -n velerovelero backup describe <backupname>orkubectl get backup/<backupname> -n velero -o yamlvelero backup logs <backupname>velero restore describe <restorename>orkubectl get restore/<restorename> -n velero -o yamlvelero restore logs <restorename>Anything else you would like to add:
Environment:
velero version): v1.15.2 (both client/server)velero client config get features):kubectl version): v1.30.8-eks-2d5f260/etc/os-release):Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
The text was updated successfully, but these errors were encountered: