Minio pod crashes after updating from 24.1.0 to 24.1.1

[tomikonio]

After updating with existing data from 24.1.0 to 24.1.1 the minio container crashes.
We tested the minio without providing data and it works.

In 24.1.1 minio has been updated - from 2021.4.22-debian-10-r6 to 2024.5.10-debian-12-r2 (link to diff)

Error overview:

• The minio container was first crashing because it did not meet its readinessProbe constraints
• After disabling both readiness and liveness probes the container kept crashing - with the following error:

Defaulted container "minio" out of: minio, volume-permissions (init)
 10:58:49.36 INFO  ==> 
 10:58:49.36 INFO  ==> Welcome to the Bitnami minio container
 10:58:49.36 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
 10:58:49.36 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
 10:58:49.36 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
 10:58:49.36 INFO  ==> 
 10:58:49.36 INFO  ==> ** Starting MinIO setup **
minio 10:58:49.38 DEBUG ==> Validating settings in MINIO_* env vars..
minio 10:58:49.40 INFO  ==> Starting MinIO in background...
ERROR Unable to use the drive /data: drive not found: Invalid arguments specified
minio 10:58:59.50 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:04.58 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:09.68 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:14.77 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:19.88 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:24.98 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:30.07 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:35.18 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:40.28 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:45.38 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:50.48 INFO  ==> Adding local Minio host to 'mc' configuration...
minio 10:59:55.58 INFO  ==> Adding local Minio host to 'mc' configuration...
Failed to add temporary MinIO server
minio 11:00:00.68 INFO  ==> MinIO is already stopped...

This is our minio configuration in values.yaml (updated from the old-minio accesskey/secretkey to rootUser/rootPassword):

  minio:
    image:
      debug: true
    auth:
      rootUser: "miniouser"
      rootPassword: "miniopassword"
    persistence:
      existingClaim: minio-data
      mountPath: /data
    volumePermissions:
      enabled: true
    livenessProbe:
      enabled: false
    readinessProbe:
      enabled: false

About ERROR Unable to use the drive /data: drive not found: Invalid arguments specified - the /data dir is set to the correct minio container user (1001) due to volumePermissions.enabled=true setting:

I have no name!@my-release-minio-6b9666d756-qw9hj:/$ ls -la / | grep data
drwxrwsr-x  81 1001 1001 4096 Jul  9 08:42 data

And all data is indeed there:

I have no name!@my-release-minio-7b4bdb744d-q79b7:/data$ ls -la
total 356
drwxrwsr-x  81 1001 1001  4096 Jul  9 08:42 .
drwxr-xr-x   3 root root    18 Jun 26 05:01 ..
-rw-------   1 1001 1001     6 Jul  9 11:49 .access_key
drwxrwsr-x.  7 1001 1001  4096 Jul  9 11:57 .minio.sys
-rw-------   1 1001 1001     9 Jul  9 11:57 .root_password
-rw-------   1 1001 1001     6 Jul  9 11:57 .root_user
-rw-------   1 1001 1001    12 Jul  9 11:49 .secret_key
drwxrws---   2 1001 1001 16384 Feb 29 15:39 lost+found
drwxrwsr-x.  2 1001 1001  4096 Feb 29 17:47 prj-1
drwxrwsr-x.  4 1001 1001  4096 Feb 29 17:47 prj-102
.................................

Reverting back to 24.1.0 -> all is working correctly with the data.

Any idea?

[Pink-Bumblebee]

@tomikonio , I think that you could use minio version from 24.1.0 in your installation of 24.1.1.

[hlebkanonik]

Hi! Have you updated Chart.yaml? We updated MinIO Chart from 14.5.0 to 14.6.13 in ReportPortal 24.1.2 release. It is a known issue from bitnami/minio.

[tomikonio]

@tomikonio , I think that you could use minio version from 24.1.0 in your installation of 24.1.1.

Changing the minio image tag to 2021.4.22-debian-10-r6 did not work with the newer minio 14.5.0 helm chart -> it results in different errors, causing an instant crash (I guess some env vars changed)

Hi! Have you updated Chart.yaml? We updated MinIO Chart from 14.5.0 to 14.6.13 in ReportPortal 24.1.2 release. It is a known issue from bitnami/minio.

Tried 24.1.2 now, sadly it results in the same issue with minio.

Thank you both :)

We are using the ReportPortal helm chart as a dependency so we take its release as a whole including all of its dependencies.
That means we cannot change the minio helm chart version at our will -> we can solve it by disabling the RP helm chart minio and (re)adding the minio helm chart as a direct dependency of our parent chart.

This should solve the issue, but we will have to stay on the old minio version, so this is an undesirable solution.

[tomikonio]

Update:

we can solve it by disabling the RP helm chart minio and (re)adding the minio helm chart as a direct dependency of our parent chart.

We have implemented the above suggestion - used minio.install: false and imported the old 6.7.7 (from 24.1.0) minio helm chart as a direct dependency.
This resulted in the following error:

charts/minio/charts/common/templates/_names.tpl:6:18: executing "common.names.name" at <.Chart.Name>: nil pointer evaluating interface {}.Name

This error is caused by having another helm chart which uses the bitnami common library chart - rabbitmq.
The rabbitmq chart uses a newer common chart but due to a helm limitation both minio and rabbitmq charts end up using the same common chart - resulting in the above error (related issues are here -> bitnami/charts#19003 and here -> helm/helm#11561).

To solve the above error we had to set rabbitmq.install: false and use the older rabbitmq helm chart from 24.1.0 as a direct dependency as-well.

After implementing this, the report-portal works.

Conclusion:
We had to use an older minio version and as a result we were forced to use an older rabbitmq version as well.
This is not a good long-term solution, as we will be forced to use old versions of both services.

Dependencies:

dependencies:
  - name: reportportal
    version: 24.1.2
    repository: https://reportportal.io/kubernetes/

  - name: minio
    version: 6.7.7
    repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
    condition: minioexternal.install

  - name: rabbitmq
    version: 10.3.9
    repository: https://charts.bitnami.com/bitnami  
    condition: rabbitmqexternal.install

We will be glad to hear of other possible solutions for the issue.

[hlebkanonik]

Hi,

One possible solution to this issue is to back up the MinIO data currently stored in /data and restore it to a newly deployed version using the default path /bitnami/minio/data.

You can also experiment with backing up a PVC and restoring it using extraVolumes, extraVolumeMounts , and persistence.mountPath . Ref: https://github.com/bitnami/charts/tree/main/bitnami/minio/#persistence

The issue is probably due to data inconsistency in the old version.

[tomikonio]

Hi,
Thank you for your answer.

Unless I have misunderstood you, that is exactly what we are doing -> providing the old data to the new version of minio and that is precisely what causes the error.

I do not see what difference a restore from a backup of the volume will make, as it has the same data as the original volume.

I think the issue lies in some metadata that the old-minio stores on the volume vs what the new-minio expects, but what is the exact metadata, I do not know.