secrets must be read-only

[pohly]

Is this a request for help?: no

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:

$ helm version
Client: &version.Version{SemVer:"v2.8+unreleased", GitCommit:"c5f2174f264554c62278c0695d58f250d3e207c8", GitTreeState:"clean"}
Server: &version.Version{SemVer:"canary+unreleased", GitCommit:"fe9d36533901b71923c49142f5cf007f93fa926f", GitTreeState:"clean"}

Kubernetes master > 1.9

Which chart: ceph

What happened:

I compiled k8s master from source (commit 04634cb19843195) and brought up a local cluster with:

RUNTIME_CONFIG=storage.k8s.io/v1alpha1=true ALLOW_PRIVILEGED=1 FEATURE_GATES="BlockVolume=true,MountPropagation=true,CSIPersistentVolume=true," hack/local-up-cluster.sh -O

Then I followed http://docs.ceph.com/docs/master/start/kube-helm/#configure-your-ceph-cluster to install the ceph chart.

In that installation, the start_mon.sh script in the ceph-mon pod fails with:

+ ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /etc/ceph/ceph.mon.keyring
bufferlist::write_file(/etc/ceph/ceph.mon.keyring): failed to open file: (30) Read-only file system
could not write /etc/ceph/ceph.mon.keyring

What you expected to happen:

The script shouldn't write into a secret. The modification is not stored permanently in older Kubernetes releases and starting with 1.10, the default will be to mount secrets as read-only, even if "readonly: false" is used - see kubernetes/kubernetes#58720.

Anything else we need to know:

@intlabs said on Slack that he's going to fix this for openstack-helm/ceph. In the meantime one can use ReadOnlyAPIDataVolumes=false in FEATURE_GATES to restore the old behavior.

Here's a fix that worked for me. It's intentionally very minimal, perhaps the right solution also has to clean up the usage of secret in other pods:

diff --git a/ceph/ceph/templates/bin/_start_mon.sh.tpl b/ceph/ceph/templates/bin/_start_mon.sh.tpl
index 50e4bfd..5b3330c 100644
--- a/ceph/ceph/templates/bin/_start_mon.sh.tpl
+++ b/ceph/ceph/templates/bin/_start_mon.sh.tpl
@@ -62,8 +62,7 @@ chown ceph. /var/log/ceph
 # If we don't have a monitor keyring, this is a new monitor
 if [ ! -e "$MON_DATA_DIR/keyring" ]; then
   if [ ! -e $MON_KEYRING ]; then
-    log "ERROR- $MON_KEYRING must exist.  You can extract it from your current monitor by running 'ceph auth get mon. -o $MON_KEYRING' or use a KV Store"
-    exit 1
+    touch $MON_KEYRING
   fi
 
   if [ ! -e $MONMAP ]; then
diff --git a/ceph/ceph/templates/daemonset-mon.yaml b/ceph/ceph/templates/daemonset-mon.yaml
index 4b9c90d..3c26211 100644
--- a/ceph/ceph/templates/daemonset-mon.yaml
+++ b/ceph/ceph/templates/daemonset-mon.yaml
@@ -141,10 +141,6 @@ spec:
               mountPath: /etc/ceph/ceph.client.admin.keyring
               subPath: ceph.client.admin.keyring
               readOnly: true
-            - name: ceph-mon-keyring
-              mountPath: /etc/ceph/ceph.mon.keyring
-              subPath: ceph.mon.keyring
-              readOnly: false
             - name: ceph-bin
               mountPath: /variables_entrypoint.sh
               subPath: variables_entrypoint.sh
@@ -195,9 +191,6 @@ spec:
         - name: ceph-client-admin-keyring
           secret:
             secretName: {{ .Values.secrets.keyrings.admin }}
-        - name: ceph-mon-keyring
-          secret:
-            secretName: {{ .Values.secrets.keyrings.mon }}
         - name: ceph-bootstrap-osd-keyring
           secret:
             secretName: {{ .Values.secrets.keyrings.osd }}

[whmzsu]

facing the same problem. I had a mini cluster with kubernetes 1.10.1 created by kubeadm. I will test the patch!!

[foxundermoon]

how to resolve?
@whmzsu i meet the same problem. same env
can you find solutation?

[whmzsu]

see the PR #54
My env is working !!