The cluster registry is a Kubernetes-style API and API server that provides an
endpoint for interacting with a list of clusters and associated metadata. If it
helps, you can think of the cluster registry as a hosted kubeconfig file.
However, since it's a Kubernetes-style API, the cluster registry allows custom
annotations and filtering across labels, can be used with kubectl
and
Kubernetes-style generated client libraries, and supports having controllers
watch for updates.
The crinit
tool allows you to deploy a cluster registry into an existing
Kubernetes cluster. Run the following commands to download the latest nightly
build of the tool:
Currently the tool is only released for Linux 64-bit. You'll need to build it yourself if you want to use it on a different platform. See the development docs.
PACKAGE=client
LATEST=$(curl https://storage.googleapis.com/crreleases/nightly/latest)
curl -O http://storage.googleapis.com/crreleases/nightly/$LATEST/clusterregistry-$PACKAGE.tar.gz
tar xzf clusterregistry-$PACKAGE.tar.gz
You can deploy the cluster registry as a standalone API server like so:
./crinit standalone init <cluster_registry_instance_name> --host-cluster-context=<your_cluster_context>
where cluster_registry_instance_name
is the name you want to give this cluster
registry instance and your_cluster_context
is a context entry in your
kubeconfig file referencing a cluster into which the cluster registry will be
deployed. The cluster_registry_instance_name
will be used to name resources
that will be created in your cluster for the cluster registry, such as a Service
and a Deployment, and should be named appropriately. When the command completes,
it will create an entry in your kubeconfig file for the cluster registry API
server named with the provided cluster_registry_instance_name
.
You can interact with it using kubectl:
$ kubectl get clusters --context <cluster_registry_instance_name>
No resources found
$
Before deploying the cluster registry as an aggregated API server, take a look at
https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/#enable-apiserver-flags
which talks about what kube-apiserver
flags need to be enabled in order to
enable the aggregation layer.
Once the kube-apiserver
aggregation layer is enabled, you can deploy the
cluster registry as an aggregated API server like so:
./crinit aggregated init <cluster_registry_context> --host-cluster-context=<your_cluster_context>
where your_cluster_context
is a context entry in your kubeconfig file
referencing a cluster into which the cluster registry will be deployed and to
which the cluster registry API server will be added as an aggregated API server.
When the command completes, you will have a cluster registry running as an
aggregated API server with the cluster API server identified by
your_cluster_context
. You can interact with it using kubectl:
$ kubectl get clusters --context <your_cluster_context>
No resources found
$
In these examples, context
is either the cluster_registry_instance_name
if
you did a standalone deployment, or your_cluster_context
if you did an
aggregated deployment.
Try creating a cluster:
kubectl apply -f - --context <context> <<EOF
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
name: test-cluster
spec:
kubernetesApiEndpoints:
serverEndpoints:
- clientCIDR: "0.0.0.0/0"
serverAddress: "100.0.0.0"
EOF
And then reading it back:
kubectl get clusters --context <context>
crinit
takes care of a lot of the plumbing work to get a cluster registry
running, but the cluster registry can be run anywhere you like, as long as you
configure it correctly.
Since the cluster registry is an API server, all of the guidance around
deploying and managing aggregated API servers applies.
The Kubernetes docs
have some pointers about aggregation. You can also look at the
API Server Concepts docs
for some more technical guidance about aggregating. If you wish to run the
cluster registry as a standalone server, you will need to manage the
certificates and client authentication/authorization yourself. You can use any
authentication mode supported by the clusterregistry
command, but you will
probably need to use an
authorizing webhook,
since the cluster registry does not provide RBAC support inherently.
crinit
by default deploys etcd
in a container in the same pod as the
clusterregistry
container. This makes it easy to get started, but is not
necessarily a good strategy for deploying a cluster registry for production use.
You may want to look into the
etcd operator for production
deployments.
The cluster registry is a Kubernetes-style API server, and you can interact with
it using standard kubectl
commands. It provides one API type, clusters
,
which you can create, get, list and delete like any other Kubernetes object. See
Try it out! above for some sample commands.
There is a generated Go client library for the cluster registry in /pkg/client. You can vendor in the cluster registry repository and use the client library directly from your Go code.
There is an OpenAPI spec file provided here. You can use it to generate client libraries in a language of your choice.
If you are updating the cluster registry deployment by hand, you may run into errors like the following:
Multi-Attach error for volume "vol" Volume is already exclusively attached to one node and can't be attached to another
This is because the new clusterregistry
Pod
will not be able to claim the
persistent volume if it is started on a different node than the existing Pod
.
A simple fix is to change the deployement.spec.strategy.type
to Recreate
.
There are more details in kubernetes/kubernetes#48968.