[DOC] - Missing documentation on how to external repo via registries.yaml

[alex-ioma]

Environmental Info:
RKE2 Version: 1.28.10+rke2r1

Node(s) CPU architecture, OS, and Version: N/A

Cluster Configuration: 3 master + 3 workers

Describe the bug:

Apologies for opening this ticket but I could not find a way to add GCP Artifact Registry to the RKE2 containerd runtime via the registries.yaml file.

Steps To Reproduce:

• Create a Service Account Key on GCP/IAM with access to GCP Artifact Registry
• Test registry access with said key (it correctly pull from local)
• Try and add key but cannot figure out which how to complete the registries.yaml config file.

Expected behavior:

Be able to pull images via crictl or inside k8s.

Actual behavior:

Pull doesn't work - failing with:
Failed to pull image "europe-docker.pkg.dev/cm-controls/lgm/lgm-hmi:v0.1.0-alpha": failed to pull and unpack image "europe-docker.pkg.dev/cm-controls/lgm/lgm-hmi:v0.1.0-alpha": failed to resolve reference "europe-docker.pkg.dev/cm-controls/lgm/lgm-hmi:v0.1.0-alpha": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://europe-docker.pkg.dev/v2/token?scope=repository%3Acm-controls%2Flgm%2Flgm-hmi%3Apull&service=europe-docker.pkg.dev: 403 Forbidden
Additional context / logs:

It's likely an error on my end on how to properly configure registries.yaml but I feel the documentation can be improved. Also it is not clear if this configuration should be applied to all nodes or only to one master and/or worker and is automatically replicated.

Thank you in advance for the support.

[brandond]

It looks like you're using europe-docker.pkg.dev as the registry, so you'd want to add an entry to the config section with credentials for europe-docker.pkg.dev. This is covered at https://docs.rke2.io/install/containerd_registry_configuration#configs

Registries.yaml is not a cluster resource. The configuration is local to each node, so you must place this configuration on every node that you want to be able to authenticate to your registry. This is also covered at the top of the above-linked page.

[alex-ioma]

Thank you very much - definitely flew through that. Finally getting closer but still not working - error is:

E0605 00:02:27.456638  357011 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"europe-docker.pkg.dev/repo/image:v0.1.0-alpha\": failed to resolve reference \"europe-docker.pkg.dev[...]": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://europe-docker.pkg.dev/v2/token?scope=repository%3A....: 401 Unauthorized

It seems that ctr is able to use the key provided through registries.yaml file, however, it is not able to use all of it's properties such as the fact that token should be obtain at the token_uri.

For example:

{
  ...
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/registry-ro%40cm-controls.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Are you aware of any issue on containerd side about this. Any suggestion on how to troubleshoot.

Best regards.

[brandond]

I would probably go take a look at the upstream docs, and open an issue if necessary.

[alex-ioma]

I've tested different configurations also by interacting directly with config.toml.tmpl within containerd. Issue remains.
I've open a ticket on the source. Will keep you posted.

[alex-ioma]

@brandond apologies for keeping this thread open - just realized that I did not circled back once the solution was found.

Indeed it was as easy as adding these lines to the registries.yaml config:

configs:
  "europe-docker.pkg.dev":
    auth:
      username: _json_key_base64
      password: 'your base-64 formatted key file'

of course it also works with:

configs:
  "europe-docker.pkg.dev":
    auth:
      username: _json_key
      password: 'your key file here'

but in this latter case, you must pay attention to single apexes and special characters of the base-64 formatted certs that may break the password field.

Also, there are sources/forums online that make use of *.pkg.dev as source. In our case, this has proven to be an issue as containerd or RKE2 expected strict URI and was not accepting * instead of europe-docker. If you still have issue, you may try to use the full URL.

Perhaps, incorporating these notes into the official docs might help others.

Thanks for the great work.