[feat] Add EKS Pod Identity Skip Tag

[DerekTBrown]

/kind feature

• When using the kops-installed EKS Pod Identity Webhook in combination with other features, we observed circular-dependency issues, preventing clusters from being provisioned without manual intervention.
  • For example, when installing a CNI in the cluster, the Pod Identity Webhook may not function until the CNI is installed, but the CNI control plane is blocked on the Pod Identity Webhook because it is a Fail mutating webhook.
• To mitigate this issue, I merged an upstream change in the amazon-eks-pod-identity-webhook repository which adds a canonical tag for excluding certain pods from the webhook: feat: cannonical label for skipping pod identity webhook aws/amazon-eks-pod-identity-webhook#216
• We should pull these changes into the kops-managed amazon-eks-pod-identity-webhook, as this would allow users to extensibly define pods that should skip the blocking webhook.

[lukasmrtvy]

@DerekTBrown Hi, how does this work? Should I just add this annotation to an addon and how? I think there are more problems with admission hooks, for example I wanted to deploy almost "full stack" in AWS, but at least aws-load-balancer-webhook, cert-manager-webhook, pod-identity-webhook are blocking themselves, preventing proper cluster bootstrap. I guess this was not tested at all..

[DerekTBrown]

Hi, how does this work? Should I just add this annotation to an addon and how?

You would need to create a patched version of kops that includes my proposed changes:

https://github.com/kubernetes/kops/pull/16636/files

With these changes, the amazon-eks-pod-identity-webhook will skip over anything with the eks.amazonaws.com/skip-pod-identity-webhook annotation, so you just add the eks.amazonaws.com/skip-pod-identity-webhook annotation to your CNI pods.

[DerekTBrown]

For example I wanted to deploy almost "full stack" in AWS, but at least aws-load-balancer-webhook, cert-manager-webhook, pod-identity-webhook are blocking themselves, preventing proper cluster bootstrap.

IMO, every webhook should follow the pattern of allowing certain pods to be excluded. However, it ultimately falls on the chart itself to implement this behavior.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.