Karpenter nodes get provisioned with default domain in a custom domain cluster

[cristian-bratu-l2]

Description

Observed Behavior:

Hi folks! I have a cluster with a custom domain (ue2.green.aws). Nodes that are provisioned with cluster-autoscaler have the correct search list set in the resolv.conf stanza:

cat /etc/resolv.conf
search <namespace>.svc.ue2.green.aws svc.ue2.green.aws ue2.green.aws us-east-2.compute.internal
nameserver 172.21.0.10
options ndots:5

If I attempt to provision new nodes with Karpenter the search domains reverts back to cluster.local. Breaking internal DNS references.

cat /etc/resolv.conf
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.21.0.10
options ndots:5

This happens when I set settings.kubernetes.cluster-domain to my domain or not. I have attempted to set settings.dns.search-list however all it does is appends my configured search list to resolv.conf resulting in something like:

cat /etc/resolv.conf
search <namespace>.svc.cluster.local svc.cluster.local cluster.local svc.ue2.green.aws ue2.green.aws
nameserver 172.21.0.10
options ndots:5

Taking a look at cluster-autoscaler provisioned nodes userData I don't see anything obvious:

settings.kubernetes.cluster-name = 'l2f-sandbox-green'
settings.kubernetes.api-server = 'https://[REDACTED].sk1.us-east-2.eks.amazonaws.com'
settings.kubernetes.cluster-certificate = [ REDACTED ]
settings.kubernetes.cluster-dns-ip = '172.21.0.10'
settings.kubernetes.max-pods = 110
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup-image' = 'ami-065cf04d9a8a3b598'
settings.kubernetes.node-labels.'eks.amazonaws.com/capacityType' = 'SPOT'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateVersion' = '4'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'platform_arm-20240117172858363100000001'
settings.kubernetes.node-labels.'bottlerocket.aws/updater-interface-version' = '2.0.0'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateId' = 'lt-070a70a171473a9cc'
settings.kubernetes.cluster-domain = 'ue2.green.aws'%

Expected Behavior:

The custom domain be respect for the search subdomains.

cat /etc/resolv.conf
search <namespace>.svc.ue2.green.aws svc.ue2.green.aws ue2.green.aws us-east-2.compute.internal
nameserver 172.21.0.10
options ndots:5

Probably related: aws/karpenter-provider-aws#3166

Reproduction Steps (Please include YAML):

Provision an EKS cluster with a custom domain. Install karpenter and use the following ec2nodeclass and nodepool:

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default-spot
spec:
  amiFamily: Bottlerocket
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      deleteOnTermination: true
      encrypted: true
      iops: 3000
      throughput: 150
      volumeSize: 5Gi
      volumeType: gp3
  - deviceName: /dev/xvdb
    ebs:
      deleteOnTermination: true
      encrypted: true
      iops: 3000
      throughput: 150
      volumeSize: 50Gi
      volumeType: gp3
  role: < NODE_ROLE >
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: enabled
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: enabled
  tags:
    cloud.layer2financial.com/autoscaler: karpenter
    karpenter.sh/discovery: l2f-sandbox-green
    Name: karpenter.sh/provisioner-name/platform
  userData: |
    [settings.kubernetes]
    cluster-domain = "ue2.green.aws"
    [settings.dns]
    search-list = ["svc.ue2.green.aws", "ue2.green.aws"]

The nodepool:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: platform
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 168h0m0s
  limits:
    cpu: "128"
  template:
    metadata: {}
    spec:
      nodeClassRef:
        name: default-spot
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
          - spot
        - key: kubernetes.io/arch
          operator: In
          values:
          - amd64
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values:
          - "8"
          - "16"
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values:
          - "5"
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
          - a
          - c
          - m
          - r
        - key: kubernetes.io/os
          operator: In
          values:
          - linux
  weight: 100

Versions:

• Chart Version: 0.33.1
• Kubernetes Version (kubectl version): v1.27.8-eks-8cb36c9

[jonathan-innis]

This issue looks specifically-related to AWS. Do you mind opening this issue over in aws/karpenter-provider-aws? That'll make it easier to triage it for the provider maintainers.

[jonathan-innis]

Closing this since this is AWS-specific. Ask is to open this in the provider repo.