Bottlerocket cluster-domain support in kubernetes.settings

[cristian-bratu-l2]

Description

Description

Observed Behavior:

Hi folks! I have a cluster with a custom domain (ue2.green.aws). Nodes that are provisioned with cluster-autoscaler have the correct search list set in the resolv.conf stanza:

cat /etc/resolv.conf
search <namespace>.svc.ue2.green.aws svc.ue2.green.aws ue2.green.aws us-east-2.compute.internal
nameserver 172.21.0.10
options ndots:5

If I attempt to provision new nodes with Karpenter the search domains reverts back to cluster.local. Breaking internal DNS references.

cat /etc/resolv.conf
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.21.0.10
options ndots:5

This happens when I set settings.kubernetes.cluster-domain to my domain or not. I have attempted to set settings.dns.search-list however all it does is appends my configured search list to resolv.conf resulting in something like:

cat /etc/resolv.conf
search <namespace>.svc.cluster.local svc.cluster.local cluster.local svc.ue2.green.aws ue2.green.aws
nameserver 172.21.0.10
options ndots:5

Taking a look at cluster-autoscaler provisioned nodes userData I don't see anything obvious:

settings.kubernetes.cluster-name = 'l2f-sandbox-green'
settings.kubernetes.api-server = 'https://[REDACTED].sk1.us-east-2.eks.amazonaws.com'
settings.kubernetes.cluster-certificate = [ REDACTED ]
settings.kubernetes.cluster-dns-ip = '172.21.0.10'
settings.kubernetes.max-pods = 110
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup-image' = 'ami-065cf04d9a8a3b598'
settings.kubernetes.node-labels.'eks.amazonaws.com/capacityType' = 'SPOT'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateVersion' = '4'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'platform_arm-20240117172858363100000001'
settings.kubernetes.node-labels.'bottlerocket.aws/updater-interface-version' = '2.0.0'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateId' = 'lt-070a70a171473a9cc'
settings.kubernetes.cluster-domain = 'ue2.green.aws'%

Expected Behavior:

The custom domain be respect for the search subdomains.

cat /etc/resolv.conf
search <namespace>.svc.ue2.green.aws svc.ue2.green.aws ue2.green.aws us-east-2.compute.internal
nameserver 172.21.0.10
options ndots:5

Probably related: #3166

Reproduction Steps (Please include YAML):

Provision an EKS cluster with a custom domain. Install karpenter and use the following ec2nodeclass and nodepool:

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default-spot
spec:
  amiFamily: Bottlerocket
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      deleteOnTermination: true
      encrypted: true
      iops: 3000
      throughput: 150
      volumeSize: 5Gi
      volumeType: gp3
  - deviceName: /dev/xvdb
    ebs:
      deleteOnTermination: true
      encrypted: true
      iops: 3000
      throughput: 150
      volumeSize: 50Gi
      volumeType: gp3
  role: < NODE_ROLE >
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: enabled
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: enabled
  tags:
    cloud.layer2financial.com/autoscaler: karpenter
    karpenter.sh/discovery: l2f-sandbox-green
    Name: karpenter.sh/provisioner-name/platform
  userData: |
    [settings.kubernetes]
    cluster-domain = "ue2.green.aws"
    [settings.dns]
    search-list = ["svc.ue2.green.aws", "ue2.green.aws"]

The nodepool:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: platform
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 168h0m0s
  limits:
    cpu: "128"
  template:
    metadata: {}
    spec:
      nodeClassRef:
        name: default-spot
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
          - spot
        - key: kubernetes.io/arch
          operator: In
          values:
          - amd64
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values:
          - "8"
          - "16"
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values:
          - "5"
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
          - a
          - c
          - m
          - r
        - key: kubernetes.io/os
          operator: In
          values:
          - linux
  weight: 100

Versions:

• Chart Version: 0.33.1
• Kubernetes Version (kubectl version): v1.27.8-eks-8cb36c9

[jonathan-innis]

Can you show what the result is if you look at the userData from a Karpenter-provisioned node? Does it show the custom domain that you are trying to target?

[cristian-bratu-l2]

Can you show what the result is if you look at the userData from a Karpenter-provisioned node? Does it show the custom domain that you are trying to target?

Of course, here it is:

aws ec2 describe-instance-attribute --instance-id <ID_HERE> --attribute userData --output text --query "UserData.Value" --region=us-east-2| base64 --decode
[settings]
[settings.dns]
search-list = ['svc.ue2.green.aws', 'ue2.green.aws']

[settings.kubernetes]
api-server = 'https://[ REDACTED ].sk1.us-east-2.eks.amazonaws.com'
cluster-certificate = '[ REDACTED ]'
cluster-name = 'l2f-sandbox-green'
cluster-dns-ip = '172.21.0.10'
max-pods = 58

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'spot'
'karpenter.sh/nodepool' = 'platform'

I brought up a node this morning, and I can't see the cluster-domain value added to [settings.kubernetes] even though the ec2nodeclass has it setup.

[jonathan-innis]

Looks like it isn't part of the https://github.com/aws/karpenter-provider-aws/blob/main/pkg/providers/amifamily/bootstrap/bottlerocketsettings.go#L45 so we wouldn't be serializing it if you were specifying it on merge.

You'd need to add it into our version of the struct so that we serialize and overlay it into TOML correctly

[dlmather]

@jonathan-innis are there any plans to provide an escape hatch or something for adding things like this as raw settings? In general karpenter seems to be missing a few fields from the latest Bottlerocket API spec for settings.kubernetes (I'm looking to add seccomp-default for example). For settings not scoped under settings.kubernetes the SettingsRaw setup lets them be added, anything under settings.kubernetes gets overwritten https://github.com/aws/karpenter-provider-aws/blob/main/pkg/providers/amifamily/bootstrap/bottlerocketsettings.go#L113. It feels like it's cumbersome for contributors to have to raise a PR to update this struct every time they want to access a field.

[jonathan-innis]

It feels like it's cumbersome for contributors to have to raise a PR to update this struct every time they want to access a field

Completely agree. We'd be more for auto-gen-ing it or not relying on the structured data directly and just allowing arbitrary formatting without validation. See #6259 which I just opened.