rke2-nginx-ingress tcp-services-configmap not working

[gauravkarki]

I am having the same issue as @Fabyao (#2481). I am running bitbucket in a container and trying to expose TCP port (7999). I followed the instruction provided but I can not connect to the TCP port.
Any help/advise is greatly appreciated.

Environment: rke2 v1.24.7 on RHEL8.6

Service

#kubectl -n source-management get svc bitbucket -o yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: bitbucket
  name: bitbucket
  namespace: source-management
spec:
  ports:
  - name: http
    port: 7990
    protocol: TCP
    targetPort: 7990
  - name: ssh
    port: 7999
    protocol: TCP
    targetPort: 7999
  selector:
    app: bitbucket
    pod: bitbucket
  type: ClusterIP

Ingress

#kubectl -n source-management get ingress bitbucket -o yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: bitbucket
    description: ingress-for-bitbucket
  name: bitbucket
  namespace: source-management
spec:
  rules:
  - host: bitbucket.<domain.com>
    http:
      paths:
      - backend:
          service:
            name: bitbucket
            port:
              number: 7990
        path: /
        pathType: Prefix
      - backend:
          service:
            name: bitbucket
            port:
              number: 7999
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - bitbucket.<domain.com>
    secretName: wildcard-cert
status:
  loadBalancer: ## I am not using any LoadBalancer. These Ingress IPs are the IPs of worker Nodes
    ingress:
    - ip: 10.100.x.x
    - ip: 10.100.x.x
    - ip: 10.100.x.x
# kubectl -n source-management get ingress
NAME             CLASS    HOSTS                                     ADDRESS          PORTS     AGE
bitbucket        <none>   bitbucket.<domain.com>   10.100.x.x,10.100.x.x,10.100.x.x   80, 443   21d   <-- TCP port 7999 is not listed in the ingress

HelmChartConfigs

#kubectl get helmchartconfigs.helm.cattle.io -n kube-system rke2-ingress-nginx -o yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      extraArgs:
        tcp-services-configmap: "kube-system/rke2-ingress-tcp-service"
    tcp:
       7999: "source-management/bitbucket:7999"

RKE2 nixing ingress controller Daemonsets

#kubectl -n kube-system describe daemonsets.apps rke2-ingress-nginx-controller | grep tcp-services-configmap
--tcp-services-configmap=kube-system/rke2-ingress-tcp-service

#kubectl -n kube-system describe pod rke2-ingress-nginx-controller-f6tz2 | grep tcp-services-configmap
--tcp-services-configmap=kube-system/rke2-ingress-tcp-service

ConfigMap

#kubectl -n kube-system get cm rke2-ingress-tcp-service -o yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: rke2-ingress-tcp-service
  namespace: kube-system
data:
  "7999": source-management/bitbucket:7999

Connection Test

#telnet 10.100.x.x 443 <-- Connection to 443 is successful.
Trying 10.100.x.x...
Connected to 10.100.x.x.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

#telnet 10.100.x.x 7999 <-- Connection to TCP port using Ingress IP is not successful.
Trying 10.100.x.x...
telnet: connect to address 10.100.x.x: Connection refused

#telnet 10.43.1x.x 7999 <-- However connection to TCP port using the ClusterIP of the service is successful.
Trying 10.43.x.x...
Connected to 10.43.x.x.
Escape character is '^]'.
SSH-2.0-APACHE-SSHD-2.8.0
^]
telnet> quit
Connection closed.

Originally posted by @gauravkarki in #2481 (comment)

[brandond]

Converted this to a discussion, since you're asking for help with configuration, not reporting a problem.

See the response at #2481 (comment)

[gauravkarki]

I found a Solution that is working for me rancher/rancher#14744
@brandond With the default rke2 installation which contains pre-configured rke2-ingress-inginx-controller as daemonset, I do not see any process that is listening on port 443 but I can still connect to it. Is there anything that I am missing?
#ss -tupln

Netid   State    Recv-Q    Send-Q       Local Address:Port        Peer Address:Port   Process                                      
udp     UNCONN   0         0                  0.0.0.0:8472             0.0.0.0:*                                                   
udp     UNCONN   0         0                127.0.0.1:323              0.0.0.0:*       users:(("chronyd",pid=756,fd=6))            
udp     UNCONN   0         0                    [::1]:323                 [::]:*       users:(("chronyd",pid=756,fd=7))            
tcp     LISTEN   0         128              127.0.0.1:10248            0.0.0.0:*       users:(("kubelet",pid=1007,fd=35))          
tcp     LISTEN   0         128              127.0.0.1:10249            0.0.0.0:*       users:(("kube-proxy",pid=1468,fd=16))       
tcp     LISTEN   0         128              127.0.0.1:9099             0.0.0.0:*       users:(("calico-node",pid=3759,fd=8))       
tcp     LISTEN   0         128              127.0.0.1:2379             0.0.0.0:*       users:(("etcd",pid=1466,fd=10))             
tcp     LISTEN   0         128          10.100.xx.xx:2379             0.0.0.0:*       users:(("etcd",pid=1466,fd=9))              
tcp     LISTEN   0         128              127.0.0.1:2380             0.0.0.0:*       users:(("etcd",pid=1466,fd=8))              
tcp     LISTEN   0         128          10.100.xx.xx:2380             0.0.0.0:*       users:(("etcd",pid=1466,fd=7))              
tcp     LISTEN   0         128              127.0.0.1:2381             0.0.0.0:*       users:(("etcd",pid=1466,fd=15))             
tcp     LISTEN   0         128              127.0.0.1:10256            0.0.0.0:*       users:(("kube-proxy",pid=1468,fd=15))       
tcp     LISTEN   0         128              127.0.0.1:10257            0.0.0.0:*       users:(("kube-controller",pid=3978,fd=7))   
tcp     LISTEN   0         128              127.0.0.1:10258            0.0.0.0:*       users:(("cloud-controlle",pid=4051,fd=7))   
tcp     LISTEN   0         128              127.0.0.1:10259            0.0.0.0:*       users:(("kube-scheduler",pid=1510,fd=7))    
tcp     LISTEN   0         128                0.0.0.0:22               0.0.0.0:*       users:(("sshd",pid=929,fd=5))               
tcp     LISTEN   0         128              127.0.0.1:10010            0.0.0.0:*       users:(("containerd",pid=994,fd=18))        
tcp     LISTEN   0         128                      *:9091                   *:*       users:(("calico-node",pid=3759,fd=13))      
tcp     LISTEN   0         128                      *:10250                  *:*       users:(("kubelet",pid=1007,fd=32))          
tcp     LISTEN   0         128                      *:6443                   *:*       users:(("kube-apiserver",pid=1508,fd=7))    
tcp     LISTEN   0         128                   [::]:22                  [::]:*       users:(("sshd",pid=929,fd=7))               
tcp     LISTEN   0         128                      *:9345                   *:*       users:(("rke2",pid=953,fd=8)) 

#telnet 10.100.xx.xx 443

Trying 10.100.100.23...
Connected to 10.100.100.23.
Escape character is '^]'.
^CConnection closed by foreign host.

Working Solution for TCP

I updated my HelmChartConfigs, ConfigMap, rke2-ingress-controller-admission service, and Network-Ingress-Policy

HelmChartConfigs

#kubectl get helmchartconfigs.helm.cattle.io -n kube-system rke2-ingress-nginx -o yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      extraArgs:
        tcp-services-configmap: "source-management/rke2-ingress-tcp-service"
    containerPort:
       ssh: 7999

ConfigMap

#kubectl -n kube-system get cm rke2-ingress-tcp-service -o yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: rke2-ingress-tcp-service
  namespace: source-management
data:
  "7999": source-management/bitbucket:7999

Edit rke2-ingress-nginx-controller-admission to accept port 7999

#kubectl -n kube-system edit service rke2-ingress-nginx-controller-admission
Add Below Entries

  - appProtocol: ssh
    name: bitbucket-ssh
    port: 7999
    protocol: TCP
    targetPort: ssh

Edit Default network-ingress-policy

#kubectl -n kube-system edit NetworkPolicy default-network-ingress-policy
Add Below Entries

  - port: 7999
    protocol: TCP

[brandond]

With the default rke2 installation which contains pre-configured rke2-ingress-inginx-controller as daemonset, I do not see any process that is listening on port 443 but I can still connect to it. Is there anything that I am missing?

Yes, the fact that Kubernetes pods that don't run with host network use a completely different network namespace that doesn't show up in ss. If you want to poke at the pods, you can do something like this:

for NS in /var/run/netns/*; do ss -tulpn -N $(basename $NS); done

[gauravkarki]

Thank you @brandond. I am able to see the ports with your command.

#for NS in /var/run/netns/*; do ss -tulpn -N $(basename $NS); done | grep 7999
tcp   LISTEN 0      128          0.0.0.0:7999       0.0.0.0:*    users:(("nginx",pid=323727,fd=145)

Since the port were exposed as hostPort, I was expecting that it would show on ss commands output.

        name: rke2-ingress-nginx-controller
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        - containerPort: 7999
          hostPort: 7999
          name: ssh
          protocol: TCP

[brandond]

Since the port were exposed as hostPort, I was expecting that it would show on ss commands output.

That is not how hostPorts work. hostNetwork runs the entire pod in the host's network namespace. hostPort just sets up iptables rules to forward traffic from the host port into the pod.

[michal-kuzak]

I know, old thread .. so on RKE1 you only had to do this:

Terraform:

   ingress {
      extra_args = {
        tcp-services-configmap = "$(POD_NAMESPACE)/tcp-services"
        udp-services-configmap = "$(POD_NAMESPACE)/udp-services"
      }
    }

  kubectl patch configmap tcp-services -n ingress-nginx --patch '{"data":{"9042":"k8ssandra/k8ssandra-dc1-stargate-service:9042"}}'

Now you have to do all these steps (HelmChartConfigs, ConfigMap, rke2-ingress-controller-admission service, and Network-Ingress-Policy) to just add a port ?