k3s-agent loadbalancer is not using the HTTP*_PROXY var #8976

pierre-az · 2023-12-01T13:49:47Z

Environmental Info:
K3s Version:
k3s version v1.27.7+k3s2 (575bce7) go version go1.20.10

Node(s) CPU architecture, OS, and Version:

Linux azmed 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 servers setup behind a loadbalancer, 50ish agents.
Agents are installed in various areas with unique network configuration each time, which we can't access fully so logs will be limited to the agent environment.

Describe the bug:

Agents fail to contact & register the loadbalancer whenever there is a proxy in use, typically set as an env variable with HTTPS_PROXY, HTTP_PROXY, NO_PROXY and its lowercase alternative (CONTAINERD_ prefixed as well.

The agent logs from a startup sequence:

`déc. 01 13:29:33 azmed systemd[1]: k3s-agent.service: Succeeded.
déc. 01 13:29:33 azmed systemd[1]: Stopped Lightweight Kubernetes.
déc. 01 13:29:33 azmed systemd[1]: Starting Lightweight Kubernetes...
déc. 01 13:29:33 azmed sh[3590590]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
déc. 01 13:29:33 azmed sh[3590598]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
déc. 01 13:29:33 azmed k3s[3590612]: time="2023-12-01T13:29:33Z" level=info msg="Starting k3s agent v1.27.7+k3s2 (575bce76)"
déc. 01 13:29:33 azmed k3s[3590612]: time="2023-12-01T13:29:33Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: redacted:443"
déc. 01 13:29:33 azmed k3s[3590612]: time="2023-12-01T13:29:33Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [redacted:443] [default: redacted:443]"
déc. 01 13:29:53 azmed k3s[3590612]: time="2023-12-01T13:29:53Z" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
déc. 01 13:30:15 azmed k3s[3590612]: time="2023-12-01T13:30:15Z" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"`

To test the URLs is reachable from the machine, we did try:

` curl -vs https:///redacted:443/cacerts -k
*   Trying redacted...
* TCP_NODELAY set
* Connected to redacted (redacted) port redacted (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to redacted:443
> CONNECT redacted:443 HTTP/1.1
> Host: redacted:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Oct 24 07:41:52 2023 GMT
*  expire date: Nov 19 12:03:35 2024 GMT
*  issuer: CN=k3s-server-ca@1698133312
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5642f96f1630)
> GET /cacerts HTTP/2
> Host: redacted
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< content-type: text/plain
< content-length: 566
< date: Fri, 01 Dec 2023 13:33:38 GMT
<
-----BEGIN CERTIFICATE-----
<certificate>
-----END CERTIFICATE-----
* Connection #0 to host redacted left intact`

Confirming it is reachable using the proxy variables set on the system as follows:
env |grep proxy https_proxy=redacted http_proxy=redacted

We also try to reach the k3s-agent loadbalancer locally:

`curl https://127.0.0.1:6444/cacerts --noproxy '127.0.0.1' -v -k
*   Trying 127.0.0.1:6444...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444`

Confirming a bit more the issue with the loadbalancer.

I'm not pasting tcpdumps logs here, but capturing outgoing packets to the IP of the master node directly shows dropped traffic, whereas it should show nothing since we should go through the configured proxy, which is to me the final hint of the proxy not being used by k3s loadbalancer.

Steps To Reproduce:

Installed K3s:
We installed K3S through:
curl -sfL https://get.k3s.io | K3S_URL='' K3S_TOKEN='' K3S_NODE_NAME='' sh -

Expected behavior:

We expected the traffic to go through the configured HTTP_PROXY & HTTPS_PROXY.
Actual behavior:

The traffic is actually bypassing our proxy settings.

Additional context / logs:

Just to confirm our ENV variables were taken into account by the service, we checked the /proc//environ:

`PATH=redacted                                                                     
NOTIFY_SOCKET=/run/systemd/notify                                                                                INVOCATION_ID=acafcbd61fd44d2a9093435050ba7b58                                                                   JOURNAL_STREAM=9:51168612                                                                                       
K3S_NODE_NAME=node
K3S_TOKEN=redacted         
K3S_URL=redacted                                                                   
HTTPS_PROXY=redacted                                                                                   
HTTP_PROXY=redacted                                                                                    
http_proxy=redacted                                                                                   
https_proxy=redacted                                                                                   RES_OPTIONS=                                                                                                     
K3S_DATA_DIR=redacted4`

I'm available for any tests/logs you need !

The text was updated successfully, but these errors were encountered:

brandond · 2023-12-01T22:49:50Z

The agent load-balancer is not supposed to use a proxy to connect to the server. You should ensure that the agent has direct connectivity to all of the listed ports on the servers, as well as the CNI ports on all cluster members. This is covered in the docs at https://docs.k3s.io/installation/requirements#networking

Even if we did allow the load-balancer to use a proxy, you'd still need to open up other ports to make the CNI work - so why bother.

pierre-az · 2023-12-02T14:19:45Z

Thanks for your answer.
To add a bit more context, our use ok K3S is a bit unusual, as agent nodes don't communicate with other agents in any way, thus us not taking into account the CNI ports on other agents.
As I'm no Go expert, I'm not certain of the implications of a change in the loadbalancer functions, however I will try to fork and implement an option for that purpose. If I was to manage that, would you have an interest in the functionnality or would you consider that outside K3S philosophy?

caroline-suse-rancher · 2024-01-02T20:04:02Z

Reopened because the associate PR is still in progress

fmoral2 · 2024-01-15T21:04:22Z

Hey all!

so here follows my findings:

TLDR:

Commit id before these changes its working fine for me with setting https_proxy vars i can join normally an agent using the proxy

Commit id for last changes its also working fine also with the usage of K3S_AGENT_HTTP_PROXY_ALLOWED=true the only "odd" behavior is that even proxy is working fine and showing comms happening thourgh that , on tcpdumps logs still shows communications between server and agent

Validated on Version:

-$ k3s version 1.29.0+k3s-c87e6e5f

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"

Cluster Configuration:
1 node server
1 agent
1 proxy server

Steps to validate the fix

Create and configure a proxy server to accept cluster ips and respective ports
Create a server
export http_proxy=http://:6443 and https_proxy=https://:6433 on the agent
Install and join agent using the K3S_AGENT_HTTP_PROXY_ALLOWED=true
Validate it works
Validate Nodes and pods are ok
Validate that when requested agent node calls proxy to then proxy calls the server

Reproduction Issue:

 

- CANT REPRODUCE IT
 
---------  Proxy node  ------------
$ sudo apt-get update
$ sudo apt-get install squid

$ sudo nano /etc/squid/squid.conf

acl agent src AGENT-IP
acl  server src 172.31.19.68 
http_access allow  agent
http_access allow server

$ sudo systemctl restart squid

$ sudo systemctl enable squid

--------- AFTER JOIN AGENT ON THE CONTROL PLANE -----------

~$ sudo tail -f /var/log/squid/access.log
1705350952.255    642 AGENT-IP TCP_TUNNEL/200 37234 CONNECT get.k3s.io:443 - HIER_DIRECT/ 
1705350952.494    156 AGENT-IP TCP_TUNNEL/200 6057 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705350954.862   2357 AGENT-IP TCP_TUNNEL/200 61039063 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705351047.729    171 AGENT-IP TCP_TUNNEL/200 37227 CONNECT get.k3s.io:443 - HIER_DIRECT/ 
1705351048.337    144 AGENT-IP TCP_TUNNEL/200 6057 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705351056.215     41 AGENT-IP TCP_TUNNEL/200 6252 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351057.147     40 AGENT-IP TCP_TUNNEL/200 6244 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351058.445   1254 AGENT-IP TCP_TUNNEL/200 9782 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351061.109   5043 AGENT-IP TCP_TUNNEL/200 10362 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351061.162   5166 AGENT-IP TCP_TUNNEL/200 6594 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351061.260   5044 AGENT-IP TCP_TUNNEL/200 10357 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351061.537   5274 AGENT-IP TCP_TUNNEL/200 11633 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351062.032   5047 AGENT-IP TCP_TUNNEL/200 10353 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351062.098   5151 AGENT-IP TCP_TUNNEL/200 6590 CONNECT registry-1.docker.io:443 - HIER_DIRECT -
1705351062.189   5041 AGENT-IP TCP_TUNNEL/200 10353 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351062.529   5051 AGENT-IP TCP_TUNNEL/200 6163 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351062.536   5050 AGENT-IP TCP_TUNNEL/200 6165 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351086.594  30166 AGENT-IP TCP_TUNNEL/200 304206 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 
1705351087.626  30244 AGENT-IP TCP_TUNNEL/200 3413470 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 
1705351088.469  30931 AGENT-IP TCP_TUNNEL/200 1527271 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 


--------AGENT NODE----------------

export HTTP_PROXY="http://PROXY-IP:3128"
export HTTPS_PROXY="http://PROXY-IP:3128"
export http_proxy="http://PROXY-IP:3128"
export https_proxy="http://PROXY-IP:3128"

$ curl -sfL https://get.k3s.io |  INSTALL_K3S_COMMIT=aca1c2fd114f2c755c6a307a7e7b4a827dfd6b32 K3S_URL='https://SERVER-IP:6443'  K3S_AGENT_NODE_NAME='agent-node'  sh -s - agent --token   :server:secret

$ curl -vs  https://SERVER-IP:6443/cacerts -k

Uses proxy env variable https_proxy == 'http:/PROXY-IP:3128'
*   Trying PROXY-IP:3128...
* Connected to (nil) (PROXY-IP) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to SERVER-IP:6443
> CONNECT SERVER-IP:6443 HTTP/1.1
> Host: SERVER-IP:6443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Jan 15 20:32:59 2024 GMT
*  expire date: Jan 14 20:33:00 2025 GMT
*  issuer: CN=k3s-server 
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5654ffa53550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /cacerts HTTP/2
> Host: SERVER-IP:6443
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< content-type: text/plain
< content-length: 566
< date: Mon, 15 Jan 2024 20:46:38 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
-----BEGIN CERTIFICATE-----
 ..............
-----END CERTIFICATE-----


$ sudo cat  /etc/systemd/system/k3s-agent.service.env
K3S_AGENT_NODE_NAME='agent-node'
K3S_URL='https://SERVER-IP:6443'
HTTPS_PROXY='http://PROXY-IP:3128'
HTTP_PROXY='http://PROXY-IP:3128'
http_proxy='http://PROXY-IP:3128'
https_proxy='http://PROXY-IP:3128'





------- SERVER NODE ----------

$ kubectl get nodes,pods -A
NAME                    STATUS   ROLES                       AGE   VERSION
node/    Ready    control-plane,etcd,master   15m   v1.29.0+k3s-aca1c2fd
node/    Ready    <none>                      11m   v1.29.0+k3s-aca1c2fd

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-6799fbcd5-zbtkf                   1/1     Running     0          15m
kube-system   pod/helm-install-traefik-5mthx                0/1     Completed   1          15m
kube-system   pod/helm-install-traefik-crd-nkwrk            0/1     Completed   0          15m
kube-system   pod/local-path-provisioner-84db5d44d9-j59hk   1/1     Running     0          15m
kube-system   pod/metrics-server-67c658944b-c7bq8           1/1     Running     0          15m
kube-system   pod/svclb-traefik-14bfa2d1-5xs7z              2/2     Running     0          11m
kube-system   pod/svclb-traefik-14bfa2d1-mzwp9              2/2     Running     0          15m
kube-system   pod/traefik-f4564c4f4-66w2t                   1/1     Running     0          15m
 



------ SERVER NODE --------

 LOGS FROM TCPDUMPS ON HOST AGENT-IP
....

20:47:03.639043 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42248 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 36017, win 443, options [nop,nop,TS val 35  ecr 35 ], length 0
20:47:04.933355 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42160 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 4636:4664, ack 4228, win 480, options [nop,nop,TS val 35 5 ecr 3 ], length 28
20:47:04.933462 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42160: Flags [P.], seq 4228:4252, ack 4664, win 479, options [nop,nop,TS val 35  ecr 352], length 24
20:47:04.933596 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42160 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 4252, win 480, options [nop,nop,TS val 35  ecr 352 ], length 0
20:47:05.334888 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 66588:66628, ack 505811, win 3004, options [nop,nop,TS val 35 ecr 352 ], length 40
20:47:05.334889 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 66628:67155, ack 505811, win 3004, options [nop,nop,TS val 35  ecr 352 ], length 527
20:47:05.334945 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [.], ack 66628, win 710, options [nop,nop,TS val 3520  ecr 352 ], length 0
20:47:05.334964 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [.], ack 67155, win 763, options [nop,nop,TS val 352  ecr 352 ], length 0
20:47:05.338012 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 505811:505901, ack 67155, win 763, options [nop,nop,TS val 35  ecr 352 ], length 90
20:47:05.338036 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 505901:506428, ack 67155, win 763, options [nop,nop,TS val 35  ecr 35 ], length 527
20:47:05.338216 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 506428, win 3004, options [nop,nop,TS val 35  ecr 35 ], length 0
20:47:06.570214 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 67155:67193, ack 506428, win 3004, options [nop,nop,TS val 3  ecr 35 ], length 38
20:47:06.571556 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 506428:506518, ack 67193, win 763, options [nop,nop,TS val 35  ecr 352 ], length 90
20:47:06.571599 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > ip-172 
....

Validation Results:

---------  Proxy node  ------------
$ sudo apt-get update
$ sudo apt-get install squid

$ sudo nano /etc/squid/squid.conf

acl agent src AGENT-IP
acl  server src SERVER-IP
http_access allow  agent
http_access allow server

$ sudo systemctl restart squid

$ sudo systemctl enable squid

--------- AFTER JOIN AGENT ON THE CONTROL PLANE -----------

~$ sudo tail -f /var/log/squid/access.log
1705350952.255    642 AGENT-IP TCP_TUNNEL/200 37234 CONNECT get.k3s.io:443 - HIER_DIRECT/ 
1705350952.494    156 AGENT-IP TCP_TUNNEL/200 6057 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705350954.862   2357 AGENT-IP TCP_TUNNEL/200 61039063 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705351047.729    171 AGENT-IP TCP_TUNNEL/200 37227 CONNECT get.k3s.io:443 - HIER_DIRECT/ 
1705351048.337    144 AGENT-IP TCP_TUNNEL/200 6057 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705351056.215     41 AGENT-IP TCP_TUNNEL/200 6252 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351057.147     40 AGENT-IP TCP_TUNNEL/200 6244 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351058.445   1254 AGENT-IP TCP_TUNNEL/200 9782 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351061.109   5043 AGENT-IP TCP_TUNNEL/200 10362 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351061.162   5166 AGENT-IP TCP_TUNNEL/200 6594 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351061.260   5044 AGENT-IP TCP_TUNNEL/200 10357 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351061.537   5274 AGENT-IP TCP_TUNNEL/200 11633 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351062.032   5047 AGENT-IP TCP_TUNNEL/200 10353 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351062.098   5151 AGENT-IP TCP_TUNNEL/200 6590 CONNECT registry-1.docker.io:443 - HIER_DIRECT -
1705351062.189   5041 AGENT-IP TCP_TUNNEL/200 10353 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351062.529   5051 AGENT-IP TCP_TUNNEL/200 6163 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351062.536   5050 AGENT-IP TCP_TUNNEL/200 6165 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351086.594  30166 AGENT-IP TCP_TUNNEL/200 304206 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 
1705351087.626  30244 AGENT-IP TCP_TUNNEL/200 3413470 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 
1705351088.469  30931 AGENT-IP TCP_TUNNEL/200 1527271 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 


--------AGENT NODE----------------

curl -sfL https://get.k3s.io |  INSTALL_K3S_COMMIT=bbd68f3a507903442ea093ae5226a7026b307964  K3S_AGENT_HTTP_PROXY_ALLOWED=true  K3S_AGENT_NODE_NAME='agent-node'  sh -s - agent --token ::server:secre  --server  https://server-ip:6443


~$ k3s -v
k3s version v1.29.0+k3s-c87e6e5f (c87e6e5f)
go version go1.21.5
 

~$ kubectl get nodes,pods -A
NAME                    STATUS   ROLES                       AGE    VERSION
node/ip    Ready    <none>                      6m7s   v1.29.0+k3s-c87e6e5f
node/ip    Ready    control-plane,etcd,master   39m    v1.29.0+k3s-c87e6e5f

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-6799fbcd5-jmqfq                   1/1     Running     0          38m
kube-system   pod/helm-install-traefik-crd-4nkk7            0/1     Completed   0          38m
kube-system   pod/helm-install-traefik-zxvzl                0/1     Completed   2          38m
kube-system   pod/local-path-provisioner-84db5d44d9-778xg   1/1     Running     0          38m
kube-system   pod/metrics-server-67c658944b-pwzj7           1/1     Running     0          38m
kube-system   pod/svclb-traefik-a4e6455c-kcb5l              2/2     Running     2          29m
kube-system   pod/svclb-traefik-a4e6455c-xkwcc              2/2     Running     0          38m
kube-system   pod/traefik-f4564c4f4-6wfrq                   1/1     Running     0          38m



------ Reach any server on the internet ------
~$ curl -v https://api.github.com
* Uses proxy env variable https_proxy == 'http://proxy-ip:3128'
*   Trying proxy-ip:3128...
* Connected to (nil) (proxy-ip) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api.github.com:443
> CONNECT api.github.com:443 HTTP/1.1
> Host: api.github.com:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):


-------- logs from the proxy --------

1705063229.003    124 agent-ip TCP_TUNNEL/200 6490 CONNECT api.github.com:443 - HIER_DIRECT/  -
1705063264.512    180 agent-ip TCP_TUNNEL/200 6512 CONNECT api.github.com:443 - HIER_DIRECT/  -


-------
$ curl -vs  https://server-IP:6444/cacerts -k
 Uses proxy env variable https_proxy == 'http://proxy-ip:3128'
*   Trying proxy-ip:3128...
* Connected to (nil) (proxy-ip) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to server-ip:6443
> CONNECT server-ip:6443 HTTP/1.1
> Host: server-ip:6443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Jan 12 11:59:12 2024 GMT
*  expire date: Jan 11 11:59:12 2025 GMT
*  issuer: CN=k3s-server- 
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5581351eb550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /cacerts HTTP/2
> Host: server-ip:6443
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< content-type: text/plain
< content-length: 570
< date: Fri, 12 Jan 2024 12:46:00 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
-----BEGIN CERTIFICATE-----
redacted**********
**************
-----END CERTIFICATE-----



$ sudo cat  /etc/systemd/system/k3s-agent.service.env
K3S_AGENT_HTTP_PROXY_ALLOWED='true'  
K3S_AGENT_NODE_NAME='agent-node'
HTTPS_PROXY='http://PROXY-IP:3128'
HTTP_PROXY='http://PROXY-IP:3128'
http_proxy='http://PROXY-IP:3128'
https_proxy='http://PROXY-IP:3128'



------ SERVER NODE ---------


LOGS FROM TCPDUMPS ON HOST AGENT-IP
....

20:47:03.639043 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42248 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 36017, win 443, options [nop,nop,TS val 35  ecr 35 ], length 0
20:47:04.933355 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42160 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 4636:4664, ack 4228, win 480, options [nop,nop,TS val 35 5 ecr 3 ], length 28
20:47:04.933462 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42160: Flags [P.], seq 4228:4252, ack 4664, win 479, options [nop,nop,TS val 35  ecr 352], length 24
20:47:04.933596 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42160 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 4252, win 480, options [nop,nop,TS val 35  ecr 352 ], length 0
20:47:05.334888 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 66588:66628, ack 505811, win 3004, options [nop,nop,TS val 35 ecr 352 ], length 40
20:47:05.334889 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 66628:67155, ack 505811, win 3004, options [nop,nop,TS val 35  ecr 352 ], length 527
20:47:05.334945 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [.], ack 66628, win 710, options [nop,nop,TS val 3520  ecr 352 ], length 0
20:47:05.334964 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [.], ack 67155, win 763, options [nop,nop,TS val 352  ecr 352 ], length 0
20:47:05.338012 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 505811:505901, ack 67155, win 763, options [nop,nop,TS val 35  ecr 352 ], length 90
20:47:05.338036 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 505901:506428, ack 67155, win 763, options [nop,nop,TS val 35  ecr 35 ], length 527
20:47:05.338216 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 506428, win 3004, options [nop,nop,TS val 35  ecr 35 ], length 0
20:47:06.570214 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 67155:67193, ack 506428, win 3004, options [nop,nop,TS val 3  ecr 35 ], length 38
20:47:06.571556 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 506428:506518, ack 67193, win 763, options [nop,nop,TS val 35  ecr 352 ], length 90
20:47:06.571599 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > ip-172 
....

brandond · 2024-01-15T21:10:03Z

@fmoral2

$ sudo cat  /etc/systemd/system/k3s-agent.service.env
K3S_AGENT_NODE_NAME='agent-node'
K3S_URL='https://SERVER-IP:6443'
HTTPS_PROXY='http://PROXY-IP:3128'
HTTP_PROXY='http://PROXY-IP:3128'
http_proxy='http://PROXY-IP:3128'
https_proxy='http://PROXY-IP:3128'

Shouldn't K3S_AGENT_HTTP_PROXY_ALLOWED=true be in here? Otherwise, I'm not sure you're actually testing this change.

fmoral2 · 2024-01-15T21:53:32Z

@fmoral2
$ sudo cat  /etc/systemd/system/k3s-agent.service.env
K3S_AGENT_NODE_NAME='agent-node'
K3S_URL='https://SERVER-IP:6443'
HTTPS_PROXY='http://PROXY-IP:3128'
HTTP_PROXY='http://PROXY-IP:3128'
http_proxy='http://PROXY-IP:3128'
https_proxy='http://PROXY-IP:3128'
Shouldn't K3S_AGENT_HTTP_PROXY_ALLOWED=true be in here? Otherwise, I'm not sure you're actually testing this change.
@brandond

Tk you ! it was in there , this was just a copy and paste mistake from the reproduction steps, which does not have this var

Already fixed, tks again

github-project-automation bot added this to K3s Development Dec 1, 2023

github-project-automation bot moved this to New in K3s Development Dec 1, 2023

brandond closed this as completed Dec 1, 2023

github-project-automation bot moved this from New to Done Issue in K3s Development Dec 1, 2023

pierre-az mentioned this issue Dec 4, 2023

Added support for env *_PROXY variables for agent loadbalancer #8982

Closed

pierre-az mentioned this issue Dec 19, 2023

Added support for env *_PROXY variables for agent loadbalancer #9070

Merged

caroline-suse-rancher reopened this Jan 2, 2024

caroline-suse-rancher assigned dereknola Jan 2, 2024

caroline-suse-rancher moved this from Done Issue to Peer Review in K3s Development Jan 2, 2024

caroline-suse-rancher added the kind/bug Something isn't working label Jan 2, 2024

This was referenced Jan 2, 2024

[Release-1.26] - k3s-agent loadbalancer is not using the HTTP*_PROXY var #9113

Closed

[Release-1.27] - k3s-agent loadbalancer is not using the HTTP*_PROXY var #9114

Closed

[Release-1.28] - k3s-agent loadbalancer is not using the HTTP*_PROXY var #9115

Closed

dereknola moved this from Peer Review to To Test in K3s Development Jan 3, 2024

VestigeJ assigned fmoral2 Jan 8, 2024

brandond mentioned this issue Jan 11, 2024

Move proxy dialer out of init() and fix crash when using K3S_AGENT_HTTP_PROXY_ALLOWED=true #9219

Merged

fmoral2 closed this as completed Jan 15, 2024

github-project-automation bot moved this from To Test to Done Issue in K3s Development Jan 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k3s-agent loadbalancer is not using the HTTP*_PROXY var #8976

k3s-agent loadbalancer is not using the HTTP*_PROXY var #8976

pierre-az commented Dec 1, 2023

brandond commented Dec 1, 2023 •

edited

Loading

pierre-az commented Dec 2, 2023 •

edited

Loading

caroline-suse-rancher commented Jan 2, 2024

fmoral2 commented Jan 15, 2024 •

edited

Loading

Validation Results:

brandond commented Jan 15, 2024

fmoral2 commented Jan 15, 2024

k3s-agent loadbalancer is not using the HTTP*_PROXY var #8976

k3s-agent loadbalancer is not using the HTTP*_PROXY var #8976

Comments

pierre-az commented Dec 1, 2023

brandond commented Dec 1, 2023 • edited Loading

pierre-az commented Dec 2, 2023 • edited Loading

caroline-suse-rancher commented Jan 2, 2024

fmoral2 commented Jan 15, 2024 • edited Loading

Validated on Version:

Environment Details

Steps to validate the fix

Reproduction Issue:

Validation Results:

brandond commented Jan 15, 2024

fmoral2 commented Jan 15, 2024

brandond commented Dec 1, 2023 •

edited

Loading

pierre-az commented Dec 2, 2023 •

edited

Loading

fmoral2 commented Jan 15, 2024 •

edited

Loading