Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.27] - k3s-agent loadbalancer is not using the HTTP*_PROXY var #9114

Closed
dereknola opened this issue Jan 2, 2024 · 1 comment
Closed

[Release-1.27] - k3s-agent loadbalancer is not using the HTTP*_PROXY var #9114

dereknola opened this issue Jan 2, 2024 · 1 comment
Assignees
@dereknola @fmoral2

Comments

@dereknola
Copy link
Member

Backport fix for k3s-agent loadbalancer is not using the HTTP*_PROXY var
@dereknola dereknola self-assigned this Jan 2, 2024
@dereknola dereknola mentioned this issue Jan 2, 2024
Merged
@dereknola dereknola moved this from New to Peer Review in K3s Development Jan 3, 2024
@dereknola dereknola mentioned this issue Jan 11, 2024
Merged
@fmoral2
Copy link
Contributor

fmoral2 commented Jan 15, 2024 •
edited
Loading

Validated on Version:

-$ k3s version v1.27.9+k3s-138c5c99 (138c5c99)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"

Cluster Configuration:
1 node server
1 agent
1 proxy server

Steps to validate the fix

  1. Create and configure a proxy server to accept cluster ips and respective ports
  2. Create a server
  3. export http_proxy=http://:6443 and https_proxy=https://:6433 on the agent
  4. Install and join agent using the K3S_AGENT_HTTP_PROXY_ALLOWED=true
  5. Validate it works
  6. Validate Nodes and pods are ok
  7. Validate that when requested agent node calls proxy to then proxy calls the server

Reproduction Issue:

Validation Results:

---------  Proxy node  ------------
$ sudo apt-get update
$ sudo apt-get install squid

$ sudo nano /etc/squid/squid.conf

acl agent src AGENT-IP
acl  server src SERVER-IP 
http_access allow  agent
http_access allow server

$ sudo systemctl restart squid

$ sudo systemctl enable squid

--------- AFTER JOIN AGENT ON THE CONTROL PLANE -----------

~$ sudo tail -f /var/log/squid/access.log
1705350952.255    642 AGENT-IP TCP_TUNNEL/200 37234 CONNECT get.k3s.io:443 - HIER_DIRECT/ 
1705350952.494    156 AGENT-IP TCP_TUNNEL/200 6057 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705350954.862   2357 AGENT-IP TCP_TUNNEL/200 61039063 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705351047.729    171 AGENT-IP TCP_TUNNEL/200 37227 CONNECT get.k3s.io:443 - HIER_DIRECT/ 
1705351048.337    144 AGENT-IP TCP_TUNNEL/200 6057 CONNECT k3s-ci-builds.s3.amazonaws.com:443 - HIER_DIRECT/ 
1705351056.215     41 AGENT-IP TCP_TUNNEL/200 6252 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351057.147     40 AGENT-IP TCP_TUNNEL/200 6244 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351058.445   1254 AGENT-IP TCP_TUNNEL/200 9782 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351061.109   5043 AGENT-IP TCP_TUNNEL/200 10362 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351061.162   5166 AGENT-IP TCP_TUNNEL/200 6594 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351061.260   5044 AGENT-IP TCP_TUNNEL/200 10357 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351061.537   5274 AGENT-IP TCP_TUNNEL/200 11633 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351062.032   5047 AGENT-IP TCP_TUNNEL/200 10353 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351062.098   5151 AGENT-IP TCP_TUNNEL/200 6590 CONNECT registry-1.docker.io:443 - HIER_DIRECT -
1705351062.189   5041 AGENT-IP TCP_TUNNEL/200 10353 CONNECT auth.docker.io:443 - HIER_DIRECT/ 
1705351062.529   5051 AGENT-IP TCP_TUNNEL/200 6163 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351062.536   5050 AGENT-IP TCP_TUNNEL/200 6165 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705351086.594  30166 AGENT-IP TCP_TUNNEL/200 304206 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 
1705351087.626  30244 AGENT-IP TCP_TUNNEL/200 3413470 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 
1705351088.469  30931 AGENT-IP TCP_TUNNEL/200 1527271 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/ 


--------AGENT NODE----------------

curl -sfL https://get.k3s.io |  INSTALL_K3S_COMMIT=138c5c99ae3e304ba1e5a20419b51e2c988e9cc0  K3S_AGENT_HTTP_PROXY_ALLOWED=true  K3S_AGENT_NODE_NAME='agent-node'  sh -s - agent --token  :server:secret --server  https://SERVER-IP:6443


~$ k3s -v
k3s version v1.27.9+k3s-138c5c99 (138c5c99)
go version go1.20.12


~$  kubectl get nodes,pods -A
NAME                    STATUS   ROLES                       AGE   VERSION
node/ip-    Ready    control-plane,etcd,master   89s   v1.27.9+k3s-138c5c99
node/ip-    Ready    <none>                      54s   v1.27.9+k3s-138c5c99

NAMESPACE     NAME                                         READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-77ccd57875-ntppb                 1/1     Running     0          74s
kube-system   pod/helm-install-traefik-crd-rwcv9           0/1     Completed   0          75s
kube-system   pod/helm-install-traefik-wrfbd               0/1     Completed   1          75s
kube-system   pod/local-path-provisioner-957fdf8bc-5wplt   1/1     Running     0          74s
kube-system   pod/metrics-server-648b5df564-2ctmt          1/1     Running     0          74s
kube-system   pod/svclb-traefik-4ef6a241-4z6t7             2/2     Running     0          53s
kube-system   pod/svclb-traefik-4ef6a241-t2m7f             2/2     Running     0          60s
kube-system   pod/traefik-768bdcdcdd-c6hgc                 1/1     Running     0          60s



------ Reach any server on the internet ------
~$ curl -v https://api.github.com
* Uses proxy env variable https_proxy == 'http://proxy-ip:3128'
*   Trying proxy-ip:3128...
* Connected to (nil) (proxy-ip) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api.github.com:443
> CONNECT api.github.com:443 HTTP/1.1
> Host: api.github.com:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):


-------- logs from the proxy --------

1705063229.003    124 agent-ip TCP_TUNNEL/200 6490 CONNECT api.github.com:443 - HIER_DIRECT/  -
1705063264.512    180 agent-ip TCP_TUNNEL/200 6512 CONNECT api.github.com:443 - HIER_DIRECT/  -


1705352928.786     16 AGENT-IP TCP_TUNNEL/200 1808 CONNECT SERVER-IP:6443 - HIER_DIRECT/SERVER-IP -
1705352928.791     20 AGENT-IP TCP_TUNNEL/200 1808 CONNECT SERVER-IP:6443 - HIER_DIRECT/SERVER-IP -
1705352928.791      8 AGENT-IP TCP_TUNNEL/200 2138 CONNECT SERVER-IP:6443 - HIER_DIRECT/SERVER-IP -
1705352928.944    151 AGENT-IP TCP_TUNNEL/200 1563 CONNECT SERVER-IP:6443 - HIER_DIRECT/SERVER-IP -
1705352929.116    171 AGENT-IP TCP_TUNNEL/200 3762 CONNECT SERVER-IP:6443 - HIER_DIRECT/SERVER-IP -
1705352930.081     44 AGENT-IP TCP_TUNNEL/200 6252 CONNECT registry-1.docker.io:443 - HIER_DIRECT/ 
1705352931.138     42 AGENT-IP TCP_TUNNEL/200 6244 CONNECT registry-1.docker.io:443 - HIER_DIRECT/
1705352931.621     49 AGENT-IP TCP_TUNNEL/200 6139 CONNECT registry-1.docker.io:443 - HIER_DIRECT/
1705352931.806    196 AGENT-IP TCP_TUNNEL/200 4427981 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/
1705352934.956   5041 AGENT-IP TCP_TUNNEL/200 10357 CONNECT auth.docker.io:443 - HIER_DIRECT/
1705352935.019   5175 AGENT-IP TCP_TUNNEL/200 6594 CONNECT registry-1.docker.io:443 - HIER_DIRECT/
1705352935.134   5048 AGENT-IP TCP_TUNNEL/200 10357 CONNECT auth.docker.io:443 - HIER_DIRECT/
1705352935.460   5324 AGENT-IP TCP_TUNNEL/200 11633 CONNECT registry-1.docker.io:443 - HIER_DIRECT/
1705352936.023   5043 AGENT-IP TCP_TUNNEL/200 10347 CONNECT auth.docker.io:443 - HIER_DIRECT/
1705352936.086   5153 AGENT-IP TCP_TUNNEL/200 6590 CONNECT registry-1.docker.io:443 - HIER_DIRECT/3
1705352936.181   5042 AGENT-IP TCP_TUNNEL/200 10346 CONNECT auth.docker.io:443 - HIER_DIRECT/5
1705352936.609   5426 AGENT-IP TCP_TUNNEL/200 10261 CONNECT registry-1.docker.io:443 - HIER_DIRECT/
1705352946.644  15022 AGENT-IP TCP_TUNNEL/200 3020 CONNECT production.cloudflare.docker.com:443 - HIER_DIRECT/

-------
$ curl -vs  https://server-IP:6444/cacerts -k
 Uses proxy env variable https_proxy == 'http://proxy-ip:3128'
*   Trying proxy-ip:3128...
* Connected to (nil) (proxy-ip) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to server-ip:6443
> CONNECT server-ip:6443 HTTP/1.1
> Host: server-ip:6443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Jan 12 11:59:12 2024 GMT
*  expire date: Jan 11 11:59:12 2025 GMT
*  issuer: CN=k3s-server- 
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5581351eb550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /cacerts HTTP/2
> Host: server-ip:6443
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< content-type: text/plain
< content-length: 570
< date: Fri, 12 Jan 2024 12:46:00 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
-----BEGIN CERTIFICATE-----
redacted**********
**************
-----END CERTIFICATE-----



$ sudo cat  /etc/systemd/system/k3s-agent.service.env
K3S_AGENT_HTTP_PROXY_ALLOWED='true' 
K3S_AGENT_NODE_NAME='agent-node'
 
HTTPS_PROXY='http://PROXY-IP:3128'
HTTP_PROXY='http://PROXY-IP:3128'
http_proxy='http://PROXY-IP:3128'
https_proxy='http://PROXY-IP:3128'



------ SERVER NODE ---------


LOGS FROM TCPDUMPS ON HOST AGENT-IP
....

20:47:03.639043 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42248 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 36017, win 443, options [nop,nop,TS val 35  ecr 35 ], length 0
20:47:04.933355 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42160 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 4636:4664, ack 4228, win 480, options [nop,nop,TS val 35 5 ecr 3 ], length 28
20:47:04.933462 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42160: Flags [P.], seq 4228:4252, ack 4664, win 479, options [nop,nop,TS val 35  ecr 352], length 24
20:47:04.933596 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42160 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 4252, win 480, options [nop,nop,TS val 35  ecr 352 ], length 0
20:47:05.334888 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 66588:66628, ack 505811, win 3004, options [nop,nop,TS val 35 ecr 352 ], length 40
20:47:05.334889 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 66628:67155, ack 505811, win 3004, options [nop,nop,TS val 35  ecr 352 ], length 527
20:47:05.334945 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [.], ack 66628, win 710, options [nop,nop,TS val 3520  ecr 352 ], length 0
20:47:05.334964 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [.], ack 67155, win 763, options [nop,nop,TS val 352  ecr 352 ], length 0
20:47:05.338012 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 505811:505901, ack 67155, win 763, options [nop,nop,TS val 35  ecr 352 ], length 90
20:47:05.338036 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 505901:506428, ack 67155, win 763, options [nop,nop,TS val 35  ecr 35 ], length 527
20:47:05.338216 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [.], ack 506428, win 3004, options [nop,nop,TS val 35  ecr 35 ], length 0
20:47:06.570214 ens5  In  IP AGENT-IP.us-east-2.compute.internal.42200 > SERVER-IP.us-east-2.compute.internal.6443: Flags [P.], seq 67155:67193, ack 506428, win 3004, options [nop,nop,TS val 3  ecr 35 ], length 38
20:47:06.571556 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > AGENT-IP.us-east-2.compute.internal.42200: Flags [P.], seq 506428:506518, ack 67193, win 763, options [nop,nop,TS val 35  ecr 352 ], length 90
20:47:06.571599 ens5  Out IP SERVER-IP.us-east-2.compute.internal.6443 > ip-172 
....

@fmoral2 fmoral2 closed this as completed Jan 15, 2024
@github-project-automation github-project-automation bot moved this from Peer Review to Done Issue in K3s Development Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dereknola dereknola

@fmoral2 fmoral2

Labels
None yet
Projects
K3s Development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dereknola @fmoral2