Issue with AWS multi-cluster setup

[fbozic]

Hi folks,
We have a peculiar issue with multi-cluster setup. We have 2 clusters, 1 in GCP and 1 in AWS. TL;DR communication from GCP to AWS doesn't work for some services.
Setup:

• 2 clusters
  • 1 in GCP
  • 1 in AWS.
• 2 namespaces in each cluster
  • l5d-server
    • nginx deployment and service - injected proxy to pod and exported service
    • http-echo deployment and service - injected proxy to pod and exported service
  • l5d-client
    • curl deployment - injected proxy to pod

Results:

• From curl deployment in AWS cluster we can communicate with all 4 services (2 x nginx, 2 x httepcho) ✅
• From curl deployment in GCP cluster we can communicate with only 3 services. It doesn't work for a nginx in AWS cluster. ❌
  Error:

< HTTP/1.1 504 Gateway Timeout
< l5d-proxy-error: client 10.0.4.23:54439: server: 10.193.120.101:4143: logical service nginx.l5d-server.svc.cluster.local:80: service in fail-fast
< date: Mon, 11 Dec 2023 13:04:44 GMT
< content-length: 0

I've checked that services are replicated and I've checked endpoints (with linkerd diagnostics endpoints). Everything looks okay to me.
I've checked this document: https://linkerd.buoyant.io/t/what-are-failfast-errors-in-linkerd-and-how-do-you-debug-them/85 but it didn't help.

I've attached nginx and http-echo manifests and screenshots of curl requests.
If trace proxy logs are needed let me know.

Curl manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
  namespace: l5d-client
spec:
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        # config.linkerd.io/proxy-log-level: trace
      labels:
        app: curl
    spec:
      containers:
        - name: curl
          image: curlimages/curl
          command: [/bin/sleep, 3650d]
          imagePullPolicy: IfNotPresent
      restartPolicy: Always

Nginx manifests

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: l5d-server
  labels:
    mirror.linkerd.io/exported: "true"
spec:
  selector:
    app: nginx
  type: ClusterIP
  ports:
    - name: app
      port: 80
      protocol: TCP
      targetPort: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: l5d-server
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:stable-alpine3.17
          ports:
            - name: app
              containerPort: 80

Http echo manifests

apiVersion: v1
kind: Service
metadata:
  name: httpecho
  namespace: l5d-server
  labels:
    mirror.linkerd.io/exported: "true"
spec:
  selector:
    app: httpecho
  type: ClusterIP
  ports:
    - name: app
      port: 80
      protocol: TCP
      targetPort: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpecho
  namespace: l5d-server
  labels:
    app: httpecho
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpecho
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: httpecho
    spec:
      containers:
        - name: httpecho
          image: mendhak/http-https-echo:31
          ports:
            - name: app
              containerPort: 8080

[alpeb]

Hi, thanks for the detailed information 👍
It's odd indeed that you can connect to http-echo in AWS but not to nginx... Can you provide the logs for the proxy on that target pod when the connection fails?

[alpeb]

I see no sign of l5d-client in those logs so the connection is not reaching that pod. Next up would be to check the proxy logs in both the curl pod in GCP and linkerd-multicluster in AWS.

[fbozic]

Here is the zip archive of linkerd multicluster proxies. We are running linkerd in HA mode so we have 3 proxies. I've found l5d-client in the logs but I am having a hard time understanding what exactly is going on.
logs.aws-l5d-mc-proxy.zip

[alpeb]

Those logs show that the gateway (3) is receiving the request, but is not able to forward it to the nginx pod, so it goes into fail-fast mode. Is there anything forbidding the gateway to reach nginx? A network policy?

[fbozic]

This is a fresh AWS cluster created just for linkerd integration. Not sure if it is worth noting but we are running AWS VPC CNI with a custom networking setup.
We don't use network policies at all.

> k get networkpolicies.networking.k8s.io -A
No resources found

What is super confusing for me is that httpecho works and nginx doesn't. The only difference I can spot between those 2 workloads is the port on which the container is listening (nginx is on 80, httpecho is on 8080).

Let me know if you need more info. We would love to wrap up AWS integration.

[fbozic]

Hi @alpeb. Is there anything else I can share or do so we can resolve this issue?