gRPC load-balancing does not seem to work with Envoy as client

[ori-nat]

Hi,

I'm trying to use Linkerd to achieve functional gRPC load-balancing with pod scaling awareness in my cluster.
Basically I have an Envoy pod that leverages an HTTP filter to send out calls to a cluster-local gRPC service. Currently, even with Linkerd installed and my cluster fully meshed, only one gRPC server pod seems to receive and handle the calls sent out from Envoy.

The configuration on Envoy's side is as follows:

grpc_service:
  initial_metadata:
  - key: l5d-dst-override
    value: target-grpc-service.default.svc.cluster.local:443
  google_grpc:
    target_uri: "target-grpc-service.default.svc.cluster.local:443"
    stat_prefix: ext_proc
    channel_credentials:
      ssl_credentials:
        root_certs:
          filename: "/etc/envoy/ca.crt"

The initial_metadata section is supposed to inject the l5d-dst-override custom header to every initial gRPC request sent out by the client, as required by Linkerd. The metadata key-value pairs are implemented as HTTP/2 headers, so logically it should be working.

My only theory is that Envoy will have to communicate with the target gRPC service via non-secure HTTP/2 for that to work, based on this quote:

When an HTTP (not HTTPS) request is received by a Linkerd proxy, the destination service of that request is identified.
(Source)

But unfortunately, my target gRPC service does not support non-secure connections.

Kubernetes version:

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Linkerd version:

Client version: stable-2.14.8
Server version: stable-2.14.8

linkerd check output:

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

Status check results are √

Any ideas what could be missing?
Thanks!

[ori-nat]

Update:
Since Linkerd will not necessarily spread the load evenly across the target pods, I did some testing with very heavy traffic fired towards Envoy. I found that at best, only 2-3 gRPC server pods will handle the requests. However, if you manually delete one of them, the new pod created right after will not receive any requests. Also, if you start off from one pod and scale out, the new pods will never receive any traffic while the first one struggles under 100% of the load.

Bottom line, we have no awareness of new pods.

This is the same behavior we witnessed with just using K8s's default load balancing mechanism, which is L4-based and therefore not suitable for HTTP/2, so I assume that the proxy is not doing anything here for some reason.