Linkerd2 tap won't work properly

[yxzhou1212]

Hello：
I deployed linkerd2(stable-2.11.1) via helm install，when i use the tap component，found some errors in the tap

time="2022-03-31T19:34:17Z" level=info msg="Establishing tap on 192.168.91.252:4190"
time="2022-03-31T19:34:17Z" level=error msg="[192.168.91.252] encountered an error: rpc error: code = Unauthenticated desc = no_tls_from_remote"

Ask:
What is the cause of this, any suggestions? thanks

[edgarkz]

any updates, please?
I’ve followed both control plane and webhook rotating guides -
https://linkerd.io/2.11/tasks/automatically-rotating-control-plane-tls-credentials/
https://linkerd.io/2.11/tasks/automatically-rotating-webhook-tls-credentials/

Can you please explain which one of ca.crt I should use during installations? because we generated 2 pairs -
root.linkerd.cluster.local
and one for
webhook.linkerd.cluster.local

helm installation commands - linkerd2
helm install linkerd2 \ --set-file identityTrustAnchorsPEM=controlplane/ca.crt \ --set identity.issuer.scheme=kubernetes.io/tls \ --set policyValidator.externalSecret=true \ --set-file policyValidator.caBundle=webhook/ca.crt \ --set proxyInjector.externalSecret=true \ --set-file proxyInjector.caBundle=webhook/ca.crt \ --set profileValidator.externalSecret=true \ --set-file profileValidator.caBundle=webhook/ca.crt \ --set installNamespace=false \ -f linkerd2/values-ha.yaml \ -n linkerd \ linkerd/linkerd2

linkerd-viz
helm install linkerd-viz \ --set installNamespace=false \ --set tap.externalSecret=true \ --set-file tap.caBundle=ca.crt \ --set tapInjector.externalSecret=true \ --set-file tapInjector.caBundle=ca.crt \ linkerd/linkerd-viz \ -n linkerd-viz

[kopparam]

@edgarkz If you see the documentation of https://linkerd.io/2.11/tasks/automatically-rotating-webhook-tls-credentials/.

it says,

These certificates are different from the ones that the Linkerd proxies use to secure pod-to-pod communication and use a completely separate trust chain.

Therefore, you need to create 2 trust anchors. One for control plane and one for webhooks.

I've tried using the same ca.crt and separate one, but I still cannot tap requests.

[Joseph94m]

We are experiencing the same issues.
linkerd check returns that everything is okay. In general linkerd is working fine, except for tap.

We're also getting the infamous from the tap logs.

 encountered an error: rpc error: code = Unauthenticated desc = no_tls_from_remote" 

We've also followed the steps in:
https://linkerd.io/2.11/tasks/automatically-rotating-control-plane-tls-credentials/
https://linkerd.io/2.11/tasks/automatically-rotating-webhook-tls-credentials/

[adleong]

I see that you said that linkerd check doesn't report any errors. Please also run linkerd check --proxy -n <data plane namespace>. This will run additional data-plane checks on the proxies in the given namespace and may be able to detect if the pods in that namespace are able to be tapped.

[Joseph94m]

Hey @adleong, the check seems good:

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.11.4 but the latest stable version is 2.12.1
    see https://linkerd.io/2.11/checks/#l5d-version-cli for hints

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-5d99cbb7-nwbc2 (stable-2.11.4)
        * linkerd-destination-5d99cbb7-tpzkp (stable-2.11.4)
        * linkerd-identity-6b8df9975c-2cdlv (stable-2.11.4)
        * linkerd-identity-6b8df9975c-bmn6p (stable-2.11.4)
        * linkerd-proxy-injector-7c964d88bd-kb4vh (stable-2.11.4)
        * linkerd-proxy-injector-7c964d88bd-s52kr (stable-2.11.4)
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints
√ control plane proxies and cli versions match

linkerd-data-plane
------------------
√ data plane namespace exists
√ data plane proxies are ready
‼ data plane is up-to-date
    some proxies are not running the current version:
        * linkerd-destination-5d99cbb7-nwbc2 (stable-2.11.4)
        * linkerd-destination-5d99cbb7-tpzkp (stable-2.11.4)
        * linkerd-identity-6b8df9975c-2cdlv (stable-2.11.4)
        * linkerd-identity-6b8df9975c-bmn6p (stable-2.11.4)
        * linkerd-proxy-injector-7c964d88bd-kb4vh (stable-2.11.4)
        * linkerd-proxy-injector-7c964d88bd-s52kr (stable-2.11.4)
    see https://linkerd.io/2.11/checks/#l5d-data-plane-version for hints
√ data plane and cli versions match
√ data plane pod labels are configured correctly
√ data plane service labels are configured correctly
√ data plane service annotations are configured correctly
√ opaque ports are properly annotated

Status check results are √

Only the pods in the data plane namespace can be tapped ? (linkerd namespace in my case)

[Joseph94m]

Do we have anything on this ?