Introduce new external endpoints controller

[mateiidavid]

For mesh expansion, we need to register an ExternalWorkload's service membership. Service memberships describe which Service objects an ExternalWorkload is part of (i.e. which service can be used to route traffic to an external endpoint).

Service membership will allow the control plane to discover configuration associated with an external endpoint when performing discovery on a service target.

To build these memberships, we introduce a new controller to the destination service, responsible for watching Service and ExternalWorkload objects, and for writing out EndpointSlice objects for each Service that selects one or more external endpoints.

As a first step, we add a new ExternalEndpointManager in the destination service's package that watches services and workloads. In a follow-up change, the ExternalEndpointManager will additionally perform the necessary reconciliation by writing EndpointSlice objects.

Since Linkerd's control plane may run in HA, we also add a lease object that will be used by the manager. When a lease is claimed, a flag is turned on in the manager to let it know it may perform writes.

A more compact list of changes:

• Add a new ExternalEndpointManager file and struct that wires up the necessary mechanisms to watch resources.
  • We also add a new test file; the tests are rudimentary for now.
• Add RBAC rules to the destination service:
  • Allow policy and destination to read ExternalWorkload objects
  • Allow destination to create / update / read Lease objects
• Wire-up mock clients for ExternalWorkload objects in anticipation of more involved tests

---

Note to reviewers

You can run the controller locally to test the behaviour. You need to make a small change to controller/destination/main

• Add k8s.ExtWorkload to informers configured (make sure it's the branch that uses endpointslices)
• Create and start the endpoints controller.

diff --git a/controller/cmd/destination/main.go b/controller/cmd/destination/main.go
index 845c9e03c..3090e1401 100644
--- a/controller/cmd/destination/main.go
+++ b/controller/cmd/destination/main.go
@@ -9,6 +9,7 @@ import (
        "syscall"

        "github.com/linkerd/linkerd2/controller/api/destination"
+       externalworkload "github.com/linkerd/linkerd2/controller/api/destination/external-workload"
        "github.com/linkerd/linkerd2/controller/api/destination/watcher"
        "github.com/linkerd/linkerd2/controller/k8s"
        "github.com/linkerd/linkerd2/pkg/admin"
@@ -107,7 +108,7 @@ func Main(args []string) {
                        *kubeConfigPath,
                        true,
                        "local",
-                       k8s.Endpoint, k8s.ES, k8s.Pod, k8s.Svc, k8s.SP, k8s.Job, k8s.Srv,
+                       k8s.Endpoint, k8s.ES, k8s.Pod, k8s.Svc, k8s.SP, k8s.Job, k8s.Srv, k8s.ExtWorkload,
                )
        } else {
                k8sAPI, err = k8s.InitializeAPI(
@@ -159,6 +160,9 @@ func Main(args []string) {
        metadataAPI.Sync(nil)
        clusterStore.Sync(nil)

+       ec, _ := externalworkload.NewEndpointsController(k8sAPI, "my-hostname", "default", done)
+       ec.Start()
+

You can then run it with kubectl proxy (as a bg job) and go run controller/cmd/main.go destination. You should see logs from the new external-endpoints-controller component.

INFO[2024-01-16T14:38:53Z] attempting to acquire leader lease default/linkerd-destination-endpoint-write...
INFO[2024-01-16T14:38:53Z] received default/kubernetes                   component=external-endpoints-controller
INFO[2024-01-16T14:38:53Z] received default/nginx-svc                    component=external-endpoints-controller

Sample manifests:

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    app: nginx
  ports:
    - name: http
      protocol: TCP
      port: 80
---
 apiVersion: workload.linkerd.io/v1alpha1
 kind: ExternalWorkload
 metadata:
  name: test
  labels:
    app: nginx
 spec:
  meshTls:
    serverName: "foo"
    identity: "foo"
  workloadIPs:
  - ip: "192.0.2.0"
  ports:
  - port: 80

[zaharidichev]

Good job. We are getting there. I left some feedback that can hopefully make things easier to read. This diffing logic is always quite error-prone. This is why I think you need to rely on more isolated tests like the one I provided in the comments. This should make it easier to think through and verify stuff.

Also, it would be good you can provide some functional testing instructions for this change so anyone can try it out themselves.

[zaharidichev]

Looks good, one last question to think through.

[zaharidichev]

LGTM. Great job.

[adleong]

It may be worth noting in a comment that this is structurally based on https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go and perhaps even noting cases where they significantly differ (if any).