[WIP] Replace CRI-RM binary with NRI balloons plugin

[ppalucki]

Just for testing purposes.

Status as of 7/16/2024:

nri-plugin can be deployed but requires installation of dbus in machine--local--local pod in the followin way, before enabling plugin:

# create shoot without extension
kubectl apply -f ./examples/shoot.yaml # ../gardener-extension-cri-resmgr/examples/shoot.yaml - extension needs to be disabled
# (or disable extension manually)
kubectl patch shoot local -n garden-local -p '{"spec":{"extensions": [ {"type": "cri-resmgr-extension", "disabled": true} ] } }'
# then install and check dbus service in machine shoot--local--local pod
kubectl exec -i -n shoot--local--local  `kgp -n shoot--local--local -l app=machine -o name` -- bash -c "apt -y update ; apt -y install dbus; rm -rf /run/dbus/system_bus_socket ; systemctl start dbus.socket dbus.service; systemctl statu
# and then enable extension for shoot
kubectl patch shoot local -n garden-local -p '{"spec":{"extensions": [ {"type": "cri-resmgr-extension", "disabled": false} ] } }'
# then the end result is following
cd gardener
./hack/usage/generate-admin-kubeconf.sh > admin-kubeconf.yaml
KUBECONFIG='admin-kubeconf.yaml' kubectl get pods -n kube-system -l app.kubernetes.io/instance=balloons -o wide
KUBECONFIG='admin-kubeconf.yaml' kubectl logs -n kube-system -l app.kubernetes.io/instance=balloons
Defaulted container "nri-resource-policy-balloons" out of: nri-resource-policy-balloons, patch-runtime (init)
I: [           policy           ] assigning container kube-system/coredns-5c4c97c6f-zpctw/coredns to balloon reserved[0]{cpus:"0", mems:"0"}
I: [           policy           ] assigning container kube-system/nri-resource-policy-balloons-8nkg5/nri-resource-policy-balloons to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/calico-node-vertical-autoscaler-6cf9c8bf8d-gmnjk/autoscaler to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/calico-kube-controllers-56f86cc77b-7dlp5/calico-kube-controllers to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/blackbox-exporter-6d5b5499b-f8wfq/blackbox-exporter to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/calico-typha-horizontal-autoscaler-78ff9f78d8-kxrf9/autoscaler to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/apiserver-proxy-g49kl/sidecar to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/apiserver-proxy-g49kl/proxy to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [           policy           ] assigning container kube-system/calico-node-89vwz/calico-node to balloon reserved[0]{cpus:"0,48", mems:"0"}
I: [         nri-plugin         ] <= Synchronize

Bug,Issues,Things to do:

• Gardener kind-based setup for E2E locally works only with cgroup v2 (e.g. SLES15SP6) with systemd driver (for docker two containerd's) - to be reported to Gardener team or containerd in machine pod needs to changed to cgroupfs!
• dbus is required by NRI plugin init container - find a way to automate deployment of dbus service in machine pod some pods inside shoot worker node already try to map /run/dbus/system_bus_socket and it becomes directory which need to be deleted before starting dbus.service/socket with rm -rf /run/dbus/system_bus_socket
• Consider renaming plugin (cri-resmgr to nri)
• Support image vector (providing images in Gardener native way)
• Config propagation (current config aren't considered by NRI, require configuration in a form of custom resources "Balloons Policy"
• Consider supporting other nri-plugins: like memtierd, topology aware ...