Enhancement Request: Support for Physical device peer

[tahir24434]

Request
Currently Meshnet assume that all devices in topology are virtual and running inside K8s pods.
In real world, you may have a topology where some of the devices are physical while some other running as virtual devices in K8s.

Summary
With physical device support, user will be able to define a device in topology file for which a pod will not be created. Nothing else will be changed, all other objects like CustomeResource will be created as before.
The configuration of this device will be delegated to the operator. Meshnet will not be responsible for its configuration as there are many possible combination of vendor/series and way of configurations.

Motivation

• Devices running outside of the pods can become part of the topology.
• Any device for which virtual image is not readily available can be tested.
• Before bringing up a site, one can bring up a virtual site and test it with existing physical site.

Implementation Details
Changes are needed in both k8s-topo and meshnet.
K8s-topo changes
Links between virtual devices will be bring up as before. Changes will be needed only if one of the endpoint in link is physical.

• Introduce 'phy' keyword to represent a physical device.
  endpoints: ["vmx-1:eth2:10.40.5.2/24", "phy-1:eth1:172.31.26.56/24"]
  phy-1 will be a physical device. vmx-1 is virtual device.
• No pod will be created for the 'phy' peer.
• vxlan link will be created on the virtual device with 'phy' ip as remote address.

Meshnet changes

• Just like in macvlan case, see whether peer device has 'phy' prefix? If it does, then just create a vxlan link on virtual device with 'phy' ip as remote and continue.

[tahir24434]

k8s-topo change: https://github.com/networkop/k8s-topo/pull/8/files
meshnet change: https://github.com/networkop/meshnet-cni/pull/14/files.

[networkop]

What would be the steps for operator to configure the "phy" side of the link? Do you find the node where the pod got scheduled and configure that as a remote VTEP IP on the physical device?

I have no problems with these changes in principle. My only concern is the way vxlan link information is encoded. The ip address in the device:intf:ip is supposed to encode the internal pod IP. In this case, it becomes overloaded with additional meaning as "destination VTEP IP", so in this way the link definition contains overlay IP on one end and underlay IP on the other, which can become confusing.

Would it make sense to define remote VTEP attributes as extra link options? e.g.

- endpoints: ["vmx-1:eth2:10.40.5.2/24", "phy-1:vxlan1"]
  link_options:
    remote_vtep: 172.31.26.56
    vni: 5050

[tahir24434]

Regarding vxlan link information
I agree with you that it will become overloaded. I thought about it while making changes.
I took assumption that developer will understand that in case of 'phy', there is no pod involved. So, IP address mentioned their will be a special IP (in our case VTEP).

But I like your suggestion better. I'll make those changes over the weekend.

Regarding configuration of 'phy' side of link
There are so many different vendors/series/versions of devices and many ways to configure them. It is best that Meshnet relieves himself from responsibility of configuration of other end. It should be the responsibility of Operator to configure that end. Using CustomResourceObject 'kubectl get Topology phy-1 -o yaml', operator can get the picture of peer pod link to 'phy' and configure it according to whatever way suits her.