VyOS firewall messes up nftables rules applied by LBaaS agent

[ghavag]

Description

The LBaaS stops working every time a new firewall configuration has been committed. The problem has been observed with VyOS Sagitta.

The problem seems to be, that VyOS flushes nftables rules before in order to apply the new rules which implements the VyOS firewall configuration.

Snipped of nftables rules with working LBaaS:

table ip vyos_nat {
        set A_ALLOWED-CLIENTS {
                type ipv4_addr
                flags interval
                auto-merge
                elements = { x.x.x.x-x.x.x.x, x.x.x.x/x,
                             x.x.x.x, x.x.x.x }
        }

        set P_K8S-INGRESS {
                type inet_service
                flags interval
                auto-merge
                elements = { 32080, 32443 }
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                counter packets 3 bytes 180 jump VYOS_PRE_DNAT_HOOK
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter packets 8 bytes 613 jump VYOS_PRE_SNAT_HOOK
                oifname "bond0.5" ip saddr 10.2.128.0/24 counter packets 0 bytes 0 snat to x.x.x.x comment "SRC-NAT-1"
                oifname "bond0.5" ip saddr 10.2.0.0/20 counter packets 0 bytes 0 snat to x.x.x.x comment "SRC-NAT-2"
                oifname "bond0.5" ip saddr 10.254.20.0/27 counter packets 0 bytes 0 snat to x.x.x.x comment "SRC-NAT-3"
        }

        chain VYOS_PRE_DNAT_HOOK {
                ip daddr x.x.x.x tcp dport 80 meta mark set 0x00000001 ct mark set meta mark dnat to numgen inc mod 1 map { 0 : 10.2.1.10 }:32080
                ip daddr x.x.x.x tcp dport 443 meta mark set 0x00000001 ct mark set meta mark dnat to numgen inc mod 1 map { 0 : 10.2.1.10 }:32443
        }

        chain VYOS_PRE_SNAT_HOOK {
        }
}

Snipped of nftables rules after changes to the firewall configuration have been applied:

table ip vyos_nat {
        set A_ALLOWED-CLIENTS {
                type ipv4_addr
                flags interval
                auto-merge
                elements = { x.x.x.x-x.x.x.x, x.x.x.x/x,
                             x.x.x.x, x.x.x.x }
        }

        set P_K8S-INGRESS {
                type inet_service
                flags interval
                auto-merge
                elements = { 32080, 32443 }
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                counter packets 3 bytes 180 jump VYOS_PRE_DNAT_HOOK
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter packets 0 bytes 0 jump VYOS_PRE_SNAT_HOOK
                oifname "bond0.5" ip saddr 10.2.128.0/24 counter packets 0 bytes 0 snat to x.x.x.x comment "SRC-NAT-1"
                oifname "bond0.5" ip saddr 10.2.0.0/20 counter packets 0 bytes 0 snat to x.x.x.x comment "SRC-NAT-2"
                oifname "bond0.5" ip saddr 10.254.20.0/27 counter packets 0 bytes 0 snat to x.x.x.x comment "SRC-NAT-3"
        }

        chain VYOS_PRE_DNAT_HOOK {
                return
        }

        chain VYOS_PRE_SNAT_HOOK {
                return
        }
}

As one can see, the chain VYOS_PRE_?NAT_HOOK has changed.

To fix that the LBaaS agent needs to be restarted.

To reproduce

1. Have an running LBaaS
2. Change firewall configuration, for example:

   configure
   
   set firewall ipv4 name FOO default-action drop
   # OR
   delete firewall ipv4 name FOO
   
   commit
   

Workaround

If not exist, create the commit post hook scripts directory:

sudo mkdir -p /config/scripts/commit/post-hooks.d
sudo chmod -R 775 /config/scripts/commit

Create a script /config/scripts/commit/post-hooks.d/10-restart-ch-lbaas-agent with the following content:

#!/bin/sh

# This is a post commit hook script to be placed under
# /config/scripts/commit/post-hooks.d. It will be executed after each commit in
# order bypass a problem with the Cloud and Heat LBaaS agent and VyOS.

if systemctl is-active ch-k8s-lbaas; then
	sudo systemctl restart ch-k8s-lbaas
fi

Make sure that the script has execution permissions. It will run after each commit of a new configuration and restart the LBaaS agent.

Note that there must be a systemd unit called ch-k8s-lbaas which is responsible for starting, stopping etc. the LBaaS agent on the VyOS firewall.