Refactor aro-dnsmasq-pre.sh to not overwrite /etc/resolv.conf

[ventifus]

Which issue this PR addresses:

Fixes ARO-15180

What this PR does / why we need it:

We've been overwriting /etc/resolv.conf. NetworkManager owns this file and if NetworkManager needs to refresh it we will lose our changes. Instead, create a NetworkManager drop-in /etc/NetworkManager/conf.d/dns-servers.conf with the node's IP.

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

No, but the change needs to be socialized amongst ARO SRE since it affects how nameservers are managed.

How do you know this will function as expected in production?

Testing is required.

[ventifus]

/azp run ci,e2e

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[kimorris27]

What testing do you think we should do before this merges?

[tsatam]

Do we need to make a corresponding change to what the installer puts down? My understanding is that for new clusters, the changes in our operator won't get applied until the cluster is first upgraded, and the cluster will run with what the installer has until then.

[hawkowl]

@tsatam When the Operator is first installed, it is set to allow all reconciliations, and then that is switched to only on upgrades at the end of the install process. So, this will apply to new clusters (at the cost of a reboot + install time, so we should also update the installer wrapper).

[ventifus]

I've tested this now in a UDR + misconfigured DNS cluster (vnet dns = 172.16.0.0). I set aro.dnsmasq.enabled: "false" and manually edited 99-master-aro-dns and 99-worker-aro-dns to have the new content.

After all nodes roll out they have the following config, which looks good

sh-5.1# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 10.0.0.7

sh-5.1# cat /etc/resolv.conf.dnsmasq
# Generated by NetworkManager
search reddog.microsoft.com
nameserver 172.16.0.0

sh-5.1# cat /etc/NetworkManager/conf.d/dns-servers.conf
# Added by dnsmasq.service
[global-dns-domain-*]
servers=10.0.0.7

I made sure all the cluster operators were healthy, and worker machinesets can scale up.

N.B. even with this change we still end up touching /etc/resolv.conf with dnsmasq.service's

ExecStopPost=/bin/bash -c '/bin/mv /etc/resolv.conf.dnsmasq /etc/resolv.conf; /usr/sbin/restorecon /etc/resolv.conf'

I'll fix that up to delete dns-servers.conf instead.