[BUG] DNS failure after upgrading to fedora 41 when docker is installed on host (firewall DROP policy)

[es02]

General information

• OS: Linux
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)? Yes
• Running CRC on: Laptop

CRC version

WARN A new version (2.44.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.44.0/crc-linux-amd64.tar.xz 
CRC version: 2.43.0+268795
OpenShift version: 4.17.1
MicroShift version: 4.17.1

CRC status

CRC VM:          Stopped
OpenShift:       Stopped (v4.17.1)
RAM Usage:       0B of 0B
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     46.53GB
Cache Directory: /home/thobson/.crc/cache

CRC config

- consent-telemetry                     : yes
- cpus                                  : 8
- disk-size                             : 64
- memory                                : 24000

Host Operating System

NAME="Fedora Linux"
VERSION="41 (Workstation Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

1. After upgrade to Fedora 41
2. Start CRC
3. Note DNS failure
4. Note Pull Secret Save Failure

Expected

CRC starts without errors

Actual

CRC starts with errors.
New pods experience imagePullBackOff

Logs

Before gather the logs try following if that fix your issue
Done

output of crc start --log-level debug

[gbraad]

This is related to a network issue, but we have not been able to reproduce this. It might be an issue with systemd-resolved, but we can not be sure yet. Too little information to go by.

As an alternative, you can use the usermode network-mode.

[savitojs]

Using network-mode fixes the issues for me.

If anyone is interested, here are the commands to fix:

❯ crc config view |grep network

❯ crc config set network-mode user
Network mode changed. Please run `crc cleanup` and `crc setup`.

❯ crc config view |grep network
- network-mode                          : user

❯ crc config set host-network-access true
Changes to configuration property 'host-network-access' are only applied during 'crc setup'.
Please run 'crc cleanup' followed by 'crc setup' for this configuration to take effect.

❯ crc config view |grep network
- host-network-access                   : true
- network-mode                          : user

❯ crc cleanup

btw, here is some more info if it helps dev(s) reproduce:

NAME="Fedora Linux"
VERSION="41 (Workstation Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="Workstation Edition"
VARIANT_ID=workstation

❯ rpm -qa | egrep -i 'resolved|dns|udev|kernel'
libmicrodns-0.2.0-10.fc41.x86_64
kernel-srpm-macros-1.0-24.fc41.noarch
kernel-headers-6.11.3-300.fc41.x86_64
libgudev-238-6.fc41.x86_64
kernel-modules-core-6.11.5-300.fc41.x86_64
kernel-core-6.11.5-300.fc41.x86_64
nss-mdns-0.15.1-12.fc41.x86_64
kernel-modules-6.11.5-300.fc41.x86_64
kernel-6.11.5-300.fc41.x86_64
dnsmasq-2.90-3.fc41.x86_64
mod_dnssd-0.6-32.fc41.x86_64
kernel-devel-6.11.5-300.fc41.x86_64
libreport-plugin-kerneloops-2.17.15-3.fc41.x86_64
python3-pyudev-0.24.3-3.fc41.noarch
abrt-addon-kerneloops-2.17.6-2.fc41.x86_64
system-config-printer-udev-1.5.18-11.fc41.x86_64
kernel-modules-extra-6.11.5-300.fc41.x86_64
kernel-modules-core-6.11.7-300.fc41.x86_64
kernel-core-6.11.7-300.fc41.x86_64
kernel-modules-6.11.7-300.fc41.x86_64
aardvark-dns-1.13.1-1.fc41.x86_64
kernel-modules-extra-6.11.7-300.fc41.x86_64
kernel-6.11.7-300.fc41.x86_64
kernel-devel-6.11.7-300.fc41.x86_64
systemd-udev-256.8-1.fc41.x86_64
systemd-resolved-256.8-1.fc41.x86_64
kernel-modules-core-6.11.8-300.fc41.x86_64
kernel-core-6.11.8-300.fc41.x86_64
kernel-modules-6.11.8-300.fc41.x86_64
kernel-tools-libs-6.11.8-300.fc41.x86_64
kernel-tools-6.11.8-300.fc41.x86_64
kernel-modules-extra-6.11.8-300.fc41.x86_64
kernel-6.11.8-300.fc41.x86_64
kernel-devel-6.11.8-300.fc41.x86_64

❯ uname -a
Linux 6.11.8-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Nov 14 20:37:39 UTC 2024 x86_64 GNU/Linux

❯ crc version
CRC version: 2.44.0+facc77
OpenShift version: 4.17.3
MicroShift version: 4.17.3

[gbraad]

The OP switched between VPN use and experienced issues with DNS not being resolved inside the VM. The preflight for curl quay.io failed. Though it was not obvious what caused this.

[mscherer]

My coworker @gansheer faced the same issue, and we found that this was caused by Docker engine adding a DROP policy on the FORWARD chain in the firewall, similar to that: https://fedoraproject.org/wiki/Changes/NetavarkNftablesDefault#Known_Issue_with_docker but affecting libvirtd networking (and the default setup of CRC).

[praveenkumar]

@mscherer Thank you for letting us what causing us. Is there any specific reason of using docker instead podman (which is by default)? Any missing feature from podman side?