bcm27xx: WATCHDOG: eth0 (bcmgenet): transmit queue 1 timed out

[schuettecarsten]

Describe the bug

This kernel exception occurs from time to time on my Raspberry Pi 4 on the onboard eth device. After this, the local network adapter (eth0) is dead, reboot is required to bring the device back online. The USB-attached network device is still active.

[95463.517310] ------------[ cut here ]------------
[95463.521970] NETDEV WATCHDOG: eth0 (bcmgenet): transmit queue 1 timed out 2890 ms
[95463.529426] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x244/0x24c
[95463.537729] Modules linked in: brcmfmac_wcc(O) pppoe ppp_async brcmfmac(O) wireguard pppox ppp_generic nft_fib_inet nf_flow_table_inet lz4 libchacha20poly1305 ipt_REJECT chacha_neon cfg80211(O) cdc_ncm cdc_ether zstd xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connlabel xt_connbytes xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_REDIRECT xt_NETMAP xt_MASQUERADE xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbnet usbhid ts_fsm ts_bm tcp_bbr slhc sch_cake r8169 r8152 poly1305_neon nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_compat nft_chain_nat nf_tables nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp(O) nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_syslog
[95463.538191]  nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp(O) nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount macvlan lzo_rle lzo lz4_decompress lz4_compress libcurve25519_generic libchacha lan78xx iptable_nat iptable_mangle iptable_filter ipt_ECN ip_tables hid_generic crc_ccitt compat(O) cls_flower brcmutil(O) br_netfilter asn1_decoder act_vlan fuse ntfs3 em_ipset cls_bpf act_bpf act_ctinfo act_connmark sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs hid evdev raspberrypi_hwmon gpio_fan hwmon i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port
[95463.627984]  ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfs ip6_gre ip_gre gre ifb dummy ip6_udp_tunnel udp_tunnel ip6_tunnel oid_registry tunnel6 ip_tunnel veth lockd sunrpc grace autofs4 dns_resolver nls_utf8 bfq xxhash_generic md5 blake2b_generic crypto_acompress nls_iso8859_1 nls_cp437 ehci_pci fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd vfat fat exfat btrfs xor zstd_decompress zstd_compress zstd_common zlib_inflate zlib_deflate xxhash xor_neon raid6_pq lzo_decompress lzo_compress libcrc32c realtek microchip
[95463.777070] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           O       6.6.43 #0
[95463.784481] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[95463.790318] pstate: 40400005 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[95463.797291] pc : dev_watchdog+0x244/0x24c
[95463.801314] lr : dev_watchdog+0x244/0x24c
[95463.805331] sp : ffffffc080ccbe40
[95463.808647] x29: ffffffc080ccbe40 x28: ffffffc080b37778 x27: ffffffc080b450c0
[95463.815805] x26: ffffffc080b469c0 x25: ffffffc080ccbf00 x24: 0000000000000b4a
[95463.822959] x23: ffffffc080b46000 x22: 0000000000000001 x21: ffffff8040cb0000
[95463.830113] x20: ffffff804037d940 x19: ffffff8040cb0440 x18: ffffffc080b5c150
[95463.837265] x17: 3938322074756f20 x16: 64656d6974203120 x15: 000000000000078a
[95463.844417] x14: 000000000000578a x13: 00000000ffffffea x12: ffffffc080b858c0
[95463.851570] x11: ffffffc080b5c150 x10: ffffffc080bb4150 x9 : 00000000003a4be0
[95463.858723] x8 : ffffffc080b4b000 x7 : ffffffc080b4b270 x6 : 0000000000000004
[95463.865876] x5 : 0000000000000001 x4 : 0000000000000004 x3 : 0000000000040005
[95463.873028] x2 : 0000000000000002 x1 : 0000000000000005 x0 : 0000000000000044
[95463.880180] Call trace:
[95463.882628]  dev_watchdog+0x244/0x24c
[95463.886301]  call_timer_fn.isra.0+0x24/0x80
[95463.890495]  __run_timers.part.0+0x20c/0x270
[95463.894772]  run_timer_softirq+0x3c/0x74
[95463.898702]  handle_softirqs+0xf8/0x22c
[95463.902548]  __do_softirq+0x14/0x20
[95463.906041]  ____do_softirq+0x10/0x1c
[95463.909709]  call_on_irq_stack+0x24/0x4c
[95463.913638]  do_softirq_own_stack+0x1c/0x28
[95463.917827]  irq_exit_rcu+0x90/0xc8
[95463.921323]  el1_interrupt+0x38/0x54
[95463.924907]  el1h_64_irq_handler+0x18/0x24
[95463.929011]  el1h_64_irq+0x68/0x6c
[95463.932416]  default_idle_call+0x28/0x3c
[95463.936348]  do_idle+0x1dc/0x228
[95463.939583]  cpu_startup_entry+0x38/0x3c
[95463.943512]  __cpu_disable+0x0/0xd4
[95463.947008]  __secondary_switched+0xb8/0xbc
[95463.951203] ---[ end trace 0000000000000000 ]---

Perhaps related to raspberrypi/linux#4485

OpenWrt version

r27059+22-6bb334c5cf

OpenWrt release

SNAPSHOT

OpenWrt target/subtarget

bcm27xx/bcm2711

Device

Raspberry Pi 4 Model B Rev 1.5

Image kind

Self-built image

Steps to reproduce

No response

Actual behaviour

No response

Expected behaviour

No response

Additional info

No response

Diffconfig

No response

Terms

• I am reporting an issue for OpenWrt, not an unsupported fork.

[github-actions]

Invalid Version reported. r27059+22-6bb334c5cf
Is this from a clean repository?

[schuettecarsten]

There's a PR - raspberrypi/linux#5277 - that adds a module parameter and a dtparam (which just sets the module parameter). Either add genet.eee=N to /boot/cmdline.txt or add dtparam=eee=off to config.txt. Unfortunately, it looks like this patch is not applied to OpenWRT.

[schuettecarsten]

As a workaround, adding these two commands to startup mitigates the crashes:

# Workaround for https://github.com/openwrt/openwrt/issues/16077 and https://github.com/raspberrypi/linux/issues/4485
ethtool --set-eee eth0 eee off

@Noltari
Any idea how to get the RPI patch into OpenWRT drivers?

[Noltari]

There's a PR - raspberrypi/linux#5277 - that adds a module parameter and a dtparam (which just sets the module parameter). Either add genet.eee=N to /boot/cmdline.txt or add dtparam=eee=off to config.txt. Unfortunately, it looks like this patch is not applied to OpenWRT.

As a workaround, adding these two commands to startup mitigates the crashes:

# Workaround for https://github.com/openwrt/openwrt/issues/16077 and https://github.com/raspberrypi/linux/issues/4485
ethtool --set-eee eth0 eee off

@Noltari Any idea how to get the RPI patch into OpenWRT drivers?

The module parameter patch is present in OpenWrt (I haven't tested it though):
https://github.com/openwrt/openwrt/blob/main/target/linux/bcm27xx/patches-6.6/950-0400-net-bcmgenet-Add-eee-module-parameter.patch

I'm not sure why it isn't working, but it should...
Can you test it with genet.eee=0 instead of genet.eee=N?

BTW, just for reference, the ethtool command works on RPi 5 too:

root@rpi5:~# ethtool --show-eee eth1
EEE Settings for eth1:
        EEE status: enabled - inactive
        Tx LPI: disabled
        Supported EEE link modes:  100baseT/Full
                                   1000baseT/Full
        Advertised EEE link modes:  100baseT/Full
                                    1000baseT/Full
        Link partner advertised EEE link modes:  Not reported

root@rpi5:~# ethtool --set-eee eth1 eee off

root@rpi5:~# ethtool --show-eee eth1
EEE Settings for eth1:
        EEE status: disabled
        Tx LPI: disabled
        Supported EEE link modes:  100baseT/Full
                                   1000baseT/Full
        Advertised EEE link modes:  Not reported
        Link partner advertised EEE link modes:  Not reported

[schuettecarsten]

@Noltari
You're right - what about this patch, I think that's why my config.txt changes did not work. Will try cmdline.txt.
raspberrypi/linux@64f1b87

Update:
Adding genet.eee=N to cmdline.txt works.

Modifications to cmdline.txt and config.txt are not persistent during sysupgrade, correct?
Any solution for this except patching the source code before?

[Noltari]

@Noltari You're right - what about this patch, I think that's why my config.txt changes did not work. Will try cmdline.txt. raspberrypi/linux@64f1b87

The patch you provided is outdated, but it's present on

openwrt/target/linux/bcm27xx/patches-6.6/950-0040-BCM2708-Add-core-Device-Tree-support.patch

Line 5743 in 7993a52

+ eee = <&chosen>,"bootargs{on='',off='genet.eee=N'}";

Update: Adding genet.eee=N to cmdline.txt works.

Awesome.

Modifications to cmdline.txt and config.txt are not persistent during sysupgrade, correct? Any solution for this except patching the source code before?

They should be persistent: https://github.com/openwrt/openwrt/blob/main/target/linux/bcm27xx/base-files/lib/upgrade/keep.d/platform

[schuettecarsten]

Ok, great - I was quite sure that it did not work about one year ago :-D
So, as patches are available, this can be closed, or should genet.eee=N added as a default?

[Noltari]

So, as patches are available, this can be closed, or should genet.eee=N added as a default?

I wouldn't disable EEE as default since it works with some devices.
In fact, it's not disabled by default on Raspbian.

[schuettecarsten]

Closed as there is a working workaround:
Add genet.eee=N to /boot/cmdline.txt