mediatek mt7915e firmware instability

[szyper-sakrona]

Hello, I've tested nearly all your releases for my WSM20 machines - but current releases has latest mediatek firmware (mt7915e) from 202310xxx. I've noticed that it generates problems with wifi - timeouts and wifi is not working normally. I've uploaded older firmware from 20230418xxx - and after 1 day it is working good. I'll be informing is that better for longer periods.
Maybe it is more safe to get back to this firmware on that routers.
Regards

[ptpt52]

I would revert to older firmware for mt7915e in next release.

[szyper-sakrona]

After testing for a few days - something wrong is with usteer and these firmwares...I do not know, but after some hours there is these symptoms (dmesg dump):

[ 4027.942671] mt7915e 0000:02:00.0: Message 000025ed (seq 11) timeout
[ 4027.948975] phy1-ap0: failed to set key (0, cc:42:10:45:bd:dc) to hardware (- 145)
[ 4048.422139] mt7915e 0000:02:00.0: Message 00005aed (seq 12) timeout
[ 4068.901634] mt7915e 0000:02:00.0: Message 00005aed (seq 13) timeout
[ 4089.381003] mt7915e 0000:02:00.0: Message 000025ed (seq 14) timeout
[ 4109.860433] mt7915e 0000:02:00.0: Message 00005aed (seq 15) timeout
[ 4130.339856] mt7915e 0000:02:00.0: Message 00005aed (seq 1) timeout
..........................................................................................................................................................
[ 9475.495869] phy0-ap0: HW problem - can not stop rx aggregation for 6a:98:b9:6 c:72:e6 tid 0
[ 9475.504241] phy0-ap0: HW problem - can not stop rx aggregation for 6a:98:b9:6 c:72:e6 tid 6
[ 9475.562797] ------------[ cut here ]------------
[ 9475.567482] WARNING: CPU: 2 PID: 1856 at /mnt/Data/Sources/openwrt/x-wrt/tmpf s/target-mipsel_24kc_musl/linux-ramips_mt7621/mac80211-regular/backports-6.6.15/ net/mac80211/agg-tx.c:407 ___ieee80211_stop_tx_ba_session+0x2ac/0x2cc [mac80211]
[ 9475.588675] Modules linked in: pppoe ppp_async wireguard pptp pppox ppp_mppe ppp_generic nft_fib_inet mt7915e mt76_connac_lib mt76 mac80211 libchacha20poly13 05 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_rece nt xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_HL xt_DSCP xt_DNETMAP xt_DHCPMAC xt_CLASSIFY ts_fsm ts_bm tcp_scalable tcp_bbr slhc sch_cake poly1305_mips nft_tproxy nft_so cket nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_rejec t nft_redir nft_quota nft_queue nft_objref nft_numgen nft_nat nft_meta_bridge nf t_masq nft_log nft_limit nft_hash nft_fwd_netdev nft_fib_ipv6 nft_fib_ipv4 nft_f ib nft_dup_netdev nft_ct nft_counter nft_compat nft_chain_nat nfnetlink_queue nf tproxy_ipv6 nf_tproxy_ipv4 nf_tables nf_socket_ipv6 nf_socket_ipv4 nf_reject_ip v4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip
[ 9475.589192] nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_l og_syslog nf_dup_netdev nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntr ack_ftp nf_conntrack_broadcast nf_conntrack_bridge ts_kmp nf_conntrack_amanda nf _conncount macvlan libcurve25519_generic libcrc32c ipvlan iptable_nat iptable_ma ngle iptable_filter ipt_ECN ip_tables hwmon crc_ccitt compat_xtables compat chac ha_mips br_netfilter asn1_decoder natflow natcap nf_nat nf_conntrack sch_tbf sch ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact cryptodev xt_set ip_set_list_set ip_se t_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_po rt ip_set_bitmap_ipmac ip_set_bitmap_ip
[ 9475.676690] ip_set nfnetlink jool_siit jool nf_defrag_ipv4 jool_common ip6ta ble_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 bondin g tls ip6_gre ip_gre gre nat46 nf_defrag_ipv6 sit ip6_tunnel tunnel6 tunnel4 ip tunnel udp_diag tcp_diag raw_diag inet_diag tun ovpn_dco_v2 udp_tunnel ip6_udp_t unnel nls_utf8 nls_iso8859_1 nls_base crypto_user algif_skcipher algif_rng algif _hash algif_aead af_alg sha512_generic sha256_generic libsha256 sha1_generic seq iv jitterentropy_rng drbg md5 crypto_hw_eip93 hmac ecb des_generic libdes cmac a uthencesn authenc arc4 leds_gpio softdog gpio_button_hotplug crc32c_generic
[ 9475.819065] CPU: 2 PID: 1856 Comm: hostapd Not tainted 5.15.148 #0

and wifi is not working normally...
Any chances that maybe you can migrate this device to for ex. new 6.x kernels change "hostap" version?

[ptpt52]

try latest version now.

[szyper-sakrona]

I temporary disable usteer on all my APs - maybe the firmware is "innocent" and there is hostapd+usteer conflict?

[szyper-sakrona]

Dear ptpt52, the problem is described here openwrt/openwrt#11931
Is there any chance to introduce 6.1 kernel to ramips builds? maybe it could resolve errors - the latest "stable" builds was late 2023y. Thanks!

[ptpt52]

the lastest version is now kernel 6.1

[godshades]

@ptpt52 wifi unstable, sometime wifi disconnect, sometime router highload cpu
i'm using 2x cr6608 with mesh: router 1 have internet, router 2 not (can't access lan too), after reboot router 1, both have network, idk why

[szyper-sakrona]

latest build also have seq timeouts - wifi unstable - I think the hostapd build is broken from some months, maybe latest patches could fix this cohabitation.

[ptpt52]

next release would update to latest mt76 driver, then try again when next release is ready

[szyper-sakrona]

now I'm got back to official 23.05.02 version with older software - testing different combinations. We will see!

[szyper-sakrona]

Dear Ptpt52 - could you make a test version of Xwrt with these hostapd patches from this repo?: https://github.com/cmonroe/feed-wifi-master/tree/smartrg-master/hostapd
Anyway I try latest version - I notice that it is connected with AX clients, usteer, hostapd - the firmware is innocent.
Thanks

[ptpt52]

x-wrt new release is now ready.

Dear Ptpt52 - could you make a test version of Xwrt with these hostapd patches from this repo?:

It looks like not that easy to build agaist with that hostapd

[szyper-sakrona]

Hello, latest update seems to works good, also with latest mediatek firmware 7915e (date 20231215).
Thanks!
Regards!

[szyper-sakrona]

Hello Ptpt52, after testing various configurations - your kernel's version builds and mediatek firmwares I found only one combination which is surprising stable. I used it with DAWN (have several AP with roaming).
This is the best stable (30+ days) one:
X-WRT 24.01_b202403222222, Mantic
with latest CMonroe version mt76 - mt7915e from 2024040X (https://github.com/cmonroe/feed-wifi-master/tree/smartrg-master/mt76/files/firmware)
Anything newer gives also kernel "...Message 00005aed (seq .....) timeout" errors after a half day or less/more.
Interesting discussion is here openwrt/mt76#690
What do You thing about this specific combination?
Regards
I didn't test latest builds - is there any chance that latest version could fix mt76/hostapd problems?
Thx