WiFi password issue migrating from 4.4.5 to 4.4.6 (IDFGH-11721)

[coratron]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v4.4.6

Espressif SoC revision.

ESP32-D0WD (revision v1.0)

Operating System used.

Linux

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

None

Development Kit.

Custom Board

Power Supply used.

External 3.3V

What is the expected behavior?

Updating the binary using OTA + SoftAP, I expected the untouched password that has been used for several years to still work. This password has not been modified in the code base, the only difference has been moving from 4.4.5 to 4.4.6.

What is the actual behavior?

The Wifi password is not accepted/rejected for any client that tries to connect to the SoftAP of the esp32 despite being EXACTLY the same as usual (as present in the codebase). This has been tested on multiple devices - it is not a hardware issue.

Steps to reproduce.

Case 1

4.4.5 + arduino-esp32 2.0.13 -> compile code and flash over serial, works as it always has

4.4.6 + arduino-esp32 2.0.14 -> compile same code without changes and flash over serial, the usual password is no longer recognised as valid

4.4.6 + arduino-esp32 2.0.14 -> introduce 1 extra character in the password, compile and flash over serial. The new password is valid

Case 2

4.4.5 + arduino-esp32 2.0.13 -> compile code and flash over serial, works as it always has

4.4.6 + arduino-esp32 2.0.14 -> compile same code without changes and flash over serial, the password is no longer recognised as valid

4.4.5 + arduino-esp32 2.0.13 (rollback frameworks) -> compile same code without changes and flash over serial, the old password works again as it always has

I can paste the whole code but the general idea is as follows:

const char* password = "password1234";
WiFi.softAP(ssid, password);
WiFi.softAPConfig(apIP, apIP, NMask);

That's the core of it. The codebase has not changed a single character, only the framework. Case 2 highlights the consistency of the issue. It seems to be related to where/how the NVS containing the WiFi parameters are parsed and stored.

Debug Logs.

No response

More Information.

See following thread that started at arduino-esp32:
espressif/arduino-esp32#8828

[SuGlider]

Similar to espressif/arduino-esp32#8828
Possible work around in espressif/arduino-esp32#8828 (comment)

Edit: Issue is related to a change done in IDF 4.4.6: ad08d94

[SuGlider]

IDF 4.4.5 was released on June 8th and the commit ad08d94 was done in the next day (June 9th) becoming part of IDF 4.4.6.

[MMI]

FWIW

I'm here because I'm looking into a similar issue with clients being unable to connect to the softAP -- but only on a specific device. Our devices have been happily using v4.3.2-114-g7467c68a17. I recently received a report that one of them was not able to upgrade by OTA. Long story short -- the problem is "fixed" when using v4.3.6 or v4.4.6-227-g23e7fa801b.

tl;dr one device works fine regardless of ESP-IDF version and another device requires more current ESP-IDF versions

Has anyone notices variations between modules? Is there a way to query the chip (all the obvious esptool.py queries show my two modules as identical).

[Edit] Having now read all the commentary in #8828 I chased down the commit with the fix and checked to see if it's in any of the three versions of ESP-IDF I listed above. Sure enough, the 4.3.2 does not have the fix but the other two do have it. I was able to confirm by testing older versions of my firmware that this particular instrument would never have been able to OTA update.

[coratron]

@MMI

You can deploy some units based on an esp-idf version, and those would happily update with newer esp-idf versions...until flashed with a version that is "incompatible" with the previous version, then you end up with a "bricked" device. If you deploy devices that have the modification and update them with an esp-idf version that is "compatible" you would not see the problem. A serious issue. It would indeed initially make you think there is a hardware problem, but that's not it.

I think this scenario would be acceptable going from v4 to v5. Unacceptable from 4.4.5 to 4.4.6.

[MMI]

@MMI

You can deploy some units based on an esp-idf version, and those would happily update with newer esp-idf versions...until flashed with a version that is "incompatible" with the previous version, then you end up with a "bricked" device. If you deploy devices that have the modification and update them with an esp-idf version that is "compatible" you would not see the problem. A serious issue. It would indeed initially make you think there is a hardware problem, but that's not it.

I think this scenario would be acceptable going from v4 to v5. Unacceptable from 4.4.5 to 4.4.6.

I agree it's a serious problem. I'm still not 100% clear on the "mechanics" of the problem. Thus I'm having trouble reasoning about the problem itself and the implications; that's made more difficult since the apparent "solution" hidden inside binary ar files.

Failure to OTA is the user visible / application layer symptom but from the console log we're able to see the station join/leave cycle. I have, on my desk, two devices. One shows the problem and one does not. My testing consisted of flashing the devices with something based on today's 4.4.6, starting the AP, connecting, curling a older firmware bin. Then, using the older fw, I start the AP and test whether or not I can connect (and by extension OTA to something).

On the device that shows the problem, it shows using ESP-IDF from before the fix we're discussing (v4.3.2-114-g7467c68a17 -- the why behind choosing that specific version is a longer separate story). The other device works with any ESP-IDF version.

So what's different about the devices? Physical examination shows no difference, the information available to me by esptool shows no difference. What else can I look at?

I think for starters I'm gonna write the code to dump the NVS data. Other suggestions welcome.